
Share
Stanford researchers built a new benchmark from documented AI hallucination cases to measure generative AI's spread through state courts, and found that the tools meant to catch it are far less reliable than they appear.
Imagine you're a single parent fighting an eviction, or a worker trying to recover unpaid wages, with no lawyer and no money to hire one. You turn to a chatbot to help draft your complaint because it's free and it's there at midnight when the courthouse is closed. That scenario is no longer hypothetical. It's happening across the country, and it's reshaping how the justice system works, often without anyone noticing until something goes wrong.
Since ChatGPT's release, AI-generated text has turned up everywhere from news articles to political speeches to UN press releases. Courts have proven especially vulnerable. Over the past three years, fake legal citations, the kind a chatbot invents when it doesn't actually know the answer, have appeared in filings submitted by lawyers, district attorneys, and even judges. A Florida lawyer was referred for possible discipline after a filing judges called AI-generated. A judge's own ruling, riddled with AI errors, led to his case being reassigned. Legal commentators and the New York Times have warned that a wave of AI-drafted filings could overwhelm courts, waste scarce judicial resources, and ultimately delay justice for people who can least afford to wait.
But here's the harder question nobody had answered well: how common is AI assistance in court filings, really? And can we actually prove it, case by case, with any confidence? Researchers at the Stanford Institute for Human-Centered AI and the Regulation, Evaluation, and Governance Lab, known as RegLab, set out to answer both.
Their method started with something deceptively simple: collect and analyze complaints from county courts and look for AI's fingerprints. What they found was striking. Signs of AI-assisted drafting in Santa Clara County, California, rose from near zero in 2022 to more than a third of examined complaints by 2026. That's not a fringe phenomenon. That's a structural shift in how people interact with the legal system.
The trouble is that detecting AI text reliably is much harder than it sounds. Detection tools work by scanning for AI's telltale habits, things like an overreliance on certain phrasing or em-dashes, patterns that show up differently depending on context. Legal writing, though, doesn't look like typical prose. It leans on a narrower vocabulary and relies heavily on boilerplate language repeated across thousands of filings. That leaves less distinctive signal for a detector to latch onto. Worse, the commercial detectors built to catch AI writing likely weren't trained on much legal text in the first place.
Put those problems together and detection becomes shaky exactly where accuracy matters most. The researchers found that an AI detector flagged anywhere from 10 percent to 98 percent of the very same set of 2026 Santa Clara complaints as AI-written, depending purely on which settings were used. Same documents. Wildly different verdicts. That's not a minor calibration issue. It's a warning sign for anyone relying on off-the-shelf AI detectors to make consequential decisions in professional, specialized fields.

The team's breakthrough came from an unlikely place: cases where AI use was undeniable because the mistakes were too egregious to ignore. These are hallucinated citations, fabricated case lookups, the kind of errors that have already drawn judicial sanctions. Legal researcher Damien Charlotin has been tracking these incidents since April 2025 through his AI Hallucinations Cases Database, built specifically to document when judges call out parties for citing cases that don't exist. More than 1,500 such cases have now been documented nationwide.
Researchers paired Charlotin's database with carefully constructed control cases to build a new benchmark, called PhantomDocket, designed specifically to test AI detection in legal settings. They also developed a calibration method called ScoreMix, which compares filings against known hallucinated text rather than relying on a detector's raw, often unreliable output. Together, these tools let the team generate conservative, more trustworthy estimates of how much AI is actually present in court filings, rather than guesses swinging wildly based on arbitrary settings.
What they found paints a textured picture of how generative AI has quietly embedded itself in four state court systems. By 2026, AI usage in non-debt-collection cases reached 38 percent in Santa Clara County and 36 percent in San Francisco County. Oklahoma's Tulsa and Oklahoma counties showed notably lower adoption, at 17 percent and 16 percent respectively, suggesting real regional variation in how litigants are using these tools.
The starkest divide, though, runs along representation status. People representing themselves without a lawyer, known as pro se litigants, are adopting AI far faster than attorneys. In Santa Clara, AI-linked signs in self-represented complaints jumped from 19 percent in 2024 to 72 percent in 2026. Attorney-filed complaints rose too, but far more modestly, from 5 percent to 34 percent over the same period. Usage also varies by case type: employment and business disputes show heavy AI involvement, while templated debt-collection filings, which make up nearly half of some courts' dockets, show almost none. Within individual filings, AI traces cluster most often in the statement of facts and the causes of action, the sections where a self-represented person would most need help explaining their situation in formal legal language.
None of this means every flagged filing was actually written by AI, and that distinction matters enormously. The researchers caution explicitly that detection tools are far better suited to estimating overall prevalence across a population of filings than to pinpointing any single document. Even the best detectors in their benchmark still carried meaningful false positive and false negative rates. Treating a detector's output as proof in an individual case, the kind of proof that could affect someone's access to justice, would be a serious misuse of an imperfect tool.
What this research really surfaces is a harder policy question: is AI quietly helping people who could never afford a lawyer get their day in court, or is it flooding dockets with weak, error-prone filings that waste everyone's time? Likely some of both. The honest answer is that courts don't yet have the infrastructure to tell the difference reliably. The researchers argue for stronger disclosure policies, requiring clearer acknowledgment of AI assistance in filings, so that both accountability and accurate measurement become possible going forward. Without that transparency, courts are left guessing at the scale of a problem they can already see is growing, but struggle to pin down in any individual case. That gap between detecting a trend and proving a specific instance is exactly where the justice system's credibility is now being tested.
Tags
Original Sources
AI Might Be Flooding State Court Houses, but Proving It Is Another Matter | Stanford HAI
↗ https://hai.stanford.edu/news/ai-might-be-flooding-state-court-houses-but-proving-it-is-another-matter
About the author
Amara's entry point into AI was an epidemiology role at a London research hospital, where she spent five years studying how digital health tools reached — or conspicuously failed to reach — underserved communities. Watching early algorithmic systems in healthcare quietly entrench existing inequalities, she redirected her career toward the systemic consequences of AI at scale. She covers AI through an unflinching lens: who benefits, who bears the cost, and what evidence actually says versus what the press release claims. Her writing is calm and precise, but she doesn't mistake balance for neutrality.
More from The Steward →This Week's Edition
20 November 2026
29 articles
Related Articles

Who Decides If Your Treatment Is Medically Necessary? Industry Leaders Say It Shouldn't Be AI Alone
Policy & Regulation · 6 min

Epic Systems Pushes Back on Antitrust Criticism, Pledges Broader AI Data Access
Policy & Regulation · 5 min

Why Some Computer Scientists Doubt Superintelligence Is Coming at All
Policy & Regulation · 5 min
Related Articles

Who Decides If Your Treatment Is Medically Necessary? Industry Leaders Say It Shouldn't Be AI Alone
Policy & Regulation · 6 min

Epic Systems Pushes Back on Antitrust Criticism, Pledges Broader AI Data Access
Policy & Regulation · 5 min

Why Some Computer Scientists Doubt Superintelligence Is Coming at All
Policy & Regulation · 5 min
More Stories
© 2026 Cedar & Bloom. All rights reserved.