Our error rates,
one check at a time

How to read the tables

Each check gets its own table. We do not add them up into one score, because a missed invented case and a real case flagged as missing cost you different things. The false-flag number sits next to the catch number wherever we have both.

• Flagged means a red row: "check this". False flag means a red row on a citation that was fine.
• Confirmed (deep check) means the check found a passage that states the claim. Marked as support on a wrong pair means it did that when the passage does not support the claim. That second number is the one to watch.
• Dev: we changed the checker while looking at these results. Held out: the set was put aside and run once, at the end. Control: a set built to test one kind of failure; the checker was fixed on it, so it is not blind.
• Engine is the version of the checking code that produced the number. The site runs a later engine than some rows; where that matters, the caveat says so.

Does the case exist, and is it the case you named?

The default check looks up every citation in the register by volume, reporter and page, then compares the case name. No language model is involved.

Is the quotation in the opinion?

The quotation check looks for the quoted words in the opinion text, allowing for the edits courts make: omitted citations, brackets, dropped words.

What a review looks like on real filed briefs

141 briefs and motions filed in 29 federal district courts from 2024 on, taken from the RECAP archive; 133 had a text layer. They hold 7,067 citations, 53 per brief. This is the set that tells you how long a review takes.

These runs predate the federal appellate backfill of 2026-09-20. A re-run after it resolved more citations: Westlaw and Lexis ids left unresolved went from 1,591 to 1,294.

What the default check finds in drafts written by language models

We cut 100 published opinions just before a paragraph that cites cases and asked three Gemini models to write the next paragraph "with full reporter citations". The court's real paragraph is the floor. This measures the drafts as much as the checker.

Measured 2026-09-19. The red-row column is the v0 service in default mode; the last two columns are the bench's classification of the same paragraphs with an earlier instrument, which is why the court's floor differs between them. The prompt was "continue this opinion", not "write a brief", and three Gemini models are not every model.

Public benchmark: LePhantomCite

LePhantomCite is a public benchmark from Princeton: Liu, Stammbach and Henderson, "Who Checks the Citations? Benchmarking Legal Hallucination Detection" (arXiv 2606.21155; the data is CC BY 4.0). It takes excerpts of real federal appellate briefs, puts one kind of error into a citation, and adds case holdings written by language models from Dahl et al. (2024). Nobody here built it, so it is the one set on this page we did not choose.

We report its eval split, the 390 excerpts the paper scores, run once on 2026-09-24 through the default check of engine 0.5.0, after the engine was frozen. The changes in 0.5.0 were tuned on the other split (910 excerpts) and on our filed-briefs control, never on these 390. No language model and no paid call. Caught means a red row on the labelled citation or quotation.

False flags. Of the 786 citations the benchmark labels correct, the default check flagged 21 red (18 of 695 before 0.5.0; the new rows are short forms). We read all 21. In our reading, 8 are real errors in the source briefs that the benchmark does not label as errors, such as a wrong volume or page, or a quotation the opinion does not contain. That leaves 13 of 786 (1.7%, 95% interval 1.0 to 2.8) where the checker itself was wrong, against 10 of 695 (1.4%) before. One reader, ours, did this reading against the register's text.

Beside the paper's systems. The table below uses the paper's own metric and its own scoring code. It is not a like-for-like race: the default check is code, it addresses three of the five classes (series that do not exist, name mismatches, altered quotations), and it does not read holdings or check a pin inside the opinion. The paper's agentic systems search CourtListener and the web and read the opinions, up to 30 steps per excerpt. That is where their recall comes from.

The paper rows are copied from its Table 2; it reports more systems. Its labels treat the source-brief errors above as correct (one is marked optional), so a flag on them counts against our precision. Counting the 7 that are among our predictions as right gives 89.5%, again on our reading.

Does the case say what you cite it for? (deep check, US)

The deep check is opt-in. A judgment model (jev) reads the ten passages of the opinion that best match your sentence and answers whether one states your claim, states the opposite, or neither, and who is speaking: the court, a party or a dissent. The report shows the passage.

The set: 920 held-out pairs from published opinions (the CLERC corpus). Each pair is a sentence that cites a case, and the whole cited opinion. 300 are the court's own citations, 300 point at a passage from another case on a similar subject, 120 at a random passage, and 200 keep the true passage but negate the claim.

• A different model build. These numbers were measured on jev-1.13.0 through TypeSafe. The site's deep check now calls a dated build of jev 1.13 through OpenRouter, and the US numbers have not been re-measured on that build.
• The label is strict. Some passages from another case state the same standard rule (standard of review, leave to amend), so part of the "wrong pairs marked as support" is the label and not the check.
• Courts citing courts. The truth set is published opinions. An advocate's paraphrase is harder, and we have no hand-labelled number for it yet.
• Who is speaking. On 120 paragraphs where a court reports a party's argument, v4 marked 2 as support; the word-overlap rule marked 118.
• Court-found misstatements. On 18 citations that courts found misstated, the support check of 2026-09-19 marked 1 as support at 0.8. Read by hand, that opinion does say what the filing claimed; the court's complaint was an invented quotation.

Does the decision say it? (deep check, Swiss)

No Swiss briefs are public, so the set is court text: a Federal Supreme Court sentence that cites a leading case with a considerando, and the whole cited BGE. 757 held-out pairs, 64% German, 30% French, 6% Italian. The wrong pairs are passages from a leading case of another part of the reports in the same language, random passages, and true passages with the claim negated.

• Not blind. This held-out set was run three times, and the truth set was redesigned after reading its errors. Treat these as dev numbers until a fresh set is run once.
• The error rate is a ceiling. All 14 confident errors at 0.8 were read: 12 are leading cases that do state the proposition (procedural formulas such as Art. 42, 95, 105 and 106 BGG appear in leading cases of every part). 2 are real errors.
• Not yet measured on filed Swiss briefs. The site runs a later engine than 0.4.2.

Does a suggested case state your sentence? (Find cases, US beta)

On Find cases you paste a sentence from your draft and get up to three cases. An input check first turns away text that states no rule of law. The search then ranks candidates by the sentences in which federal appellate courts and the Supreme Court cite them and by the cases' own text, and puts cases that bind the brief's court first. A judgment model (jev) reads the opinion of the court in each of the top 10, and a case is shown only when the model finds a passage that states the sentence, at confidence 0.9 or higher (0.8 until 29 September 2026; sets C and D below ran at 0.8). Cases that a later court overruled or reversed are left out.

The sets: 50 sentences each from briefs filed in federal district courts (the RECAP archive), none used before, each run once. Set G ran on the site itself with jev 1.14, the model build the site has run since 25 October 2026. Sets E and F ran on the site with jev 1.13, both at the 0.9 floor; set E decided the move to 0.9 under a rule written before it ran. Set D ran on the site at 0.8, and set C ran three days earlier on the machine that built the index. The sets come from cases filed in 2025; the briefs themselves are mostly from 2026. Each set is reported on its own, never pooled. Two readers read every shown case, with its passage between the paragraphs before and after, without seeing each other's label or the model's score.

• Who read. The readers were language models reading blind, two per case, not practising lawyers. On set E they agreed on 84 of 92 (the cases shown at either floor), and a third model read the other 8; on set F they agreed on all 65. On set D they agreed on 91 of 94, and a third model read the other 3; on set C they agreed on 72 of 73, and a third read the one disagreement.
• The shown case in set G that did not support its sentence. Skinner v. Switzer was shown for a sentence saying that notice pleading does not require pinning a claim to a precise legal theory. Skinner does say that, but in another paragraph; the paragraph shown is its discussion of the Rooker-Feldman doctrine. Both model builds showed it.
• The shown case in set E that did not support its sentence. Washington State Grange (2008) was shown for a sentence that argues the brief's own record ("These record-review principles are not disregarded just because Plaintiffs raise constitutional claims"). The input check should have turned that sentence away.
• The five in set F. Two are real: for a sentence about a "pure question of law" under 28 U.S.C. § 1292(b), the list showed two qualified-immunity cases about the collateral-order doctrine, with the right words and the wrong doctrine. Three came from our test, not the site: the tool that strips line numbers from the briefs turned "convert Rule 5 into a rigid specification" into "convert Rule into a rigid specification", and for that altered sentence the model matched cases rejecting "a rigid test". Sent as the brief wrote it, the sentence shows no case. The tool is fixed.
• The four shown cases in set D that did not support their sentence. Each time the model matched a passage that is about something else. A one-line paragraph of a California case ("We begin with the issue of duty") was shown for a Minnesota brief. Burlington Industries v. Ellerth, on a tangible employment action, was shown for Title VII's "materially adverse action". A footnote of Barnes v. Yahoo! on the motion-to-dismiss standard was shown for a Section 230 point. NCAA v. Board of Regents, on per se rules and the rule of reason under Section 1 of the Sherman Act, was shown for a Section 2 point.
• The one shown case in set C that did not support its sentence. For "These complaints, therefore, do not constitute protected activity and cannot form the basis of a retaliation claim" (a brief filed in the District of Minnesota), the list showed Johnson v. Portfolio Recovery Associates (E.D. Va. 2009). Its paragraph says the conduct was not in the EEOC charge and so cannot support the retaliation claim: a point about exhaustion, not about protected activity. The model matched the words of the conclusion. The sentence also applies the law to the brief's own facts, which the input check let through.
• Why it is a beta. Our bar before users was 80% supporting and none that does not support its sentence. Set G, on the model the site runs, reached 83% with 1 of 66 not supporting. At the 0.9 floor with jev 1.13, set E reached 84% with 1 of 81 not supporting, and set F 80% with 5 of 65 (2 of them real). Neither clears the bar, so the list stays a beta. At the old 0.8 floor, set D had 78% and 4 of 94.
• An empty list is not a finding. At the 0.9 floor, set E covered 29 of 50 sentences and set F 22 of 50. In set D, 33 of the 50 sentences got at least one supporting case; in set C, 23 of 50. In set C, of the other 27, the input check turned away 11, and for the rest no candidate passed the check. That does not mean no case states the sentence.
• What the site runs. The settings measured are the ones the site uses: the input check, the brief's court first, the top 10 candidates read, cases a later court overruled or reversed left out. The site adds the time limits above. A candidate the model has not read in time is not shown, and the page says how many. Sets E and F ran on the site itself; set C ran on the engine 0.7.0 candidate, whose suggestion code the site kept, adding the time limits and later the 0.9 floor.
• Federal district court briefs only. How the list does for briefs filed in state courts, or for a sentence a lawyer writes from scratch, is not measured.

Does the Swiss decision exist?

What these numbers do not tell you

• Whether a case is still good law. This is not a citator, and nothing here measures overruling, reversal or later limits. Use KeyCite or Shepard's.
• How the default check does on short-form citations ("Id. at 352", "Feist, 499 U.S. at 352") in real briefs. Since engine 0.5.0 it ties them to their case and checks their quotations, but only the LePhantomCite section measures that; the other tables predate it. A short form the text does not tie to a case gets no row, and the report counts it as not checked.
• Westlaw and Lexis ids are listed, not resolved. They are about a quarter of the citations in filed briefs, and 136 of the 345 court-found invented cases above were ids.
• The register holds published federal appellate opinions from 2018 for every circuit, but only 87.2% of the 2d Circuit's are matched, and unpublished appellate opinions are not held. A miss there is "cannot verify", not a flag. Coverage.
• How the deep check does on real briefs. Both support sets are courts citing courts.
• How the Swiss checks do on filed Swiss briefs.
• Whether a practising lawyer reads the US suggestions as our model readers did, and how they do for state-court briefs.
• That an unflagged citation is right. It passed the mechanical checks. Nobody has read it for you.
• How sure we are about each rate. We have not published confidence intervals yet; the sample size is beside every number so you can judge.
• Anyone else's view. These are our measurements, on sets we built. No one outside has reproduced them yet.

How the sets were built and kept

The court-found set comes from the sanctions decisions collected in Damien Charlotin's AI Hallucination Cases database: every finding that names a reporter citation, 528 citations in all. Ten decisions from the same database are laid out citation by citation on Replays: each citation the court listed, what the register holds there, and the default check's row today. The filed briefs come from the RECAP archive of PACER documents. The support sets come from published opinions in the CLERC corpus (built on the Caselaw Access Project) and from Swiss Federal Supreme Court decisions. All of these are public records.

Each set is split. We change the checker only while looking at the dev part; the held-out part is read once, at the end, by the script that scores it. The Swiss support set is the exception, and the table says so. Each engine change that can move a verdict is re-run against the court-found control. The held-out splits stay private, so they keep measuring something.

Test it yourself

Take five briefs you filed last year. They are public on PACER, so nothing privileged leaves your desk. Run them through the checker and count two things: the rows you would have wanted to see, and the rows that were noise. Your own count will tell you more than our tables can. See a sample report first, or read how to check a citation by hand.

ProofreadCompared with other checkers