Our error rates,
one check at a time

How to read the tables

Each check gets its own table. We do not add them up into one score, because a missed invented case and a real case flagged as missing cost you different things. The false-flag number sits next to the catch number wherever we have both.

• Flagged means a red row: "check this". False flag means a red row on a citation that was fine.
• Confirmed (deep check) means the check found a passage that states the claim. Marked as support on a wrong pair means it did that when the passage does not support the claim. That second number is the one to watch.
• Dev: we changed the checker while looking at these results. Held out: the set was put aside and run once, at the end. Control: a set built to test one kind of failure; the checker was fixed on it, so it is not blind.
• Engine is the version of the checking code that produced the number. The site runs a later engine than some rows; where that matters, the caveat says so.

Does the case exist, and is it the case you named?

The default check looks up every citation in the register by volume, reporter and page, then compares the case name. No language model is involved.

Is the quotation in the opinion?

The quotation check looks for the quoted words in the opinion text, allowing for the edits courts make: omitted citations, brackets, dropped words.

What a review looks like on real filed briefs

141 briefs and motions filed in 29 federal district courts from 2024 on, taken from the RECAP archive; 133 had a text layer. They hold 7,067 citations, 53 per brief. This is the set that tells you how long a review takes.

These runs predate the federal appellate backfill of 2026-09-20. A re-run after it resolved more citations: Westlaw and Lexis ids left unresolved went from 1,591 to 1,294.

What the default check finds in drafts written by language models

We cut 100 published opinions just before a paragraph that cites cases and asked three Gemini models to write the next paragraph "with full reporter citations". The court's real paragraph is the floor. This measures the drafts as much as the checker.

Measured 2026-09-19. The red-row column is the v0 service in default mode; the last two columns are the bench's classification of the same paragraphs with an earlier instrument, which is why the court's floor differs between them. The prompt was "continue this opinion", not "write a brief", and three Gemini models are not every model.

Public benchmark: LePhantomCite

LePhantomCite is a public benchmark from Princeton: Liu, Stammbach and Henderson, "Who Checks the Citations? Benchmarking Legal Hallucination Detection" (arXiv 2606.21155; the data is CC BY 4.0). It takes excerpts of real federal appellate briefs, puts one kind of error into a citation, and adds case holdings written by language models from Dahl et al. (2024). Nobody here built it, so it is the one set on this page we did not choose.

We report its eval split, the 390 excerpts the paper scores, run once on 2026-09-24 through the default check of engine 0.5.0, after the engine was frozen. The changes in 0.5.0 were tuned on the other split (910 excerpts) and on our filed-briefs control, never on these 390. No language model and no paid call. Caught means a red row on the labelled citation or quotation.

False flags. Of the 786 citations the benchmark labels correct, the default check flagged 21 red (18 of 695 before 0.5.0; the new rows are short forms). We read all 21. In our reading, 8 are real errors in the source briefs that the benchmark does not label as errors, such as a wrong volume or page, or a quotation the opinion does not contain. That leaves 13 of 786 (1.7%, 95% interval 1.0 to 2.8) where the checker itself was wrong, against 10 of 695 (1.4%) before. One reader, ours, did this reading against the register's text.

Beside the paper's systems. The table below uses the paper's own metric and its own scoring code. It is not a like-for-like race: the default check is code, it addresses three of the five classes (series that do not exist, name mismatches, altered quotations), and it does not read holdings or check a pin inside the opinion. The paper's agentic systems search CourtListener and the web and read the opinions, up to 30 steps per excerpt. That is where their recall comes from.

The paper rows are copied from its Table 2; it reports more systems. Its labels treat the source-brief errors above as correct (one is marked optional), so a flag on them counts against our precision. Counting the 7 that are among our predictions as right gives 89.5%, again on our reading.

Does the case say what you cite it for? (deep check, US)

The deep check is opt-in. A judgment model (jev) reads the ten passages of the opinion that best match your sentence and answers whether one states your claim, states the opposite, or neither, and who is speaking: the court, a party or a dissent. The report shows the passage.

The set: 920 held-out pairs from published opinions (the CLERC corpus). Each pair is a sentence that cites a case, and the whole cited opinion. 300 are the court's own citations, 300 point at a passage from another case on a similar subject, 120 at a random passage, and 200 keep the true passage but negate the claim.

• A different model build. These numbers were measured on jev-1.13.0 through TypeSafe. The site's deep check now calls a dated build of jev 1.13 through OpenRouter, and the US numbers have not been re-measured on that build.
• The label is strict. Some passages from another case state the same standard rule (standard of review, leave to amend), so part of the "wrong pairs marked as support" is the label and not the check.
• Courts citing courts. The truth set is published opinions. An advocate's paraphrase is harder, and we have no hand-labelled number for it yet.
• Who is speaking. On 120 paragraphs where a court reports a party's argument, v4 marked 2 as support; the word-overlap rule marked 118.
• Court-found misstatements. On 18 citations that courts found misstated, the support check of 2026-09-19 marked 1 as support at 0.8. Read by hand, that opinion does say what the filing claimed; the court's complaint was an invented quotation.

Does the decision say it? (deep check, Swiss)

No Swiss briefs are public, so the set is court text: a Federal Supreme Court sentence that cites a leading case with a considerando, and the whole cited BGE. 757 held-out pairs, 64% German, 30% French, 6% Italian. The wrong pairs are passages from a leading case of another part of the reports in the same language, random passages, and true passages with the claim negated.

• Not blind. This held-out set was run three times, and the truth set was redesigned after reading its errors. Treat these as dev numbers until a fresh set is run once.
• The error rate is a ceiling. All 14 confident errors at 0.8 were read: 12 are leading cases that do state the proposition (procedural formulas such as Art. 42, 95, 105 and 106 BGG appear in leading cases of every part). 2 are real errors.
• Not yet measured on filed Swiss briefs. The site runs a later engine than 0.4.2.

Does the Swiss decision exist?

What these numbers do not tell you

• Whether a case is still good law. This is not a citator, and nothing here measures overruling, reversal or later limits. Use KeyCite or Shepard's.
• How the default check does on short-form citations ("Id. at 352", "Feist, 499 U.S. at 352") in real briefs. Since engine 0.5.0 it ties them to their case and checks their quotations, but only the LePhantomCite section measures that; the other tables predate it. A short form the text does not tie to a case gets no row, and the report counts it as not checked.
• Westlaw and Lexis ids are listed, not resolved. They are about a quarter of the citations in filed briefs, and 136 of the 345 court-found invented cases above were ids.
• The register holds published federal appellate opinions from 2018 for every circuit, but only 87.2% of the 2d Circuit's are matched, and unpublished appellate opinions are not held. A miss there is "cannot verify", not a flag. Coverage.
• How the deep check does on real briefs. Both support sets are courts citing courts.
• How the Swiss checks do on filed Swiss briefs.
• That an unflagged citation is right. It passed the mechanical checks. Nobody has read it for you.
• How sure we are about each rate. We have not published confidence intervals yet; the sample size is beside every number so you can judge.
• Anyone else's view. These are our measurements, on sets we built. No one outside has reproduced them yet.

How the sets were built and kept

The court-found set comes from the sanctions decisions collected in Damien Charlotin's AI Hallucination Cases database: every finding that names a reporter citation, 528 citations in all. Ten decisions from the same database are laid out citation by citation on Replays: each citation the court listed, what the register holds there, and the default check's row today. The filed briefs come from the RECAP archive of PACER documents. The support sets come from published opinions in the CLERC corpus (built on the Caselaw Access Project) and from Swiss Federal Supreme Court decisions. All of these are public records.

Each set is split. We change the checker only while looking at the dev part; the held-out part is read once, at the end, by the script that scores it. The Swiss support set is the exception, and the table says so. Each engine change that can move a verdict is re-run against the court-found control. The held-out splits stay private, so they keep measuring something.

Test it yourself

Take five briefs you filed last year. They are public on PACER, so nothing privileged leaves your desk. Run them through the checker and count two things: the rows you would have wanted to see, and the rows that were noise. Your own count will tell you more than our tables can. See a sample report first, or read how to check a citation by hand.

ProofreadCompared with other checkers