A legal answer is only as useful as its support
An AI system can land on the right evidence answer for the wrong reason. It can also identify a plausible rule while citing a provision that does not apply, omitting the authority that matters, or inventing a subsection altogether. For lawyers and law students, those failures are not cosmetic. They make an answer harder to verify and less useful in the work that follows.
That is why Objection Academy created EvidenceBench: a public, reproducible benchmark for evidence-law reasoning and citation reliability. It is an educational public good, not legal advice, legal research, or an endorsement of any model.
What EvidenceBench measures
EvidenceBench gives each model the same closed-book multiple-choice evidence questions and requires structured output: a selected answer, a concise explanation, and normalized Federal Rules of Evidence or case citations. The evaluator measures more than whether the selected option matches the gold answer.
- Answer accuracy requires the exact gold choice.
- Citation precision asks whether cited provisions are accepted by the attorney annotation.
- Citation recall asks whether the required citation groups are satisfied.
- Citation F1 combines precision and recall, with missing citations scoring zero.
- Hallucination rate tracks citations to provisions or subsections that do not exist.
- Unsupported-citation rate tracks valid provisions that do not support the annotated answer.
The overall score is deliberately simple: 70% answer accuracy and 30% citation F1. That weighting recognizes that a model must first reach the right evidence conclusion, while still making citation reliability a material part of the result.
The v3 benchmark: rules, cases, and current sources
The current official holdout contains 151 sealed questions. Ninety-six test core Federal Rules of Evidence concepts. Fifty-five test caselaw across federal and state authority, civil and criminal proceedings, widely cited and less familiar decisions, forensic-evidence questions, and recent authority.
The caselaw expansion includes 33 unique, citable published decisions from the DOJ/NIJ Post-PCAST dataset cataloged by Data.gov, plus six published 2026 decisions verified against official court, GovInfo, or New York Official Reports sources. The public development set contains 24 reviewable questions; the official holdout, its prompts, and item-level model outputs remain sealed to preserve the benchmark.
EvidenceBench freezes Federal Rules authority validation at the rules effective December 1, 2025. The pending Rule 801(d)(1)(A) amendment is not scored until its scheduled December 1, 2026 effective date.
What the first snapshot shows
The first v3 snapshot ran six major model families once per sealed question using the same prompt, no browsing or other tools, a fixed output limit, provider-default reasoning mode, and retries only for transport failures. Exact model routes, prompt and dataset hashes, token usage, and costs are published with the snapshot.
Those ranks are a point-in-time measurement, not a declaration of a universal winner. The model gaps also become more informative when the results are separated by source and task. On the published 2026 caselaw tranche, for example, several models selected answers well but did not supply the accepted current authority citation. The benchmark exposes that distinction instead of allowing a correct choice to conceal a weak citation.
Why the public record matters
Legal-model evaluation often asks the public to trust a private test set, opaque rubric, or changing provider label. EvidenceBench takes a different approach. The public repository includes the development questions, attorney annotations, prompt template, model manifests, deterministic scorer, aggregate holdout results, and the generated website payload. The sealed holdout is represented by a version, count, category distribution, and SHA-256 commitment—enough to audit the protocol without compromising future runs.
That design gives lawyers, educators, developers, and model providers a concrete way to inspect the benchmark's choices and challenge them. It also makes future comparisons meaningful: quarterly snapshots and interim runs for major releases can be compared against a versioned dataset and published model routes.
How to use the results
EvidenceBench answers the broad question lawyers and legal teams actually ask: which frontier model is currently the strongest evidence-law assistant? It also helps with the decisions behind that question:
- Which model is most dependable for a first pass at spotting an evidence issue?
- Which model is most likely to give an answer that comes with usable legal support, rather than a confident but uncheckable citation?
- Which models stay useful when the task moves from black-letter Federal Rules questions to federal and state cases?
- Is a newly released model meaningfully better for this work, or just marketed as better?
- Where should a lawyer slow down and verify the authority before relying on the answer?
The benchmark does not tell a lawyer which model can replace legal judgment or make litigation decisions. It does not establish whether an authority controls a live matter, whether a jurisdiction follows the Federal Rules, or whether a case remains good law.
Use the benchmark as a transparent signal about model behavior. For real legal work, verify the authority, check the local rules and governing jurisdiction, and exercise independent professional judgment.
Explore the live EvidenceBench results and inspect the public evaluator repository.