The simple version
Most AI tests ask a model to answer one question at a time. That tells us something, but it does not tell us whether the model can do the kind of work a lawyer would actually hand to a colleague.
Legal work is messier. A lawyer may need to read several files, separate important facts from distractions, identify the right rule, use real legal authority, and produce something another person can review.
EvidenceBench v4 tests both abilities. Half of the score comes from short evidence-law questions. The other half comes from working through simulated case files.
Overall score = 50% short questions + 50% case-file work.
What did the AI models have to do?
Part one: make evidence rulings
The first part contains 240 short legal problems. For each one, the model had to:
- decide whether the evidence should be admitted, excluded, limited, or considered later;
- identify the legal issue;
- cite the rule or case supporting its answer;
- connect that answer to the important facts; and
- say how confident it was.
The models could not search the internet or use outside tools for this part. We call it the Doctrine test on the results page. In ordinary language, it is the short-question test.
Part two: work through a case file
The second part contains 60 simulated legal assignments. Each assignment came with six documents. Some information mattered; some was there to distract the model.
The model had to search the files, spot the evidence problems, support its conclusions with law and facts, and create a usable written work product. We call this the Matter test. In ordinary language, it is the case-file test.
Every case-file assignment had several important requirements. A model received a “fully completed” result only if it met all of them. Missing one critical issue meant the case was not counted as fully resolved. We call this must-spot-all grading. Partial credit still counted toward the separate case-file score.
The results
The overall score is out of 100. It is a blend of the two parts of the test. A score of 71.8 does not mean that 71.8% of case files were completed perfectly.
| Model | Overall score | Short questions | Case-file work | Fully completed case files |
|---|---|---|---|---|
| Grok 4.5 | 71.8 | 74.5% | 69.0% | 0.0% |
| Gemini 3.1 Pro Preview | 63.8 | 65.1% | 62.6% | 0.0% |
| GPT-5.6 Sol | 61.8 | 61.3% | 62.3% | 0.0% |
| Claude Opus 5 | 56.2 | 60.0% | 52.3% | 0.0% |
Grok 4.5 finished first with 71.8 points. It also led both parts of the test: 74.5% on the short evidence questions and 69.0% on the case-file work.
The more important result is that no model fully completed even one case-file assignment under the must-spot-all rule. This does not mean the models did nothing useful. Most created nearly all the requested files, and much of their work earned partial credit. It means every assignment missed at least one requirement that this version of the benchmark treated as critical.
Across all four runs, the models produced 297 apparently invented legal references. Another 41 questions or assignments ended with failed or unusable output. We kept all of those failures in the scores instead of quietly deleting them after the test.
Is a 0% full-completion rate expected?
It is concerning. All-or-nothing grading naturally produces much lower numbers than partial-credit grading, because one missed requirement is enough to fail the whole assignment. Harvey's benchmark shows the same general pattern: models can satisfy most individual requirements while fully resolving far fewer tasks.
But zero across every model is still a warning sign, not a result to wave away. It could mean the models consistently missed something important. It could also mean our completion rule is too strict, or that parts of the unreviewed answer key need correction. Until independent legal reviewers examine those requirements, the safest conclusion is that the models produced useful partial work but did not demonstrate dependable end-to-end completion on this test.
The biggest warning sign was citation quality
The models were often good at producing polished-looking work. They were much less dependable at connecting that work to the right legal authority.
On the case-file assignments, scores for correctly connecting conclusions to the expected rules and cases ranged from only 25.3% to 31.5%. In other words, a document could look finished while the legal support underneath it was weak, missing, or wrong.
That distinction matters. A fluent answer can save a lawyer time. A confident answer supported by invented or irrelevant law can create more work and more risk.
What we borrowed from Harvey's benchmark
Harvey, a legal AI company, recently released a much larger Legal Agent Benchmark. It includes more than 1,200 tasks across 24 areas of law. Its models receive instructions that resemble an assignment from a supervising lawyer, read a closed set of files, and produce finished work.
That benchmark helped make an important point: legal AI should be tested on complete assignments, not just trivia-style questions.
EvidenceBench v4 follows that idea in several ways. It asks models to read files, deal with distracting information, create work products, and satisfy several requirements at once.
We also adopted LAB's must-spot-all distinction. LAB reports how many individual requirements a model satisfied, but it counts the whole task as resolved only when every requirement passes. This reveals a gap that an average score can hide: a model may do most of an assignment correctly while still missing the one issue that makes the work unsafe or incomplete.
EvidenceBench shows both views for the same reason. The case-file score gives partial credit for useful work. The “fully completed case files” column asks the harder question: did the model catch and complete everything important?
How EvidenceBench is different
Harvey's benchmark covers many areas of law. EvidenceBench focuses only on evidence law. That narrower scope lets us look more closely at specific problems: Was the ruling right? Did the model spot the issue? Did it use a real authority? Did it connect the law to the right facts?
The two benchmarks also grade work differently. Harvey uses AI models to judge the quality of written work against detailed instructions. EvidenceBench uses fixed scoring rules wherever possible. That makes EvidenceBench less flexible, but it also means the headline score does not depend on another AI model's opinion.
Keeping the short-question test also helps explain why a model struggled with a case file. The problem might be legal knowledge, issue spotting, citations, factual accuracy, or simply failing to finish the assignment.
What happens next
The next priority is independent review. Evidence lawyers, litigators, judges, academics, and experienced legal researchers can volunteer to review a small EvidenceBench packet. The public form collects interest and qualifications. The hidden answer key is shared only through the controlled review process.
Questions can also be sent to dylan@examp.com.
You can explore the interactive EvidenceBench v4 results, read about the earlier version of EvidenceBench, or inspect the public evaluator on GitHub.