
Associate Prof @TAMULawSchool JD @Harvard_Law | PhD @MITBrainandCog | Law, language, cognition, AI
Fort Worth, TX
Joined June 2020
- Tweets187
- Following406
- Followers710
- Likes1.1K
Pinned Tweet
Can AI ace law exams?
Last year, OpenAI announced GPT-4 got 90th percentile on the bar.
Here we (a) refute 90th percentile claim; (b) replicate/extend recent work on GPT capabilities; (c) discuss implications for law profession.
Now open, AI&Law: link.springer.com/article/10…
1/
This paper finds that frontier AI models correctly format Bluebook citations only 42.6% of the time on average. The best results come from combining AI with a deterministic rule engine rather than relying on the model alone. spkl.io/60117Pymc
#LegalTech #AI
Eric Martínez retweeted
Don't ask AI to Bluebook your citations until you read this study. Heck, maybe don't even ask?
Frontier models produced correct citations just 42.6% of the time.
AI can solve open math problems and find security flaws.
But can it reliably format a legal citation?
New paper w/ Matthew Dahl: across frontier models, the answer is no. We then build a system that
does much better.
papers.ssrn.com/sol3/papers.…
THREAD 1/
Eric Martínez retweeted
Disappointing but important.
"We develop a new benchmark of 2,058 Bluebook queries and show that, on average, frontier language models produce a fully compliant legal citation only 42.6% of the time in a zero-shot setting."
Excited to dive more into this...
Eric Martínez retweeted
This is an extremely interesting and well-written paper with a very surprising result. Makes me wonder whether we've really internalized how important jaggedness is and how long it will stick around.
Also, I will pay any bluebook-savvy law student who wants to earn some extra money $20 an hour to go through the methodology and experimental design with a fine toothed comb and come up with persnickety little criticisms, not because I think you're likely to find anything that discredits the validity of the central finding but because it will really piss Eric off and that will be hilarious. DM me if interested. Maybe start with the "models can't display small caps" thing, they seem very defensive about small caps in the footnotes. I smell weakness.
AI can solve open math problems and find security flaws.
But can it reliably format a legal citation?
New paper w/ Matthew Dahl: across frontier models, the answer is no. We then build a system that
does much better.
papers.ssrn.com/sol3/papers.…
THREAD 1/
AI can solve open math problems and find security flaws.
But can it reliably format a legal citation?
New paper w/ Matthew Dahl: across frontier models, the answer is no. We then build a system that
does much better.
papers.ssrn.com/sol3/papers.…
THREAD 1/
Second, for “law-following” AI.
The Bluebook is intricate, but less ambiguous than statutes or common-law standards. If models
cannot reliably follow its explicit rules, confidence in autonomous compliance with more contested +
dynamic bodies of law should be tempered.
14/
Third, for the design of legal AI.
We argue for a “formal turn”: use LLMs for language understanding, and auditable code for exact rule
execution. Start with law’s formal core before its penumbra.
Paper: papers.ssrn.com/sol3/papers.…
15/
Eric Martínez retweeted
Now out in PNAS!
Link to the paper: pnas.org/doi/10.1073/pnas.23…
News story by MIT News: news.mit.edu/2025/esperanto-…
What is a language?
Some think math, logic and music are like natural languages, but the brain doesn't treat them as such.
What about constructed languages?
Here is a preprint on what we discovered in @ev_fedorenko ’s lab: tinyurl.com/2p8f8vnp









