Journal of Empirical Legal Studies· 2026Q1
Off‐The‐Shelf Large Language Models Are Unreliable Judges
- 3citations
- Q1SCImago
- 2026year
Short summary
Off-the-shelf large language models (LLMs) are unreliable legal interpreters, as their judgments are highly sensitive to prompt phrasing, output processing, and model choice, and they fail to accurately assess linguistic ordinary meaning.
AI-generated from the title and abstract; the full text is not read.
Key points
- LLM legal judgments vary significantly based on prompt phrasing, output processing, and model selection.
- Frontier LLMs do not reliably capture the ordinary meaning of legal language.
- Tuning base LLMs into assistant models can lead to unpredictable changes in ordinary-meaning judgments.
- Assistant model tuning can decrease alignment with human judgments on ordinary meaning.
AI-generated from the title and abstract; the full text is not read.
Abstract
ABSTRACT I conduct large‐scale empirical experiments to test the reliability of large language models (LLMs) as legal interpreters. Combining computational methods with the results of a new survey, I find that LLM judgments are highly sensitive to prompt phrasing, output processing methods, and choice of model. I also find that frontier LLMs do not accurately assess linguistic ordinary meaning, and I provide original evidence that tuning base models into assistant models shifts ordinary‐meaning judgments unpredictably and can worsen their alignment with human judgments. These findings undermine LLMs' credibility as legal interpreters and cast doubt on claims that LLMs elucidate ordinary meaning.
The authors' abstract, as published at the source. Journal of Empirical Legal Studies, 2026 · DOI ↗
Continue with a free account
Ask the paper: 3 free questions a day about this paper; save it, get its citation, new summaries every day for your field. Takeaways are Premium.
Continue free on the webSign in with Google or Apple; no card needed. You come back to this paper.
On your phone:
Field: Political Science and International Relations
Political Science and International RelationsSocial Sciences