Cost, Verification, and the Standard for Judging Peer Review: A Philosophical Critique

Abstract

Peer review’s reliability on checkable dimensions has been low since the agreement studies of the 1970s and 1980s. The remedies have been known almost as long: explicit criteria, complete rather than sampled checking, and plural independent evaluation. None has been implemented at scale. This paper asks three questions. Is that unreliability a property of human judgment or of procedure? What prevented adoption of the known remedies? And what must an evaluative arrangement satisfy in order to be legitimate? The method is conceptual: the paper synthesizes the empirical literature on peer review with work in social epistemology, procedural legitimacy, and contestable artificial intelligence. It argues for three claims. Reliability on checkable dimensions varies substantially with procedural design. Cost is a necessary but insufficient explanation of non-adoption. And a verdict is legitimate to the degree that it is contestable by the person it burdens; this procedural standard is defended against the epistemic consequentialism that the adjacent literature assumes. On that standard, cheap machine verification changes less than current enthusiasm suggests. Fully specifiable operations, such as statistical recomputation and reference resolution, reach a reliability that human review never approached. Partially specifiable operations gain coverage rather than accuracy. The significance judgment gains nothing. The contribution is a defended standard for judging evaluative institutions, not a case for machine review. The governance and adversarial-robustness conditions that such an arrangement requires are specified, but they are not claimed to be met.

The authors' abstract, as published at the source. Philosophies, 2026 · DOI ↗