Training on the Standard, Judged by the Standard: The Benchmarking Paradox and Validation’s Impossibility in Human-LLM Co-Production

Abstract

I built validation frameworks for Large Language Model (LLM) use in qualitative research and discovered that the more rigorously I validated, the more impossible validation became. This article theorises structural impossibility as the benchmarking paradox : LLMs train on human-generated text, then researchers validate outputs against human interpretation, which was the training source. Human interpretation functions simultaneously as training data, evaluation standard and outcome judged. Through diffractive reading of an empirical encounter where ChatGPT analysed policy texts, including vernacular translations it generated, I develop three theoretical coordinates: mimetic recursion (exposing circular validation), function/meaning incommensurability (revealing interpretation’s always-already hybrid nature) and epistemic authority redistribution (showing how computational infrastructure stratifies whose knowledge production counts). Empirically, GPT-5’s sophisticated meta-analysis of its own vernacular renderings exposes alignment priors determining computational legibility: whose voices register as analysable, which linguistic forms count as legitimate and what inquiries remain thinkable. The article argues that co-production through LLMs does not democratise qualitative research – it risks amplifying existing hierarchies while obscuring their operation through technical discourse. Rather than pursuing impossible validation, qualitative research should ask whose interpretive labour trained the pattern, contest whose interests human-LLM assemblages serve and demand that computational mediation generates epistemic and linguistic justice rather than reproducing the hierarchies it obscures.

The authors' abstract, as published at the source. Qualitative Inquiry, 2026 · DOI ↗