Auditable large language model scoring of annual-report evidence packs: human calibration, baselines, and reproducibility checks

Abstract

Large language models (LLMs) can convert long documents into structured research variables. Auditability becomes difficult when evidence selection and model scoring are not stored. We study an inspectable scoring workflow that converts archived annual-report Evidence Packs into a five-dimension digital-transformation disclosure score. The design combines report selection, PDF-to-text normalization, bounded paragraph retrieval, and DeepSeek rubric scoring through a strict JavaScript Object Notation (JSON) interface; the original canonical prompt/run binding is not available. We evaluate three measurement questions and one secondary utility check: reproducible Evidence Pack construction, LLM–human agreement on shared Evidence Packs, ordinal error/confusion diagnostics, and non-causal panel-use stress tests. The evaluation panel contains 476 firm-year observations from 64 firms for 2013–2024. The post-validation score shows high internal coherence for the additive rubric (Cronbach’s α = 0.893; first principal component explains 0.704 of standardized dimensional variance), substantial within-firm variation (mean within-firm standard deviation (SD) 2.901; 92.19% non-zero), and strong LLM–human agreement on shared Evidence Packs (total-score Spearman 0.910, Pearson 0.921, mean absolute error (MAE) 1.273; dimension-level weighted kappa 0.665–0.823). Human–Human total-score reliability, measured by the intraclass correlation coefficient (ICC), is ICC(2,1) = 0.811. Baseline comparisons show lower error than frozen lexicon-count and firm-grouped term frequency–inverse document frequency (TF-IDF) Ridge alternatives (MAE = 1.273 vs . 6.387 and 2.737). A 96-record rule-based retrieval audit shows high mean dimension coverage (0.936) but incomplete heuristic paragraph recall@30 (micro recall = 0.419); four audited zero-hit packs are false empty (4.17%). The main reliability claim is therefore bounded to shared archived evidence. Calibration, error-distribution, and confusion-grid analyses indicate that disagreement is concentrated near adjacent rubric levels, with extreme mismatches rare, supporting score ordering conditional on shared archived Evidence Packs. Secondary fixed-effects and event-time checks show only that the score can enter panel workflows; pre-entry differences and year-effect sensitivity require non-causal interpretation. The main contribution is an evidence-bounded audit template rather than an exact endpoint-rerun claim.

The authors' abstract, as published at the source. PeerJ Computer Science, 2026 · DOI ↗