ACM Transactions on Software Engineering and Methodology· 2026Q1
Beyond Accuracy: Behavioral Dynamics of Agentic Multi-Hunk Repair
- 0citations
- Q1SCImago
- 2026year
Short summary
Four LLM coding agents (Claude Code, Codex, Gemini-cli, Qwen Code) evaluated on 404 multi-hunk bugs show significant variation in localization (75.3% success for Codex) and repair accuracy (92.82% for Claude Code), with accuracy declining with bug complexity and dispersion.
AI-generated from the title and abstract; the full text is not read.
Key points
- Codex achieved the highest localization success rate (75.3%) on multi-hunk bugs, while Qwen Code had the lowest (40.4%).
- Claude Code demonstrated the highest repair accuracy (92.82%), with accuracy decreasing as bug dispersion and complexity increased.
- High-performing agents like Claude Code and Codex showed positive average regression reduction, unlike lower-performing agents that often introduced new failures.
- Failed repairs by agents consumed 33%-440% more input tokens and took 35%-330% longer to execute than successful ones.
- The Maple tool improved Gemini-cli's repair accuracy by ~21% by enhancing localization through repository-level context.
AI-generated from the title and abstract; the full text is not read.
Abstract
Automated program repair has traditionally focused on single-hunk defects, overlooking multi-hunk bugs that are prevalent in real-world systems. Repairing these bugs requires coordinated edits across multiple, disjoint code regions, posing substantially greater challenges. We present the first systematic study of LLM-driven coding agents ( Claude Code, Codex, Gemini-cli , and Qwen Code ) on this task. We evaluate these four state-of-the-art agents on 404 multi-hunk bugs from the PolyHunk dataset, yielding 1,616 repair trajectories for large-scale behavioral analysis. We employ fine-grained metrics to assess localization, repair accuracy, regression behavior, and operational dynamics across agents. We find that localization capability varies substantially, with Codex achieving the highest success rate (75.3%) and Qwen Code the lowest (40.4%). Repair accuracy also differs widely, ranging from 26.98% ( Qwen Code ) to 92.82% ( Claude Code ), and consistently declines with increasing bug dispersion and complexity (hunk divergence and spatial proximity). High-performing agents (Claude Code and Codex) demonstrate superior semantic consistency, achieving positive average regression reduction, whereas lower-performing agents often introduce new test failures. Notably, agents do not fail fast ; failed repairs consume substantially more resources (33%–440% more input tokens) and require longer execution time (35%–330%). Additionally, we developed Maple to provide agents with repository-level context. Empirical results show that Maple improves repair accuracy of Gemini-cli by ∼21% through enhanced localization. By analyzing fine-grained metrics and trajectory-level analysis, this study moves beyond accuracy to explain how coding agents localize, reason, and act during multi-hunk repair. Our findings underscore the impact of bug divergence and spatial proximity on multi-hunk repair success for coding agents.
The authors' abstract, as published at the source. ACM Transactions on Software Engineering and Methodology, 2026 · DOI ↗
Continue with a free account
Ask the paper: 3 free questions a day about this paper; save it, get its citation, new summaries every day for your field. Takeaways are Premium.
Continue free on the webSign in with Google or Apple; no card needed. You come back to this paper.
On your phone:
Field: Software
SoftwareComputer Science