· 2018
Evaluating Fuzz Testing
- 609citations
- 2018year
Short summary
A survey of 32 fuzz testing papers revealed significant flaws in the experimental evaluations of new fuzzing techniques, with problems found in every study considered.
AI-generated from the title and abstract; the full text is not read.
Key points
- Every experimental evaluation in a survey of 32 fuzz testing papers contained problems.
- An independent experimental evaluation confirmed that common flaws lead to misleading results.
- Existing fuzz testing research may overstate the effectiveness of new techniques due to poor evaluation practices.
- Guidelines are proposed to enhance the reliability of future fuzz testing experiments.
AI-generated from the title and abstract; the full text is not read.
Abstract
Fuzz testing has enjoyed great success at discovering security critical bugs in real software. Recently, researchers have devoted significant effort to devising new fuzzing techniques, strategies, and algorithms. Such new ideas are primarily evaluated experimentally so an important question is: What experimental setup is needed to produce trustworthy results? We surveyed the recent research literature and assessed the experimental evaluations carried out by 32 fuzzing papers. We found problems in every evaluation we considered. We then performed our own extensive experimental evaluation using an existing fuzzer. Our results showed that the general problems we found in existing experimental evaluations can indeed translate to actual wrong or misleading assessments. We conclude with some guidelines that we hope will help improve experimental evaluations of fuzz testing algorithms, making reported results more robust.
The authors' abstract, as published at the source., 2018 · DOI ↗
Continue with a free account
Ask the paper: 3 free questions a day about this paper; save it, get its citation, new summaries every day for your field. Takeaways are Premium.
Continue free on the webSign in with Google or Apple; no card needed. You come back to this paper.
On your phone:
Field: Software
SoftwareComputer Science