TSR Desk · science · 31 October 2026, 01:00 UTC
DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code
- What
- DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents
- Who
- arxiv.org
- When
- 30 October 2026, 04:00 UTC
- Category
- Science
- Primary source
- https://arxiv.org/abs/2511.20709
- What is not known
- This brief does not claim independent replication. Claims that appear only on X and not in the primary source stay unknown.
Evaluation of 3 leading coding agents (Codex, OpenHands, and Claude Code) shows that iterative scaffolding is not a reliable remedy on specification-only tasks: results are strongly scaffold-dependent, with two of the three agents failing to outperform a comparable direct model. It comes from a paper posted to arXiv on 30 October 2026. Large language models (LLMs) and LLM-based coding agents are now used to generate code from natural-language specifications, yet ensuring such code is both functionally correct and secure remains a challenge. We present DualGauge, the first fully automated framework for jointly evaluating correctness and security of specification-only code generation, supported by DualGauge-Bench, a language-agnostic benchmark of 307 coding tasks each paired with functional and security tests derived from the same specification. Evaluating 10 representative LLMs across Python, C++, and JavaScript, we find that functional correctness substantially overestimates reliable code generation: even the strongest model remains below 15% joint security-functionality success in every language. Common model-side factors--scale, extended thinking, quantization, instruction tuning, and code specialization--do not reliably improve joint performance, suggesting secure-and-correct code generation does not simply emerge from stronger coding capability. A qualitative audit reveals that these failures concentrate at the output contract boundary and in guards that exist but are insufficient--patterns that only joint benchmarking reliably exposes.
Why it counts
Evaluation of 3 leading coding agents (Codex, OpenHands, and Claude Code) shows that iterative scaffolding is not a reliable remedy on specification-only tasks: results are strongly scaffold-dependent, with two of the three agents failing to outperform a comparable direct model. We present DualGauge, the first fully automated framework for jointly evaluating correctness and security of specification-only code generation, supported by DualGauge-Bench, a language-agnostic benchmark of 307 coding tasks each paired with functional and security tests derived from the same specification. Common model-side factors--scale, extended thinking, quantization, instruction tuning, and code specialization--do not reliably improve joint performance, suggesting secure-and-correct code generation does not simply emerge from stronger coding capability.
Sources
Primary source: primary source
What is not known
This brief does not claim independent replication. Claims that appear only on X and not in the primary source stay unknown.
No clip. The article still stands.