Ponytail corrects its own coding-agent benchmark
Ponytail’s useful contribution is less its “write the minimum” rules than the public correction of an inflated benchmark. The original claim of 80–94% less code used a single-shot, chatty baseline; after a contributor challenge, the author reran twelve feature tasks as real headless Claude Code sessions against the same FastAPI/React repository and measured git diff.
The revised README reports about 54% less code on average, roughly 20% lower cost, and 27% faster execution. The effect is large where an agent over-builds and near zero where the existing solution is already minimal. A safety check found Ponytail preserved validation and security guards, while a bare “YAGNI + one-liners” prompt dropped a path-traversal guard once.
The broader lesson is about evaluation: skills and prompt frameworks need reproducible agentic baselines, task-level diffs, and adversarial safety checks. The benchmark issue records both the methodological criticism and the author’s revised results, including caveats about small sample size and model dependence.