Post

Test/verification technique effectiveness with Astra

Dan Luu compares agent testing and verification techniques on Zstd, IMAP, and a harder Pandoc implementation task. The newer model largely saturated the Zstd holdout test, while several techniques appeared to improve results on Pandoc relative to the default.

The author cautions against ranking techniques from this: Pandoc had only 25 runs per condition, one large task, wide uncertainty, and no run reached full correctness. The evaluation suggests testing behavior has changed, not that a particular technique is conclusively best.