Post

Introducing FrontierHarness Eval v1.1

Runta and Baseten report that harness cost rankings changed substantially across models in a benchmark of 27,280 runs: 22 configurations, five models, 62 terminal tasks and four attempts per combination.

This is a vendor-run evaluation on a selected task set, not a universal ranking. On Hacker News, commenters questioned how tasks were selected and whether the original headline about spending on failed runs was meaningful without a baseline; one linked a larger analysis that reached a different conclusion about model–harness interactions.