How Castform + Neon Beats Frontier Models on Price and Efficiency

Castform post-trained a 4B open model to retrieve answers from a company corpus as accurately as GPT-5.6 Sol on its test, while claiming roughly 100× lower inference cost. The recipe turns existing documents into synthetic questions and answers, trains an agent to use hybrid BM25/vector search, and scores retrieval, citation, and final-answer correctness in the loop.

The useful idea is less the headline benchmark than the decomposition: retrieval quality is a model problem as well as a database problem, and proprietary documentation can supply training tasks without a hand-built labeled dataset. Neon provides the Postgres/Lakebase Search environment and branching/autoscaling for parallel rollouts; Castform supplies the post-training loop.

The caveats matter. The evaluation uses a GitLab handbook corpus, not a broad independent benchmark; the authors acknowledge data drift and say training cost was under $200 for the experiment. HN commenters also asked for total-cost break-even analysis, larger and messier corpora, and comparisons with cheaper models such as DeepSeek Flash.

• Hacker News discussion
• Open-source benchmark example