Progress on provable inference for large language models
Anthropic’s FlashProve research prototype uses zero-knowledge proofs to check that tokens came from a committed model without revealing its weights. The team reports proving 4-bit Llama 4 Maverick at roughly 110 tokens/s on one H100, but with 70–95× the GPU time of ordinary inference and about a 19 MB proof per 1,024-token request.
The proof covers integer inference with greedy decoding and contexts up to 1,024 tokens; it does not establish that sampled or floating-point outputs match. Lean formalizes the protocol and a slower verifier, but the fast C++ verifier has no formal guarantee. A VerInf contributor in the HN thread argues their prior work received insufficient attribution; another reader notes that the article does cite VerInf.