smolmaxxing: running LLMs on small devices
This experiment explores running quantized language models on old and constrained hardware using custom kernels, flash streaming and on-demand mixture-of-experts paging.
On a Xiaomi TV box with about 1 GB usable RAM, the author reports 7.2 tokens/s for an 8B MoE model and 5.1 tokens/s for a 16B MoE model, with experts paged from flash. The repository includes experiment logs and patched inference code; these are author-reported, device- and model-specific measurements, not proof of general model quality.
HN readers asked whether tiny models are useful in practice; the author pointed to narrow fine-tuned tasks and described the work as an exploration of the smallest hardware that can provide usable inference.