Reports & Studies

How five small language models argue

Findings from 2,500+ instrumented turns across 250 controlled debates and 300+ turns of public debate on Shastrarth

Five small language models — Gemma, Qwen, Llama, Mistral and Phi — argued the same 25 statements against each other in every pairing and both orders, entirely on a CPU-only home server, while an independent judge scored every debate blind. Every turn was measured: time, tokens, speed, CPU, memory and energy. The report sets out which model argues best, which is fastest, which holds its position, and what each one costs to run — with the method, every table and the caveats included.

• Gemma and Qwen argue best (3.39 and 3.32 of 5) and are statistically tied; Llama, Mistral and Phi form two tiers below
• Model size predicts neither quality nor speed — the 8B Gemma beats the 7.2B Mistral on every judged dimension
• Llama is the fastest by a wide margin: 25.9 s per turn, against Mistral's 59.3 s
• Gemma holds its assigned position eight times more firmly than Phi (hedging 0.034 against 0.287)
• Mistral uses 2.3× the energy of Llama per debate; all 250 debates together used 0.77 kWh
• Public visitors' debates rank the models in exactly the same order as the controlled benchmark

Anyone choosing a small model to run locally, and anyone curious how to measure one honestly

Deep research by a team of four small language models: measured, step by step

Findings from 15 measured Anusandhan research runs, 579 model calls and 2,300 claims, two controlled reader tests and two weeks of tuning

Anusandhan is a deep-research tool in which four small language models work as one team, each on the step it suits, with no GPU and no cloud model: they plan the searches, rank the results, read the pages, write a cited report and critique it, and code checks every quote and number against its source. This report covers 15 measured research runs on three topics: accuracy, speed, cost, the lessons it took to reach a steady state, and what it means for anyone deploying small models.

• 88.7% of 2,300 extracted claims passed code checks of quote and figures; a claim that fails never reaches the report
• The reading model decides accuracy: 84.7% verified with the 3B llama3.2, 94.6% with the 8B gemma4, under the same checks
• The bigger reader added only 16% to reading time and was cheaper per verified figure (67 s against 86 s)
• On pages under 500 words the 3B reader verified 64% of its claims and the 8B reader 99%: restraint beats volume
• The 4.7B critic's flags agreed with the mechanical checks only 19% of the time; the checks, not the critic, protect accuracy
• A deep run takes about two hours, 12 CPU-hours and ~55 Wh; reading is 49-70% of every run

Anyone building or choosing small language models for research, extraction or agent pipelines