The number is never the story
Every model launch arrives with a benchmark number bigger than the last one. Engineers who build on these systems need to know what the number actually measures — and when to ignore it. This is the reading method.
1. Read the harness, not the headline
A benchmark is a specific test with a specific scoring rule. MMLU-style suites test multiple-choice knowledge retrieval; SWE-bench-style suites test patch generation against real issues; reasoning suites test multi-step derivation. "Model X scores 92%" means it scores 92% on that harness — nothing more general survives scrutiny.
2. Check saturation
When frontier systems cluster at 88–92% on a suite, the suite stops discriminating. Differences inside the saturated band are noise and prompt-formatting effects, not capability. A saturated benchmark's information content is exhausted — the honest statement is "these systems tie on this test."
3. Check contamination
If the test data exists on the public internet, assume it was in the training corpus unless the lab demonstrates otherwise (held-out regeneration, canary strings, post-training test construction). Contaminated scores measure memorization plus capability in unknown proportions.
4. Ask what transfers
The only score that predicts your workload is one measured on data like yours. A legal-doc Q&A suite tells you little about SQL generation. Practitioner's rule: vendor benchmarks shortlist candidates; your own eval on 50–200 real examples picks the winner.
The engineer's takeaway
Treat every benchmark claim as vendor-stated until independently reproduced, and treat every reproduced score as harness-specific. Budget a small internal eval for any model decision that matters — it is the cheapest diligence in the entire build.
Independent technical analysis — not investment advice or engineering counsel.