A benchmark chart is one of the most persuasive objects on the internet. It has axes, decimals, and a bar that is visibly longer than the others. It also has a task, a sample, and an author, and those three things decide whether the chart means anything for your work.
None of this is an argument against benchmarks. It is an argument for reading them as arguments. When a vendor says a model scores 84.2 on a suite, the useful response is not "is 84.2 good?" It is "who ran the suite, on what task, with what contamination, and at what cost?" Five questions cover most of the ground.
1. Who ran it, and who paid?
A benchmark run by the vendor whose product wins is not worthless, but it is advertising until someone independent repeats it. Check whether the harness, the prompts, and the raw outputs are public. A number with no reproducible command behind it belongs in the same category as a testimonial: interesting, not load-bearing.
Independent leaderboards help, but they have their own incentives: sponsorship, submission policies, and the temptation to keep a familiar ranking stable. Read the methodology page before the leaderboard. If the page does not say how entries are submitted and verified, the ranking is a popularity board.
2. What task does the benchmark actually measure?
Benchmark names are abstractions. "Reasoning" can mean competition math, logic puzzles, or multi-step planning. "Coding" can mean writing a function from a docstring or fixing a failing test in a repository. Those are different jobs, and a model that is excellent at one can be mediocre at the other.
Find the sample. If the suite publishes five examples, read them. Ask whether the format resembles your work: the same language, the same kind of input, the same tolerance for a wrong answer. A model with a high score on short, self-contained problems tells you little about a long document with contradictory sections.
Also look at how the score is aggregated. A single number can hide a split result: excellent on arithmetic, poor on multi-step word problems, average overall. If the suite reports only a total, look for a per-category table before you assume uniform quality. The category that matters to you is the one to read, not the average.
3. Could the test data have leaked into training?
Contamination is the quiet failure mode. If the benchmark questions are public and the model was trained on the public web, some of the answers may have been in the training data. The model is not reasoning; it is recalling. Scores on contaminated suites inflate, and the inflation is invisible in the chart.
Reputable evaluations publish contamination checks or hold out private test sets. If neither is mentioned, assume the risk exists. This is not an accusation against the vendor; it is a property of the method. Benchmarks that rotate their questions are more trustworthy than benchmarks that have been on the internet for years.
4. How much variance is behind the decimal?
A score of 84.2 looks more precise than a score of 84. The extra digit is usually noise. Small sample sizes, temperature settings, retry policies, and scoring rubrics all move the result by several points. A one-point gap between two models is not a ranking; it is a tie unless the confidence interval says otherwise.
Look for repeated runs, error bars, or at least a note about seeds. If the chart shows a single number per model with no variance, read the gaps as approximate. Two models within a few points are candidates for your own test, not a settled order.
5. What does the score cost to use?
Accuracy is one axis. The others are latency, price, context limits, rate limits, and operational complexity. A model that wins by three points but costs four times as much and answers twice as slowly has not obviously won. A benchmark chart rarely shows the bill.
Convert the score into a decision: if the task runs ten thousand times a month, what is the cost per point gained? If the answer is "we would need a second job to pay for it," the chart is an interesting fact, not a procurement argument.
The only benchmark that matters
After the five questions, run your own small evaluation on the work you actually do. Pick twenty real inputs, write down the output you would accept, and score both models blind. Twenty examples will not produce a publishable number, but they will tell you which model fails on your documents. That is the only result that generalizes to your Tuesday.
Our guides cover the same discipline for other measurements: how ranking works explains what the directory's score is not, and how to read a model listing walks the fields that matter more than a single number. A benchmark is a starting point for a test, not a substitute for one.