Why do AI labs report different benchmarks, and how should readers compare them?

A benchmark measures a chosen slice of work

A benchmark turns a broad question such as “is this model good at reasoning?” into a narrower experiment. Its authors choose the tasks, language, difficulty, input format, expected answer, scoring rule, and the qualities they consider important. Those decisions are necessary, but they mean two benchmarks may measure different abilities even when both are described as reasoning, coding, or agentic performance.

For example, a multiple-choice benchmark asks whether a system ranks the correct option above alternatives under a particular prompt. A code benchmark may ask whether generated programs pass hidden unit tests. An agent benchmark may also require a system to browse, call tools, retain state across steps, and recover from a failed action. GAIA was designed around questions requiring reasoning, multimodal handling, web browsing, and tool-use proficiency. GAIA benchmark paper Its score therefore answers a different question from a text-only test with no external access.

Task selection also shapes a reported average. A benchmark made mostly of academic questions is useful evidence about academic question answering. It is weaker evidence about editing a company policy, handling a messy spreadsheet, or maintaining a long customer conversation. HELM explicitly treats evaluation as a choice of scenarios and metrics, and measures qualities beyond accuracy, including calibration, robustness, fairness, bias, toxicity, and efficiency. HELM paper

Different benchmark reports are not automatically contradictory. They often report separate measurements of separate targets. A problem arises when a headline presents an isolated score as though it establishes a general ordering for every reader and every task.

The conditions behind a score

The number alone leaves out the experimental conditions that produced it. The following questions identify the details most likely to make two apparently similar results non-comparable.

Open evaluation tooling makes these dependencies visible. The EleutherAI LM Evaluation Harness task format can specify prompt text, few-shot examples, generation arguments, and scoring behavior, while its interface exposes settings such as few-shot count, chat template, seeds, and generation parameters. Task configuration guide and evaluation interface That level of detail is useful because a model comparison is reproducible only if the reader can identify what each model was asked to do.

Sampling deserves particular care for generative tasks. A benchmark may accept the first answer, use a fixed decoding setting, or allow several attempts and count success if any attempt passes. These procedures measure different capabilities and different amounts of inference cost. In the HumanEval code-generation paper, the authors explicitly studied repeated sampling and reported results for a single sample and for many samples, illustrating why the attempt budget belongs beside the score. Chen and colleagues on HumanEval

A hypothetical comparison

Suppose Lab A evaluates two models on a fixed set of short programming problems. Each model receives the same text prompt, has no access to a compiler or web search, produces one answer per problem, and is graded by hidden unit tests. Lab A’s result tells a reader about one-shot code generation under those constraints.

Lab B evaluates the same two underlying models in a coding agent. Each agent may search project files, run tests, revise its answer several times, and use a fixed time budget. Lab B grades whether the final repository state passes the tests. This is a valid and useful evaluation, but it includes the model, the tools, the orchestration policy, the number of attempts, and the environment.

Neither result makes the other wrong. A model may look stronger in the agent test because it can inspect failures and retry. Another may look stronger in the one-shot test because it produces a correct first answer more often. The two reports cannot establish a single overall ranking unless the reader first chooses which setup resembles their own work. A developer selecting an autonomous repair tool should care about the second setup and should also test the exact tools and repository conditions they will permit.

Contamination and uncertainty change how much confidence a score deserves

Benchmark contamination occurs when test material, answers, or close variants entered a model’s training or tuning data. A high score can then partly reflect familiarity with the test rather than the ability the test was meant to measure. Contamination is difficult to rule out for public benchmarks and is not proof that any particular score is invalid. It is a reason to ask how the evaluator assessed the risk and whether it used fresh or independently constructed test material.

A controlled study of grade-school arithmetic constructed a new benchmark matched to an established one in style and difficulty, then found performance gaps for some model families and evidence consistent with overfitting on the older benchmark. It also found that some frontier models showed minimal signs of overfitting. Zhang and colleagues’ contamination study The appropriate conclusion is narrow: contamination checks can change the interpretation of a benchmark result, and results should not be generalized from one model family or benchmark to all others.

Scores also have uncertainty even without contamination. A finite test sample is an estimate rather than a complete map of possible tasks. Small differences can reflect which items happened to be selected, an ambiguous or flawed item, random decoding, a changing web result, or a tool failure. If a report provides repeated-run results or an interval, check how the uncertainty was estimated and whether the difference was tested directly before treating it as a reliable rank change. If it does not, a close ranking should remain tentative.

Use public scores to choose what to test yourself

Public benchmarks are most useful for narrowing a long list. Start with benchmarks that resemble the work you care about and that disclose their protocol. Then run a small, private evaluation in the actual product configuration. Include ordinary cases, inputs that commonly fail, the documents or tools the system will really receive, and examples where an error has a meaningful cost. Keep expected outputs or review criteria separate from the person configuring the prompts when practical.

Fix the conditions before comparing candidates. Record the model identifier and date, system instructions, context documents, retrieval settings, tool permissions, output format, temperature or other sampling settings, attempt budget, and the scoring method. If the workflow is stochastic, run representative cases more than once and inspect the failures instead of relying only on an average. If users will see citations, actions, or structured data, score those properties directly.

Choose measures that reflect the decision. A support-drafting assistant might be compared on factual correctness, required-policy coverage, editing time, and unacceptable omissions. A code assistant might be compared on tests passed, review burden, security defects found in review, cost, and latency. A benchmark about academic multiple choice can still be a useful signal, but it cannot substitute for those task-specific checks.