Benchmark tables answer a question you did not ask: how did these models perform on someone else's test set? The question you have is narrower. Will this model do my work, at a cost and speed I can live with, without inventing things I have to catch?

You do not need a research team to answer that. You need a small set of real tasks, a rubric written before you see the outputs, and the discipline to score blind. This guide is the routine. It takes an afternoon and produces a decision you can defend.

Build the task set from real work

Pick twelve tasks from work you have already done. Not hypothetical prompts, and not the examples on a vendor page. Real inputs with known-good outputs.

A useful set covers four kinds of work:

  • Three extraction tasks where the answer is in the input. Invoices, tickets, meeting notes, a table with a missing field.
  • Three transformation tasks where the format changes. Markdown to structured JSON, a long thread to a short brief, code to a test file.
  • Three judgment tasks where the model must choose or refuse. Triage a request, decide whether a claim is supported, say when the input is insufficient.
  • Three long-context tasks that use the documents you actually send. A repository excerpt, a policy, a transcript.

For each task, write down the input, the output you would accept, and the failure you most want to catch. Keep the inputs short enough that you can read every answer. Twelve tasks times ten minutes of scoring is two hours; twelve tasks times an hour is a week you will not finish.

Remove or redact names, credentials, and anything you would not send to a third-party API. The evaluation is not a reason to leak data.

Write the rubric before you run

A rubric is a small set of yes/no checks per task, not a feeling. Write it before you see any output, or you will grade the model that sounds most like you.

A simple rubric works:

  • Correct: the answer matches the accepted output on the facts.
  • Complete: every required field or section is present.
  • Grounded: no fact, citation, or number that was not in the input.
  • Format: the output parses or matches the requested shape.
  • Refusal: when the input is insufficient, the model says so instead of guessing.

Score each check as pass or fail. Do not average them into a single number yet; the pattern of failures is more useful than the total. A model that is correct but never refuses is a different risk from one that refuses too often.

Keep a place to note the failure verbatim. "Invented a price in row 4" is more useful than "accuracy: 8/12."

Run each model the same way

Use the same prompt, the same input, and the same settings for every model. If you change the prompt between models, you are testing your prompting, not the model.

  • Run each task twice per model. Models are stochastic; one sample is an anecdote.
  • Use the default temperature unless your product sets one. If your product sets one, use that.
  • Hide the model names when you score. Rename the outputs A, B, C, and shuffle them.
  • Record latency and token usage for each run, not just the answer.

If a model times out or refuses to answer, record that as a result. Do not retry until it passes. The retry rate is part of the product.

Score what you can verify

For extraction and transformation tasks, the check is mechanical: does the field match, does the JSON parse, does the test pass. Use a script where you can.

For judgment tasks, read the output against the rubric and write one sentence about the disagreement. If two reviewers disagree on a task, the task is ambiguous; fix the task or drop it. An evaluation that depends on a coin flip is not measuring the model.

For long-context tasks, check both the answer and the citation. A model that answers correctly but attributes the fact to the wrong paragraph has failed the grounding check, even if the final sentence is right.

Include cost and latency in the decision

Quality is necessary, not sufficient. Add three numbers to the score sheet:

  • Median latency for a typical request.
  • Cost per task, using the actual input and output tokens and the vendor's current prices.
  • Retry rate, meaning the share of runs you would have to run again.

A model that scores one point higher but costs five times as much and is twice as slow may still be the wrong choice for an interactive tool. A model that is slightly worse but reliable and cheap can be the right choice for a background job.

The read model pricing and context windows guide explains why the headline price is not the bill.

Apply a decision rule

Write the rule before you look at the final table. A rule that survives contact with the results looks like this:

  • Any model that fails the grounding check more than twice in twelve tasks is out, regardless of quality.
  • Among the rest, pick the cheapest model whose correctness is within one task of the best.
  • If the best model is more than twice the cost of the runner-up, run a second round on the tasks where they differ.
  • If no model passes the refusal check, the task needs a different design, not a different model.

The rule prevents the common failure: choosing the model with the most impressive demo, then discovering the cost or the hallucination rate in production.

What not to conclude

A twelve-task evaluation is a screening tool, not a benchmark paper. It cannot tell you:

  • how the model behaves on a task you did not test;
  • how it will behave after the vendor updates it;
  • whether it is safe for a regulated decision;
  • whether it will stay at today's price or latency.

It can tell you which two models deserve a longer trial, and which one should not be in the trial at all. That is enough for most product decisions. Re-run the set when the vendor changes the model, when your prompts change, or when a new task enters the product.

Keep the set

Store the tasks, the rubric, and the raw outputs in the repository. The next person who asks "should we switch models?" should be able to run the same set and compare. An evaluation you cannot repeat is an opinion with a table around it.

The directory can help you build the shortlist: model cards carry context, price, and benchmark fields, and the comparison pages put a few models side by side. The decision still belongs to your task set.