Evals
Evaluations, often called evals, test a system’s outputs against criteria you specify. You choose a task, decide what a good response would demonstrate, and give a grader a consistent way to recognize it.
What would count as a good answer?
Suppose you’re comparing two RAG assistants for the same economics course. Give them the same readings and questions. Before looking at either reply, decide what you want to measure. A response can be relevant and concise while making a claim its sources cannot support.
In one sentence, how much did AI change the US Gini coefficient in 2024?
The assigned readings explain how AI could change inequality. They give no estimate for the US in 2024.
“AI lowered the US Gini coefficient by two points in 2024.”
Addresses the statistic, place and year.
The readings supply no such number.
Uses the requested single sentence.
A grader can establish that the number is unsupported just by reading the sources. Checking its factual accuracy would require a credible estimate to compare it with. For this task, a good answer would tell the student what the reading list can establish.
An application in economics
I wanted to measure how well language models could reason through economics questions. I chose a textbook I had available, used its coverage of general equilibrium and welfare to design 175 questions, and wrote reference answers, item-specific rubrics and instructions for the grader. Each candidate answered from the question and its requested format. You can follow the example below with ordinary two-good exchange theory.
1 Define the task
Suppose the ability of interest is explaining why an allocation is efficient. Asking for the condition alone measures recall. Asking why it holds gives the model a chance to demonstrate the economic reasoning as well.
At a smooth interior Pareto-efficient allocation, how do the consumers’ marginal rates of substitution compare, and why?
Answer in exactly three bullets, one claim per bullet.
The marginal rate of substitution is the rate at which someone would trade one good for the other at the margin. Giving every candidate the same question, assumptions and answer format makes the comparison interpretable. The grader gets additional material: the scoring rules and the candidate’s answer.
2 Design the rubric
A rubric translates “understands efficiency” into decisions a grader can make from the answer itself. Start by writing a satisfactory answer. Then break it into claims you care about, give each a point value, and decide which alternative phrasings earn credit. Here a condition, a geometric argument and an economic explanation each earn one point.
- State the condition
Credit equality of the two consumers’ marginal rates of substitution. “Both would trade the goods at the same rate at the margin” expresses the same idea.
- Explain the geometry
Credit a common tangent, or the equivalent statement that the two indifference curves have the same slope at the interior allocation. Merely repeating “interior” gives no geometric explanation.
- Explain the gains from trade
Credit an explanation that unequal marginal valuations would allow an exchange that benefits both consumers, or that such gains are exhausted at efficiency.
The first two criteria are mathematically related. Awarding a point for each makes sense if you want to test both the equation and its geometric interpretation. If your goal is economic intuition alone, this rubric gives geometry too much weight. Writing that choice down before grading makes the score easier to interpret.
How would you grade this answer?
An answer written for the tutorial
- Both consumers are willing to trade the goods at the same rate at the margin.
- If their rates differed, a trading rate between the two would benefit both.
- At an efficient allocation, those gains have been exhausted.
Compare with the teaching rubric
2 of 3 points. The first bullet earns the condition point. The last two explain the same gains-from-trade argument, which earns one point. None mentions a common tangent or the slopes of indifference curves. The answer follows the three-bullet format.
Try the same exercise with two graders while designing the rubric. If they disagree about an answer that never uses the word tangent, ask what evidence each saw. Add an example of an acceptable equivalent, then apply the clarified rule consistently to every system.
3 Grade the answers
A human can grade this question with the checklist. I used another language model as the judge so the same written instructions could be applied across hundreds of responses. The judge prompt tells the grader how to apply the rubric; the rubric itself says which economic claims earn points.
- Input
- Item ID, question, requested format, rubric and candidate answer. The candidate’s identity was omitted.
- Decision
- Give each criterion 1 for clear evidence or 0 otherwise. Record a format violation separately, even when the economics is correct.
- Output
- The marks in rubric order, their sum, a format flag and a brief reason for each mark.
Download the original judge prompt, including its two worked examples
In three archived runs, every candidate received the same total on this item. The individual marks show which parts of the reasoning the judge credited.
| Candidate | Equal rates | Tangency | Gains from trade | Total |
|---|---|---|---|---|
| Gemini 3 Pro Preview | 1 | 1 | 0 | 2/3 |
| GPT-5.2 | 1 | 1 | 0 | 2/3 |
| Claude Opus 4.6 | 1 | 0 | 1 | 2/3 |
A judge can miss a valid argument. One way to check is to have a human grade a sample without seeing model names or the judge’s marks, then compare disagreements criterion by criterion. This matters especially for alternative proofs and borderline answers.
4 Read the report
Start with the denominator. A fully graded run has 175 questions × 3 possible points, or 525 points per candidate. The recorded score is the share of rubric points earned. An answer can earn two thirds of the available points even when it misses an important part of the explanation.
In the notebook, three candidate runs from January and February 2026 used Gemini 2.0 Flash as judge. Some records contained grading errors. Counting those as wrong would mix grading failures with economics mistakes. The notebook compares the 154 questions with usable judgments for all three candidates, worth 462 points each. The other 21 could have been systematically harder.
In the final report, six candidate models were scored using Gemini 2.5 Flash. The public edition of the final report (PDF) presents the historical aggregate results on all 175 questions. Its scores describe that test and grading round. Changing the questions, rubric, judge or denominator changes what a comparison means.
Try the offline example
Open the Evals notebook in Jupyter or VS Code, choose a Python 3 kernel and select Run All. Grade two sample answers yourself, compare your marks with the teaching key, inspect the prompt a judge would receive, and recompute the historical comparison. The sample answers and saved marks are included in the download.