How to Analyse an Agent: Sixty Years of Answers to a Two-Year-Old Question

01Everyone Ships, Nobody Measures

In 1962, a Philadelphia psychiatrist named Aaron Beck — the same Beck who would later found cognitive behavioural therapy — published a study that should be taped to the monitor of everyone building AI agents. His team took 153 outpatients and had each one interviewed twice in quick succession — the second interview beginning minutes after the first ended — by two experienced psychiatrists working independently; the second doctor never saw the first doctor’s conclusion. (A study designed this way today would pass through a consent process the 1960s did not have; what redeems it is that the lens was pointed at the doctors, never at the patients.) On the specific diagnosis, the two experts agreed 54% of the time.^[1] Credentialed specialists, same patient, same day — and they disagreed almost half the time.

The truly modern move was the companion paper, published the same year.^[2] Instead of stopping at the embarrassing number, Beck’s team ran the disagreements down and published the autopsy: only about five percent traced to patients actually presenting differently between interviews. Roughly a third was the clinicians — eliciting different histories, weighing the same symptom differently. And nearly two-thirds was the nosology itself: diagnostic categories so vague that two careful experts could not apply them the same way. The raters were mostly fine. The rubric was broken. When psychiatry finally rebuilt that rubric — the DSM-III of 1980, with explicit checklist criteria — its field trials reported their results in the currency this post is about: kappa.^[3] Expert judgment, unmeasured, quietly rots; measured, it tells you exactly what to fix.

Sixty-four years later, the default state of agent evaluation — wherever nobody has yet made it someone’s job — is a spreadsheet of twenty prompts, a column of checkmarks, and a feeling. When the model gets upgraded, someone re-runs the twenty prompts, squints, and declares the new version “noticeably better.” This is not evaluation. It is vibes with version control — Beck’s psychiatrists, without Beck’s honesty about checking.

The instinct behind the spreadsheet is right, though: agent outputs are open-ended, so a person (or another model) has to judge them. The moment you accept that, you have turned agent evaluation into a measurement problem — specifically, the problem of extracting a reliable signal from unreliable judges. And that problem has a literature. Psychologists, epidemiologists, and content analysts have been fighting about it since the 1950s, and the weapons they forged — kappa, alpha, paired tests — transfer to agents almost without modification.

One framing carries this whole post: your LLM judge is just another rater. Everything the field learned about human raters — that they drift, that they agree by accident, that their agreement must be measured rather than assumed — applies verbatim to a model grading another model.

02First Decide What You Are Scoring

Before any statistics, fix the unit of analysis. Agents produce trajectories — sequences of tool calls, intermediate reasoning, and a final artefact — and you can score at three altitudes:

Every downstream statistic depends on this choice: kappa on outcomes answers a different question than kappa on steps. The classic failure is mixing altitudes — a judge scoring “overall quality” while the team believes it is measuring task success. The altitudes also compose, and the aggregation rule is itself a design decision: all-steps-correct is a much stricter trajectory score than no-fatal-step, and a trajectory can be flawless while the outcome fails on an environment quirk. Pick the unit, write down how lower levels roll up into higher ones, and make every number in your eval report state which unit it describes.

031955–1980: The Agreement Wars

Suppose two judges label 100 agent trajectories as pass or fail, and they agree on 82. Is 82% good? The uncomfortable answer, known since the 1950s, is that raw percent agreement is nearly meaningless: if both judges pass 90% of everything, they will agree about 82% of the time by pure chance. William Scott made this point in 1955 with his π coefficient,^[4] built for the survey researchers of the era who employed rooms of human coders to turn open-ended interview answers into categories — the gig-work data labelling of its day. Five years later Jacob Cohen, a psychologist at NYU working amid exactly the diagnostic reliability mess Beck would soon quantify, sharpened the idea into the statistic that stuck.^[5] (The same Cohen later gave the field effect sizes and statistical power — the man spent a career forcing psychology to say how much and how sure.) Kappa measures agreement above chance, as a fraction of the agreement that was available above chance:

where $p_o$ is observed agreement and $p_e$ is the agreement two judges with these marginal rates would reach by accident. Our 82%-agreement judges land at κ ≈ 0.0 if their agreement is all prevalence — a number that tells you the truth where the raw percentage flattered you.

Landis and Koch published the interpretation scale everyone still quotes (0.61–0.80 “substantial,” 0.81+ “almost perfect”) in 1977^[6] — and admitted in the same paper that the bands were arbitrary. Treat them as vocabulary, not physics. Know also the kappa paradox documented by Feinstein and Cicchetti in 1990:^[7] when one class dominates (say, your agent passes 95% of tasks), kappa can be brutally low even when judges almost always agree. That is not a bug, but it is a symptom with several possible causes — genuinely low prevalence, skewed rater thresholds, or an eval set too easy to reveal whether the judges can discriminate. For an agent leaderboard saturating at 95%, the last one is the explanation worth ruling out first.

The paradox eventually got its own repairman. In 2008 Kilem Gwet, a Cameroonian-born statistician working in Maryland, traced the problem to kappa’s model of chance: it estimates chance agreement from the raters’ marginal totals, so under heavy skew the denominator $1 - p_e$ collapses and the coefficient nosedives on data where raters nearly always agree. His AC1 coefficient^[8] replaces that model with a more realistic one — raters only guess on the genuinely ambiguous cases, not on everything — and stays stable exactly where kappa panics. In a head-to-head on personality-disorder diagnoses, kappa swung with prevalence while AC1 stayed close to the observed agreement.^[9] Fair warning from the counter-literature, though: AC1 is not a drop-in replacement — it makes its own behavioural assumptions, and methodologists have pushed back on treating it as strictly superior.^[10] The practical move for agent evals: when your pass rate is skewed, report both. Low kappa alongside high observed agreement and high AC1 is a signal to go inspect — prevalence, marginal imbalance, whether the eval set is saturated — not a verdict that the judges are reliable. The divergence tells you where to look, not what you will find.

Joseph Fleiss generalised the idea to many raters in 1971 — his statistic, which everyone calls Fleiss’s kappa, is technically a multi-rater Scott’s π.^[11] But the instrument to reach for today came out of a very different fight. In the late 1960s, American politicians wanted to know whether television was making children violent, and the Annenberg School in Philadelphia won the job of counting: arm teams of coders with rubrics, sit them in front of prime-time TV, and tally every act of violence. It is hard to imagine a messier labelling task — the coders could not even agree on what counted as violence, and the data came in every shape the existing statistics could not handle. Klaus Krippendorff, the German-born design-theorist-turned-content-analyst on the team, worried the whole study would be dismissed over it — and built the agreement measure that could survive that reality.^[12] His α handles any number of raters, missing labels (rater three only scored half the trajectories — fine), and nominal, ordinal, or interval data under one definition: one minus the ratio of observed to expected disagreement,

Krippendorff’s own guidance: rely on conclusions only when α ≥ 0.800; treat 0.667–0.800 as tentative; below that, look first at your rubric — though ambiguous items, skewed samples, and rater training can all drag alpha down too.^[13] Those two thresholds have survived fifty years of use.

04Your Judge Is Just Another Rater

LLM-as-judge is how agent evals scale, and it works — the MT-Bench study^[14] found GPT-4 agreeing with human preferences about as often as humans agree with each other: over 80% with ties excluded, and around the mid-60s with them — a reminder that tie handling is part of what you are measuring, not an administrative detail. But “works” is an empirical claim you must re-establish for your task, your rubric, and your judge model. The protocol is exactly the one content analysts would use for a new human coder:

If the human–human alpha is 0.55, stop: labels humans cannot produce consistently are no reference to validate a judge against. Rewrite the rubric — usually by decomposing one vague “quality” question into five narrow, binary ones — until humans agree, then calibrate the judge against that. One caveat on the counts: 150–300 items is a starting heuristic, not a derived sample size — the n you actually need depends on prevalence, the number of categories, and the precision you are after, and the error-bars paper in section 08 walks through that arithmetic properly.

One warning before celebrating that 0.81: reliability is not validity, and the rubric rewrite bought its agreement partly by narrowing what is measured. Five binary questions can be answered consistently and still miss what you actually care about — was the answer appropriate, would a domain expert trust it — so high agreement can certify a consistently inadequate construct. Alongside inter-rater reliability, check the construct itself: do rubric scores track outcomes you can verify externally, and do they hold across task types and user groups rather than just on average? Reliable measurement of the wrong thing is still wrong.

Psychiatry has a second lesson here, about where judges get their standards. In the early 1970s the US–UK Diagnostic Project showed the same videotaped patient interviews to psychiatrists in New York and London: the New Yorkers diagnosed schizophrenia at roughly twice the London rate.^[15] Identical evidence, different training distributions, systematically different verdicts. An LLM judge is no different — it imports the preferences of whatever fine-tuning produced it, and you will not see the skew until you measure against an outside anchor.

The MT-Bench study catalogued the specific pathologies to design around: position bias (prefer the first answer shown — fix by scoring both orders and keeping only consistent verdicts), verbosity bias (longer looks better — fix with length-controlled rubrics), and self-preference (models favour their own prose — fix by never letting a judge know, or infer, which system produced an output). A judge that has not been tested for these is not an eval; it is an opinion with an API.

05Where the Rater Analogy Breaks

Honesty requires the counterargument. An LLM judge is not just another rater, and the places where the analogy fails are exactly the places where the classical statistics go quiet. The old machinery assumes raters who independently observe the same evidence and stay the same rater from one session to the next. Three of those assumptions do not survive contact with a model.

Independence. Beck’s two psychiatrists brought separate educations to the same patient. Two LLM judges — or a judge and the agent it grades — may share most of a pretraining corpus, which means shared blind spots and correlated errors; self-preference^[14] is just the visible tip. This matters because agreement statistics assume disagreement is informative: kappa between two judges that fail in the same direction is high while both are wrong. Psychometrics has a name for the distinction — reliability is agreement, validity is truth — and correlated raters buy you the first without the second. No agreement coefficient can see the difference; only an outside anchor with different failure modes can, which in practice means humans.

Stability. A human rater is roughly the same rater on Tuesday as on Monday. An LLM judge is a function of its prompt and its weights: reword the rubric and you have quietly hired a different judge; let the provider upgrade the model underneath you and the judge has been replaced without an exit interview. This is why move four of the calibration protocol — re-run on every judge or rubric change — is not bureaucracy. It is the only thing standing between your longitudinal metrics and a rater who keeps being swapped mid-study.

Adversarial pressure. Beck’s patients could exaggerate symptoms, but no patient could write instructions into the psychiatrist’s notes. The artefact an LLM judge scores can do exactly that: indirect prompt injection turns any text the judge reads into a channel for manipulating it,^[16] and researchers have shown that a “null model” emitting one constant, irrelevant response can reach an 86.5% length-controlled win rate on AlpacaEval 2.0 — a metric designed specifically to resist this kind of gaming.^[17] A scoring pipeline that has not been red-teamed is a rater who believes everything the examinee whispers to it.

None of this retires the statistics; it bounds them. Kappa and alpha certify that judges agree — they cannot certify that agreement means truth when the judges share a failure direction. Against correlated, unstable, manipulable raters, the classical toolkit remains necessary and stops being sufficient — which is why calibrating against humans is the one step in this post you cannot automate away.

06Comparing Two Agents: McNemar, Not Eyeballs

Now the question everyone actually asks: is agent B better than agent A? You ran both on the same 200 tasks; B scored 71%, A scored 66%. Ship B? Not so fast — the answer hides in the pairing. Because both agents saw identical tasks, the tasks where both succeed or both fail tell you nothing about the difference. All the evidence lives in the discordant pairs, and the arithmetic for them fits in five pages: a 1947 note in Psychometrika,^[18] published the same year Bell Labs first demonstrated the transistor. Its author, Quinn McNemar — Stanford psychometrician, student of Lewis Terman, co-reviser of the Stanford–Binet, writer of the Psychological Statistics textbook a generation learned from — left behind exactly the test your model-comparison spreadsheet is missing:

The test statistic uses only $b$ and $c$:

Here that is 100/50 = 2.0 — a p-value around 0.16, and a 95% interval on the paired difference running from roughly −2 to +12 points. The experiment is too imprecise to distinguish a small regression from a meaningful gain — which is the actual conclusion, and not the same as “no difference.” This is the routine, deflating discovery of paired testing: eval sets of a few hundred tasks resolve only large differences, and most model-over-model improvements are not large. When $b + c$ is small (under about 25), skip the chi-squared approximation and use the exact form: under the null, $b \sim \mathrm{Bin}(b + c,\ \tfrac{1}{2})$.

The same logic generalises. Graded scores instead of pass/fail: Wilcoxon signed-rank,^[19] again exploiting the pairing — and for measuring agreement on ordinal rubric scores, Cohen’s own weighted kappa^[20] or alpha with an ordinal distance metric. Uncertainty on any aggregate metric: bootstrap over tasks (task difficulty clusters the data; with repeated runs, resample runs within resampled tasks to keep both sources of noise).^[21] Sweeping five prompt variants against a baseline: correct for multiple comparisons (Holm’s procedure^[22] is never less powerful than Bonferroni and just as easy). None of this is exotic; all of it is routinely skipped. In fact, the whole section is pip-installable:

07Ability Is Not Reliability: pass@k versus pass^k

Agents are stochastic, so a single run per task is a sample of size one. The code-generation world standardised on pass@k^[23]: the probability that at least one of k attempts succeeds. That is the right metric when retries are cheap and verifiable — generate ten patches, keep the one that passes the tests. But a customer-facing agent gets no retries: it either books the flight or it does not. The team at Sierra — a company whose entire business is putting agents in front of other companies’ customers, and which therefore feels every failure as a support ticket — built τ-bench^[24] around simulated airline and retail conversations, and introduced the pessimist’s mirror image, pass^k: the probability that all k attempts succeed. Their own headline result: GPT-4o solved about 60% of retail tasks once, but only about 25% of them eight times in a row. With $c$ successes observed in $n$ runs of a task, the unbiased estimators are

averaged over tasks — which estimates the average of $p^k$ across tasks, not the k-th power of the average; task heterogeneity is part of the measurement, not a footnote. The two metrics tell sharply different stories about the same agent: a task solved 80% of the time contributes essentially 1.0 to pass@8 and just 0.17 to pass^8. A user who hits the same workflow k times experiences pass^k; leaderboards report pass@1. That gap is not a reporting quirk — it is the difference between the best case and every case.

Practically: run each eval task enough times to support the k you care about — the pass^k estimator needs n ≥ k and is very coarse near it (at n = 8, a task’s pass^8 estimate is literally zero or one), so precision comes from the number of tasks as much as runs. Treat 8–16 runs as a floor, report bootstrap intervals, and do the precision arithmetic for your own target rather than trusting any fixed n. Report the mean and the cross-run variance, and pick k to match how often your users will hit the same workflow. Reliability regressions hide completely inside single-run evals.

08The Newer Instruments: 2021–2026

The toolkit has not stopped growing — and the newest instruments keep rediscovering the old themes. Between Krippendorff and the money metrics sits a bridge generation that professionalised the measurement itself. HELM (2022)^[25] made the multi-metric point institutional: report calibration, robustness, fairness, and efficiency alongside accuracy, because a single column invites Goodhart. BIG-bench^[26] assembled 204 tasks with human baselines and taught the field how quickly benchmark breadth decays into training data. GAIA^[27] designed 466 questions for the human-easy, machine-hard quadrant with single unambiguous answers — the cleanest solution to the judge problem being to design the judge out entirely.

Then evaluation moved into environments. AgentBench^[28] ran one model across eight interactive worlds, from databases to operating systems; WebArena^[29] built self-hosted, fully functional websites where success is verified functionally — did the order actually get placed — rather than judged; OSWorld^[30] did the same for real operating systems across 369 tasks. Underneath, the harness layer grew up: frameworks like the UK AI Security Institute’s Inspect^[31] treat evals as versioned, reproducible code — the same CI discipline as the software being judged. Each of these embodies a measurement decision this post has already argued for: fix the unit, verify functionally where you can, keep the instrument stable.

The most striking recent metric measures capability in human time. METR’s time-horizon work^[32] asks how long a task — clocked by how long it takes a human professional — an agent can complete with 50% reliability. That horizon has doubled roughly every seven months since 2019, faster lately; and the same models’ 80%-reliability horizon runs several times shorter. That gap is section 07 all over again, denominated in hours instead of percentages. Toby Ord tentatively proposed that the curves fit a constant-hazard model — agents failing at a roughly constant rate per human-minute of task, giving each agent a characteristic half-life^[33] — though later analysis, which Ord has acknowledged, suggests the hazard rate declines on longer tasks. Treat the half-life as a first lens, not a law.

Success also stopped being the only column. τ-bench had already made policy compliance part of passing — did the agent follow the airline’s refund rules, not just placate the customer? — and its successor τ²-bench^[34] made the user an actor too (dual control: both sides operate on the environment) and attributes each failure — critical or benign, the agent’s fault or the user’s — because an unattributed failure teaches you nothing. The Princeton “AI Agents That Matter” critique^[35] pushed cost onto the axis: report accuracy–cost Pareto frontiers, after showing leaderboard-topping agents being matched by simple baselines at a fraction of the spend. And a whole family now measures agents in money: SWE-Lancer prices tasks in real Upwork payouts, from $50 bug fixes to $32,000 features;^[36] GDPval scores win rates against professionals with an average of fourteen years’ experience, blind-graded;^[37] Vending-Bench hands the agent a simulated vending-machine business for twenty-million-token runs and reports final net worth — where the telling number is the variance across runs.^[38] Different currencies, same discipline: each of these is a choice of measurement unit — compliance, dollars, hours — defended the way section 02 asks.

The judges got their own benchmarks at last — RewardBench evaluates the evaluators directly^[39] — and Chatbot Arena moved to style-controlled ratings that regress out response length and formatting before scoring,^[40] which is verbosity bias graduating from a caveat in a paper to a term in the scoring model. Even the error bars caught up: Anthropic’s “Adding Error Bars to Evals”^[41] recommends clustered standard errors and paired-difference analysis for model comparisons — the paired-comparison discipline of section 06, restated as method.

09The Whole Framework on One Screen

Assembled, the pipeline looks like this — every piece older than the models it measures:

And one closing story that ties the whole post together. SWE-bench became the de facto agent benchmark for software engineering — and in 2024 OpenAI, working with the benchmark’s authors, put 93 annotators on re-examining it. They found a large fraction of tasks underspecified or carrying broken tests, producing the filtered SWE-bench Verified.^[42] The most cited agent benchmark in the world needed exactly the treatment this post prescribes: multiple raters, explicit rubrics, agreement analysis. Measurement is not a layer you add on top of the benchmark. It is the benchmark.

There is one more inheritance worth claiming. Psychometrics stopped treating test items as interchangeable half a century ago: item response theory models every item’s difficulty and discrimination as parameters to be estimated, not assumed away.^[43] A golden set is a test instrument in exactly that sense — some tasks separate strong agents from weak ones and some separate nothing — and recent work has begun fitting IRT models to LLM benchmarks for precisely this reason — tinyBenchmarks showed 100 IRT-curated examples can stand in for MMLU’s fourteen thousand.^[44] Expect your eval items to be unequal, and measure how unequal they are.

Beck’s psychiatrists thought they were measuring patients; it took a 54% agreement rate to show they were mostly measuring themselves. Every team with an uncalibrated LLM judge is running that experiment again, at scale, without the follow-up study. The instruments to do better are sitting right there — McNemar pairing observations in 1947, Cohen correcting for chance in 1960, Krippendorff coding televised violence in 1970 — old, because the problem is old: fallible judges, messy artefacts, and the temptation to trust a number because it has a decimal point. Agents did not create that problem. They just made it everyone’s.

10References

1. [1]Beck, A. T., Ward, C. H., Mendelson, M., Mock, J. E., & Erbaugh, J. K. (1962). Reliability of psychiatric diagnoses: 2. A study of consistency of clinical judgments and ratings. American Journal of Psychiatry, 119(4), 351–357. doi:10.1176/ajp.119.4.351
2. [2]Ward, C. H., Beck, A. T., Mendelson, M., Mock, J. E., & Erbaugh, J. K. (1962). The psychiatric nomenclature: Reasons for diagnostic disagreement. Archives of General Psychiatry, 7(3), 198–205. doi:10.1001/archpsyc.1962.01720030044006
3. [3]Spitzer, R. L., Forman, J. B. W., & Nee, J. (1979). DSM-III field trials: I. Initial interrater diagnostic reliability. American Journal of Psychiatry, 136(6), 815–817. doi:10.1176/ajp.136.6.815
4. [4]Scott, W. A. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19(3), 321–325. doi:10.1086/266577
5. [5]Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. doi:10.1177/001316446002000104
6. [6]Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. doi:10.2307/2529310
7. [7]Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. doi:10.1016/0895-4356(90)90158-L
8. [8]Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29–48. doi:10.1348/000711006X126600
9. [9]Wongpakaran, N., Wongpakaran, T., Wedding, D., & Gwet, K. L. (2013). A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples. BMC Medical Research Methodology, 13, 61. doi:10.1186/1471-2288-13-61
10. [10]Vach, W., & Gerke, O. (2023). Gwet’s AC1 is not a substitute for Cohen’s kappa — A comparison of basic properties. MethodsX, 10, 102212. doi:10.1016/j.mex.2023.102212
11. [11]Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382. doi:10.1037/h0031619
12. [12]Krippendorff, K. (1970). Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement, 30(1), 61–70. doi:10.1177/001316447003000105
13. [13]Krippendorff, K. (2004). Content analysis: An introduction to its methodology (2nd ed.). Sage Publications.
14. [14]Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36. arXiv:2306.05685.
15. [15]Cooper, J. E., Kendell, R. E., Gurland, B. J., Sharpe, L., Copeland, J. R. M., & Simon, R. (1972). Psychiatric diagnosis in New York and London (Maudsley Monograph No. 20). Oxford University Press.
16. [16]Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. arXiv:2302.12173.
17. [17]Zheng, X., Pang, T., Du, C., Liu, Q., Jiang, J., & Lin, M. (2024). Cheating automatic LLM benchmarks: Null models achieve high win rates. arXiv:2410.07137.
18. [18]McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2), 153–157. doi:10.1007/BF02295996
19. [19]Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1(6), 80–83. doi:10.2307/3001968
20. [20]Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220. doi:10.1037/h0026256
21. [21]Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1–26. doi:10.1214/aos/1176344552
22. [22]Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70. jstor.org/stable/4615733
23. [23]Chen, M., Tworek, J., Jun, H., et al. (2021). Evaluating large language models trained on code. arXiv:2107.03374.
24. [24]Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024). τ-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv:2406.12045.
25. [25]Liang, P., et al. (2022). Holistic evaluation of language models. arXiv:2211.09110.
26. [26]Srivastava, A., et al. (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615.
27. [27]Mialon, G., et al. (2023). GAIA: A benchmark for general AI assistants. arXiv:2311.12983.
28. [28]Liu, X., et al. (2023). AgentBench: Evaluating LLMs as agents. arXiv:2308.03688.
29. [29]Zhou, S., et al. (2023). WebArena: A realistic web environment for building autonomous agents. arXiv:2307.13854.
30. [30]Xie, T., et al. (2024). OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv:2404.07972.
31. [31]UK AI Security Institute. (2024). Inspect: An open-source framework for large language model evaluations. inspect.aisi.org.uk.
32. [32]Kwa, T., West, B., et al. (2025). Measuring AI ability to complete long tasks. arXiv:2503.14499.
33. [33]Ord, T. (2025). Is there a half-life for the success rates of AI agents? arXiv:2505.05115.
34. [34]Barres, V., Dong, H., Ray, S., Si, X., & Narasimhan, K. (2025). τ²-Bench: Evaluating conversational agents in a dual-control environment. arXiv:2506.07982.
35. [35]Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., & Narayanan, A. (2024). AI agents that matter. arXiv:2407.01502.
36. [36]Miserendino, S., et al. (2025). SWE-Lancer: Can frontier LLMs earn $1 million from real-world freelance software engineering? arXiv:2502.12115.
37. [37]Patwardhan, T., et al. (2025). GDPval: Evaluating AI model performance on real-world economically valuable tasks. arXiv:2510.04374.
38. [38]Backlund, A., & Petersson, L. (2025). Vending-Bench: A benchmark for long-term coherence of autonomous agents. arXiv:2502.15840.
39. [39]Lambert, N., et al. (2024). RewardBench: Evaluating reward models for language modeling. arXiv:2403.13787.
40. [40]LMSYS Org. (2024). Does style matter? Disentangling style and substance in Chatbot Arena. lmsys.org/blog/2024-08-28-style-control.
41. [41]Miller, E. (2024). Adding error bars to evals: A statistical approach to language model evaluations. arXiv:2411.00640.
42. [42]OpenAI. (2024). Introducing SWE-bench Verified. openai.com/index/introducing-swe-bench-verified.
43. [43]Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Addison-Wesley.
44. [44]Maia Polo, F., Weber, L., Choshen, L., Sun, Y., Xu, G., & Yurochkin, M. (2024). tinyBenchmarks: Evaluating LLMs with fewer examples. arXiv:2402.14992.