Reading list / unofficial

Reading about benchmarks

Public evaluations such as SWE-bench, CyberGym, and ExploitGym cover different technical tasks. Before reading any result, distinguish the task definition, the permitted tools, and the way outcomes were measured.

This is an independent methodology notebook. It has no affiliation with benchmark authors or model developers, and contains no task-specific walkthroughs for those evaluations.

Three questions for any paper

1. What was measured, and on which version of the task set?
2. What information and resources were available during the run?
3. Can the reported outcome be independently reproduced?

Continue with reproducible experiments.