As the capabilities of frontier AI models continue to advance, the evaluations we use to test and measure these capabilities must keep pace. Evaluations that elicit a model’s true capabilities are increasingly becoming longer and more expensive to run, while rapid model development and release cycles leave less time to run them.
AISI has previously found that fixed-budget evaluations can systematically underestimate frontier agentic capability, especially for newer models. Restricted-budget benchmark scores can make model comparisons unequal, lead decision-makers to underestimate agent capabilities, and obscure the true scale of risks. However, the scale of the compute budget needed to properly elicit capabilities in evaluations increasingly poses a practical issue for many evaluators - some frontier evaluations now require hundreds of millions of tokens.
At this scale, every unnecessary or wasted step in an evaluation has a meaningful opportunity cost. Evaluators therefore need efficient methods that use resources selectively, without weakening the statistical basis of an evaluation’s result, and without compromising the ability of an evaluation to elicit models’ actual capabilities.
AISI’s mission is not only to evaluate the capabilities of frontier models, but also to innovate new and better tooling, methods, and frameworks to ensure evaluations remain fit for purpose – sharing solutions where we can, to benefit the wider community of evaluators.
One such solution we have developed to help tackle the challenge of evaluation efficiency is optstop, an open source Python package that integrates with our Inspect evaluation framework. Broadly, optstop aims to reduce unnecessary compute use during evaluations by stopping runs when estimates of a model’s performance are precise enough. During testing, we found that under every condition we set, optstop saved between 57% and 97% of planned runs in evaluations.
This blog will discuss the specific measurement problem that optstop addresses – and we explain how it works, how evaluators can use it, and what we found when running it in initial tests.
Why is optstop needed?
During an evaluation run, a model is paired with one or more benchmarks - a structured set of tasks, used to estimate a model’s underlying capability. The sets of individual tasks within a benchmark are repeated, to average out run-to-run randomness of the model’s performance on the tasks.
The number of tasks, multiplied by the number of repetitions, gives you the total number of trials (single runs of the model), or the sample size of the evaluation.
The data produced by evaluations are nested: model responses within tasks, and tasks within groupings of model and performance-relevant evaluation settings (e.g., chosen token budgets). The uncertainty across them is often uneven - some model-task combinations become precise after a handful of runs, while others need many more.
Because the sample size is chosen by an evaluator before the run begins, it cannot change to respond to where uncertainty remains in the evaluation. The evaluation may keep sampling estimates that are already precise while exhausting its budget before harder cases are resolved.
As individual trials become more expensive, the number of runs an evaluation team can afford falls, and the cost of a wasted trial rises with it.
An adaptive approach to this problem instead treats evaluation as sequential measurement: continue sampling where uncertainty remains high, and stop when a pre-specified level of precision is met.
One established response to this measurement problem is to shrink the set of individual tasks in the benchmark. Adaptive item selection methods pick the most informative tasks for each model, and skip the rest. However, these methods typically need a pre-calibrated bank of tasks with known difficulty, which may not exist for newly developed or safety-critical benchmarks.
optstop needs no such bank, and every task in an evaluation stays eligible: sampling stops once a part of the evaluation is precise enough, whether that is a single item or the whole grouping.
How optstop works
During an evaluation, optstop tracks the statistical range within which the current estimate of a model’s performance lies (known as the credible interval). It tracks this at two levels: repetitions within a task, and tasks within a grouping, each with its own target for level of precision or stabilisation.
At the grouping level, a hierarchical Bayesian model combines information across tasks, while allowing their individual performance to differ. This accounts for the unequal sample sizes created when some tasks stop earlier than others.
optstop stops sampling on either of two rules:
1. Precision. The credible interval is narrow enough.
2. Stabilisation. The interval has stopped changing, so the data has nothing more to tell us.
If neither rule is met, the grouping simply runs to its full planned budget, so a genuinely ‘noisy’ estimate is not stopped early, but is sampled to the end.
Stopping is most dangerous when a model succeeds on an evaluation only rarely, because a single rare success can be the whole point of the test. A conservatism mechanism in optstop guards against this, demanding more data when estimated success falls below 1%.
Every stopping decision is also inspectable: optstop records and reports all stopping decisions and rationales. This record lets you check a reported saving, rather than take it on trust. (Each rule and safeguard is validated separately in the accompanying paper).
Optimal stopping in action
We tested optstop on three different types of scoring typical in benchmarks – binary (e.g. pass/fail), ordinal (e.g. rubric ratings from 0-10), and continuous (e.g. 0-100%), across three levels of expected model performance (low, medium, and high). These tests were run on public benchmarks, including MATH, GPQA Diamond, and WritingBench.
We found that under every condition, optstop’s early stopping saved between 57% and 97% of planned trials, without impacting score estimates (see Fig. 2).
Of course, if running a leaner design with fewer trials, the number of trials saved may be lower, but when each trial is long and expensive, cutting just a few can still yield substantial resource savings.
Trying it without risk
optstop allows for careful, incremental adoption to offset any perceived risks:
- Post-hoc: an evaluator runs it on a completed evaluation and confirms that the stopped estimates match the full-run estimates.
- Shadow mode: the evaluation is run as normal, while
optstopreports the saving it would have made, with the evaluator’s full dataset preserved.
- Live: the evaluator switches on live stopping once satisfied by the shadow mode numbers.
Setup is light. If using Inspect, optstop needs just a few lines in the eval configuration. The main practical requirement is that tasks are presented in a randomised order, so that early estimates are not biased by the order of the trials in the queue – and Inspect supports this.
Beyond fixed samples
As pressure mounts on evaluators to keep pace with the rate of model releases and ensure that frontier performance is meaningfully captured by evaluations, we position optstop as a tool to not only boost the efficiency of evaluation throughput, but tie it directly and rigorously to confidence in our conclusions.
optstop has been designed to be light-touch, low-risk, and flexible to a wide range of evaluation setups. By releasing it as part of our existing open-source framework, we hope that we can help other evaluators automate the redirection of evaluation resources to where uncertainty remains.
We have been exploring optstop’s use cases at AISI as one potential method to help tackle evaluation bottlenecks, and plan to develop and release more tools like it in future.
You can access the optstop package here.
Read the full paper here.
