Introducing LEAPBench

Why measuring learning efficiency matters

Trial and error is a fundamental part of scientific discovery. A researcher first proposes a hypothesis, then designs an experiment, then analyzes the feedback, and finally refines the design for the next round. As LLMs integrate into autonomous laboratories, there is a general intuition that their vast domain knowledge and reasoning capabilities will allow them to arrive at optimal designs with more efficiency than traditional experimental approaches.

However, most benchmarks evaluate LLMs on single-shot correctness and final outcomes at a fixed horizon. If two separate models identify the same optimal configuration after 30 attempts, that kind of benchmark would treat them as equivalent. All the labor and budget constraints of running a company or laboratory mean that each iteration comes at a non-trivial cost, and the difference between 7 and 27 attempts could completely change the budget and timeline of your operation.

To evaluate this dimension of performance, we built LEAPBench (Learning Efficiency in Adaptive Processes): a benchmark intentionally designed to measure the in-context learning efficiency of LLMs across iterative scientific design tasks.

‍

‍

How LEAPBench works

The evaluation framework consists of 55 sequential optimization tasks, which include 45 biology tasks (protein engineering, cell culture optimization, nanoparticle formulation, gene editing, fermentation) alongside 10 tasks in the field of education derived from peer-reviewed randomised controlled trials, which serves as a validation set apart from the biology tasks. The latter are drawn from over 2,700 experimental data points from more than 370 published biology studies.

For each task, we trained a supervised regression model on the published literature datasets to act as a surrogate oracle. During evaluation, the LLM is given a parameter space and must propose designs iteratively. After each proposal, the oracle returns a scalar reward without revealing the underlying function. The model then receives this feedback in its context window and has up to 30 iterations to optimize the target objective.

To really evaluate learning efficiency, we compare the standard final outcome metric (which measures the best-so-far value at a given iteration) against a trajectory-integrated metric that calculates the area under the best-so-far-curve.

‍

‍

What we found

Our evaluation of contemporary LLMs reveals that the metric we choose significantly impacts how we rank model capabilities. On the 45 biology tasks, switching from final outcome scoring to trajectory scoring changes the top-ranked model on 53% of the tasks. In half of these cases, the shift also resulted in a rank-3-or-deeper reordering, which indicates that standard metrics usually overlook meaningful differences in learning efficiency.

On many of these tasks, the trajectory winner and the outcome winner eventually covered to within 1% of the same final score by their final iteration, but the trajectory winner reached that optimum much faster, with a median of 7 iterations compared to 15 iterations for the outcome winner.

We also compared the models against a classical GP-UCB baseline, which has no domain knowledge and operates purely on numeric feedback. Under the same conditions, where both the classical baseline and the domain-agnostic LLM received the exact same unlabeled parameter space, no frontier LLM consistently outperformed the GP-UCB solver.

The highest performing model under this condition was Gemini 3.1 Pro, which cleared the baseline on 64% of tasks, while the lowest was Llama 3.1 8B at 11%. None of the evaluated models cleared the 50% null hypothesis line with statistical significance across the full panel of tasks.

‍

‍

The language trap

We also examined whether exposing models to domain terminology rather than using generic variables helps or hinders their optimization trajectories. When comparing domain-aware prompting with domain-agnostic prompting on identical parameter spaces, we do observe an asymmetry. While domain-aware prompting improved performance on the education validation tasks where literature consensus aligns with the correct mechanism, it more often than not hindered performance on the biology tasks.

This drop in performance becomes even more pronounced when the standard historical consensus in the literature diverges from the empirical optimum. Across six specific literature-divergent biology tasks where the most common literature configuration is suboptimal, domain-agnostic prompting selected the optimal configuration more often than domain-aware prompting. When provided with semantic domain context, models routinely biased their proposals toward the choice that appears most frequently in the literature, even when the feedback indicates that a less common configuration would yield a higher reward.

‍

‍

Improving efficiency

To explore whether learning efficiency is a trainable property, we fine-tuned Llama 3.1 8B-IT using offline GRPO on more than 5,000 trajectories from 29 biology training tasks, with the cumulative trajectory metric as the reward target. The resulting model demonstrated improved learning efficiency on 14 of 21 held-out tasks, which included both held-out biology and unseen education tasks. This suggests that trajectory-integrated metrics can serve as effective reward functions for reinforcement learning.

As LLMs transition into more real-world scientific work, evaluating how efficiently they learn from feedback will prove to be just as important as evaluating what they eventually achieve. By focusing on trajectory-based metrics and addressing biases rooted in domain-specific language, we’re able to train models that adapt better to the realities and constraints of scientific discovery and enterprise tasks.

‍