AI Beats Human Experts at Research Experiments: TasteVal Benchmark Confirms

A new AI benchmark has provided the first direct measurements suggesting that the most advanced AI systems have surpassed human experts at two key parts of scientific research: designing experiments and understanding their results. These systems reached expert-level performance using less than half as much computing power and about one-thirtieth of the cost per attempt compared with a human researcher. The findings come from TasteVal, a benchmark created by P-Zero Research and published on October 5. It measures what researchers call "research taste": the mix of skills that helps scientists choose and carry out useful research, rather than simply complete tasks well.

The best result came from Anthropic's Claude Opus 5.5, which achieved a compute multiplier of 2.30 compared with a human expert baseline made up of 24 researchers recruited from OpenAI, Google DeepMind, Nvidia, and Microsoft Research. The 95 percent confidence interval for this result ranges from 1.15 to 4.37. This range is wide enough to include both a small advantage and a much larger one, but even its lowest value is above 1.0, the point where the AI and human performance would be equal under this measure. Across several generations of advanced AI models, the ability measured by TasteVal has roughly doubled every three months since December 2025. Before then, it took about 14 months for the ability to double. The benchmark's authors consider this sharp change one of their most important findings.

The benchmark deliberately leaves out one part of research judgment: deciding which problem to study in the first place. Its authors identify this as an important remaining ability that has not yet been measured by TasteVal. This gap limits what the results can tell us about how close AI is to automating all scientific research.

What Recursive Self-Improvement Means

Recursive self-improvement (RSI) is the point at which an AI system can train a more capable successor without any human involvement. This means more than automating individual research tasks. It means creating a closed loop in which a smarter model helps build an even smarter model, potentially speeding up progress that might otherwise take decades. The idea can be traced back to I.J. Good's 1965 paper on ultraintelligent machines and Alan Turing's earlier proposal for a "child machine." Eliezer Yudkowsky helped popularize the term "seed AI" to describe the form of recursive improvement in which an AI helps create increasingly capable versions of itself.

The question of when RSI might become possible has become more urgent as researchers have found ways to measure progress toward it. Anthropic publicly reported that AI models generated more than 80 percent of the code merged at the company by May 2026. In September 2026, OpenAI announced that its research agents were doing about 3.1 agent-workdays of work for every eight-hour human workday. However, more than half of the tasks that took four to eight hours still needed human intervention. A paper published on September 28 and co-written by researchers including Geoffrey Hinton, Yoshua Bengio, Jakub Pachocki, and Jack Clark found that AI's share of research and development work done with limited human supervision had grown from 1 percent to 26 percent between March and August 2026.

TasteVal adds to this picture by providing the first benchmark specifically designed to measure the judgment involved in research. It tests the ability to design experiments and understand their results, rather than simply measuring how quickly AI can complete tasks or write code.

How TasteVal Measures Research Judgment

TasteVal is built around a clear distinction between three connected parts of scientific judgment, according to its authors: deciding which problems are worth studying, designing experiments to answer those questions, and correctly understanding what the results mean. TasteVal measures the second and third parts. Choosing the problem to study is deliberately left out of this version of the benchmark.

To separate research judgment from coding skill, TasteVal uses a system that divides the work between a Researcher and a Coder. The AI model being tested acts as the Researcher: it suggests experiments and draws conclusions from the results. A separate, fixed Coder agent handles all the programming on a single Nvidia H100 GPU. Each test is limited to 40 GPU-hours of computing time or 120 hours of real-world time, whichever comes first. This setup prevents a model from making up for poor experiment choices by writing code faster or more efficiently. The coding work stays the same across all models.

The benchmark includes eight tasks based on advanced AI research. These cover choosing training data, pretraining, fine-tuning language models, modeling preferences, and defending against adversarial prompts. The human comparison group included 24 experts who had recently held research jobs at OpenAI, Google DeepMind, Nvidia, and Microsoft Research. At least two experts were assigned to each task. TasteVal uses the best human result for each task as the target for AI to beat, rather than the average human result. This creates a demanding comparison, but it also means the benchmark compares AI with the lower limit of expert-level performance rather than the typical results of researchers.

Performance is measured using a compute multiplier. This is the amount of computing power a human expert used to complete a task divided by the amount the AI model needed to achieve the same score. A result above 1.0 means the model reached expert-level performance with less computing power than the human. Claude Opus 5.5's multiplier of 2.30 means it achieved expert-level results using about 43 percent of the computing power used by a human expert, at roughly one-thirtieth of the cost per attempt.

The Phase Transition Since December 2025

One of TasteVal's most important findings is not just how well AI performs, but how quickly its performance is improving. Before around December 2025, the research judgment score of advanced AI models doubled about every 14 months. This matched a pattern of gradual improvement. Since then, the time needed for the score to double has fallen to about 3.0 months. The 95 percent confidence interval for this estimate is 1.7 to 5.0 months.

The TasteVal paper does not explain what caused this sudden change. One possible explanation, which the paper does not state but which fits the timing and wider industry data, is that AI systems began using other AI systems more often to help with research. Anthropic reported that the share of merged code generated by AI rose from a low single-digit percentage in early 2025 to more than 80 percent by May 2026. A paper from Cambridge researchers also found that AI's share of automated research and development work rose from 1 percent to 26 percent between March and August 2026. Together, these findings suggest a possible feedback loop: AI helps researchers develop AI, which may then make AI systems better at research themselves. If this is what is driving the faster improvement measured by TasteVal, progress might not slow simply because human researchers become less productive. This possibility makes the sudden change in the improvement rate especially important.

The TasteVal paper also warns that its results may make the trend look stronger than it really is. Because the benchmark's tasks are quick and inexpensive to score, they are the kind of tasks AI labs can easily optimize their models to perform well on. As a result, the steep improvement curve might partly reflect efforts to improve performance on this specific benchmark rather than a similar increase in underlying ability.

Where the Benchmark Deliberately Stops

TasteVal does not measure every part of research judgment. Choosing which scientific problem to investigate from a huge range of possible directions is often considered the deepest form of scientific judgment. TasteVal deliberately leaves this ability outside its scope.

Other research provides evidence of this gap. A 2026 study of how AI behaves during post-training found that AI agents working on research tasks tended to stick to their original strategy early in a task. Only 2.1 percent of training runs changed their chosen approach. Experience helped the agents carry out their plans, but it did not help them reconsider those plans. The researchers identified a missing ability: deciding to reopen a decision and consider another approach. This is exactly the kind of higher-level judgment that TasteVal leaves out by giving models a research goal in advance.

Anthropic's alignment team published a separate benchmark called TASTE on August 28, providing another useful comparison. This TASTE benchmark, which is different from TasteVal, measures whether AI models can choose the more promising of two research proposals about AI safety. It directly tests the ability to choose a research problem, which TasteVal does not measure. The best-performing AI model tested, Claude Fable 5, scored about 60 percent, compared with an estimated 77 percent for human researchers. This is still a substantial gap. Taken together, TasteVal and Anthropic's TASTE suggest that AI has probably reached or come close to expert level at designing experiments and interpreting results, but has not yet reached that level at choosing which research directions to pursue.

There is also a limitation in the human comparison group used by TasteVal. Because the benchmark compares AI with the best human result for each task rather than typical human performance, it measures AI against a lower limit for expert performance. It does not measure against the highest results the world's best researchers might achieve if they had unlimited time and information.

How Clark Applied These Findings to His Forecast

Jack Clark, a co-founder of Anthropic and the author of the widely read Import AI newsletter, updated his forecast for RSI in the newsletter's 476th edition, published on October 12. Clark wrote that he now believes RSI, which he defines as a closed-loop system that can train a more powerful successor with "no human involved" in the training, is more likely to happen in 2027 than in 2028. This is a shorter timeline than the one he gave as recently as May.

In May, in issue #455 of Import AI, Clark estimated a roughly 60 percent chance of RSI by the end of 2028 and a 30 percent chance by the end of 2027. In his October update, he changed which year he considered more likely but did not revise those probability figures. He only said that the year he considered more likely had shifted. This distinction matters because Clark is forecasting a possibility, not saying RSI is certain to happen. Whether a lab pursues RSI if it becomes technically possible will depend on the availability of computing power, safety measures, competition, and regulations.

Clark pointed to two kinds of evidence for his updated view: TasteVal's findings about AI's research abilities and a wave of reports about AI systems producing what he called "creative-seeming math breakthroughs."

One important detail should be made clear. Clark co-founded Anthropic, whose model is at the top of the TasteVal rankings. P-Zero Research, the group behind the benchmark, is independent, but the best-known person using its results to make predictions about the future is connected to the leading AI lab. Clark's newsletter is his personal publication and does not represent Anthropic's official position. Clark also co-wrote the September 28 Cambridge paper on an intelligence explosion in a personal capacity, not as an Anthropic representative. He worked on it with Hinton, Bengio, and others who have called for governments to require reports on how much AI is automating research and development, independent auditors inside AI companies, and emergency measures to pause progress in AI capabilities when needed.

Clark's membership in NAIAC, his 2023 briefing to the UN Security Council on AI threats, and his decade of work on AI policy give his updated forecast importance in discussions about AI governance. His views should be considered alongside both the benchmark evidence he cites and his connection to the industry.

What the TasteVal Paper's Own Projections Show

The TasteVal paper goes beyond Clark's newsletter by exploring possible consequences using a model called the "AI Futures Model," developed by the AI Futures Project. The authors repeatedly describe their projections as "naive extrapolation rather than a forecast." They clearly warn that the figures should not be treated as predictions.

If the measured rate of improvement on TasteVal continues without slowing, the model calculates that the chance of what the paper calls a "taste-only singularity" would rise from 51 percent to 88 percent. This scenario means that improvements in research judgment alone trigger a cycle of faster and faster progress. In the same model, the middle estimate for when artificial superintelligence (ASI) would arrive moves from 2030.5 to 2028.9. The estimated chance of ASI arriving before 2030 rises from 46 percent to 69 percent.

The authors give several important warnings about these projections. The doubling rate is uncertain, with a confidence interval of 1.7 to 5.0 months. The tasks may be easy to optimize for without improving general research ability. The calculations also depend on the observed trend continuing and apply only to research judgment, which is one part of the wider picture of RSI. The figures do not account for choosing research problems, planning research over several weeks, or carrying out experiments in the real world.

The Counterarguments

The benchmark's findings are disputed. The main objections point to the same limits that the benchmark's design already acknowledges.

AI safety researcher Herbie Bradley argued that AI systems had not shown "research taste, genuine creativity, and coherent long-term research planning" at a level that would make RSI possible. His argument became an important part of the RSI debate after Clark's May essay. TasteVal directly challenges this claim about experiment design, but it does not address the third ability Bradley named: planning research over the long term in a consistent way and choosing which problems to study.

TasteVal's system, which separates the Researcher from the Coder, helps researchers measure judgment separately from coding ability. However, it also creates a setup that differs from real research. Actual researchers must decide what to build and change their plans when unexpected results appear. The post-training study found that AI agents lacked this ability to rethink their strategies. A model that performs well when given a fixed research goal may not be able to choose research directions on its own.

Ajeya Cotra at METR describes three levels of AI research ability: adequacy, when AI can complete some research tasks without humans; parity, when AI working alone matches a human and AI working together; and supremacy, when AI working alone performs better than human-AI collaboration. TasteVal's findings suggest that AI can adequately handle experiment-level tasks and may match human performance on some specific tasks. They do not establish overall parity in research. Helen Toner at CSET has argued that using AI tools in AI research does not count as RSI under the classic definition, which requires no human involvement. No current system has reached that standard.

Broader Institutional Context

Clark's revised timeline comes at a time when major AI organizations are quickly changing how they discuss the issue. Anthropic publicly acknowledged in June 2026 that RSI had not yet been achieved, but warned that it "could come sooner than most institutions are prepared for." OpenAI, which released data on research automation in September, stressed that measurable automation of research tasks does not mean RSI should be pursued without human oversight.

The Cambridge paper on an intelligence explosion was co-written by Clark, Hinton, Bengio, Pachocki, Eric Horvitz, Dawn Song, and more than 20 researchers in total. It made specific policy requests based on the data it presented. The paper's call for governments to require reports on AI research automation, place independent auditors inside AI labs, and create emergency pause measures was based on evidence that AI's share of research and development work done with limited human supervision had risen from 1 percent to 26 percent in five months. Published three weeks later, TasteVal adds a specific measurement of AI research ability to the evidence that the Cambridge paper's authors were considering.

The main change is that the debate now has more direct evidence to work with. Discussions about when RSI might arrive have mostly relied on predictions based on general AI benchmarks, including reasoning scores, coding ability, and how long systems can complete tasks. TasteVal is an attempt to measure the specific ability that matters most: whether AI can work effectively as a scientific researcher, rather than simply write code or process documents. The findings are not conclusive, the methods are disputed, and the authors explicitly describe their projections as speculative. Even so, the benchmark marks an important shift from discussing RSI mainly through broad predictions to directly measuring a key research ability. Its results deserve careful attention.

Frequently Asked Questions

What exactly does TasteVal measure, and how is it different from other AI benchmarks?

TasteVal measures two of the three parts of scientific research judgment: designing experiments to answer a given research question and drawing correct conclusions from the results. It separates the work between a Researcher and a Coder. The AI model suggests experiments, while a separate, fixed agent writes the code. This design aims to measure judgment rather than programming skill. TasteVal does not measure problem selection, or the ability to choose which research question to pursue. Most other benchmarks of AI research ability have not specifically tested this skill either. TasteVal uses a new measure called the compute multiplier. It divides the computing power needed by a human expert by the computing power needed by an AI model to reach the same score. A multiplier above 1.0 means the AI uses computing power more efficiently than the human expert.

Has AI actually surpassed human experts at AI research tasks?

For the specific tasks covered by TasteVal, which involve designing experiments and interpreting results for research goals that have already been set, the benchmark found that the best current AI model, Claude Opus 5.5, scored a compute multiplier of 2.30 and surpassed the human expert baseline used in the study. That baseline is the best result from any of the 24 experts recruited from major AI labs, not their average result. The finding applies only to these two parts of research judgment. A separate benchmark, Anthropic's TASTE safety evaluation, found that AI models scored about 60 percent compared with 77 percent for humans when choosing research proposals. This is the skill TasteVal leaves out, suggesting that AI still performs below human experts at choosing research problems.

What is recursive self-improvement in AI, and what would it actually take to achieve it?

Recursive self-improvement means an AI system can train a more capable successor without any human involvement. It is not just research that AI helps people do faster or more cheaply. It is a complete cycle in which AI builds a better version of itself. Under the definition used by Jack Clark and others, RSI would require AI systems to choose research problems, design and run experiments, understand the results correctly, and use those results to build a better AI system, all without human involvement at any stage. TasteVal measures two of these steps. Choosing problems, planning research over several weeks, and changing a research strategy after committing to it are among the abilities that have not yet been shown at the level RSI would require.

What should I understand about the doubling rate TasteVal reports before treating it as a prediction?

According to the TasteVal paper, the reported three-month doubling time for research judgment applies only to the period since around December 2025. Before that, the doubling time was about 14 months. The authors describe the recent change as a possible shift in the rate of progress, not as an established long-term pattern. The 95 percent confidence interval ranges from 1.7 to 5.0 months, meaning the true doubling time could be much longer than the central estimate suggests. The authors explicitly call their projections "naive extrapolation rather than a forecast." They also warn that the way the benchmark tasks are designed makes them especially easy for AI labs to optimize for, which could make the measured rate of improvement look faster than the underlying progress really is.