Anthropic Research Confirms AI Still Cannot Conduct Genuine Real-World Scientific Research

Advanced coding agents can turn machine learning research into workflows that computers can run automatically. However, most existing studies focus on well-organized code projects, individual tasks, or computer science. For scientific research across different fields, where training and testing methods are often not well established, researchers still lack a systematic way to measure whether these agents can truly improve machine learning models. Testing agents directly on original research code is also unreliable. If a project does not clearly define its evaluation measures or use standard data splits, agents may take advantage of weaknesses in the tests to show improvements that do not reflect real gains in model performance.

To address this problem, the Anthropic team introduced Agent4Sci, an evaluation framework that tests how well agents can improve machine learning models for scientific tasks over multiple steps. It covers 11 scientific fields, 20 benchmarks, and 25 tasks.

Paper link: https://arxiv.org/abs/2610.28763

In the experiment, four advanced agents improved results on the 20 benchmarks. Claude Opus achieved an average relative improvement of 10.4%. However, after studying the steps the agents took, the researchers found that most of the gains came from standard machine learning engineering techniques rather than new scientific insights. After the team added three plugin skills, Opus's average relative improvement rose further to 20.1%. This work was selected for the Evaluation and Datasets track at NeurIPS 2026.

Building Agent4Sci involved two main steps: first, making scientific research tasks consistent and easier to compare; second, using a three-stage process to separate model improvement, evaluation, and auditing. This approach limits opportunities for cheating that appeared in early experiments.

The first step involved four rounds of screening, followed by manual standardization. The research team reviewed benchmarks in four stages: classifying scientific fields, automatically screening candidates, manually checking them, and conducting a detailed review. For example, from papers submitted to the NeurIPS 2025 Data and Benchmarks track, the team selected 10 benchmarks from 497 candidate papers, rejecting about 98% of the candidates. Each selected task had to pass a test showing that the researchers could reproduce the paper's baseline results. The team also had to standardize how data was split, how the software environment was set up, and how models were trained.

The second step was a three-stage evaluation process. First, the improvement agent receives the research paper, code, training data, and validation data. It changes the code, trains the model, and submits its results within a fixed computing budget, but it cannot access the test set. Next, a separate evaluation agent uses a standard procedure to calculate the final scores on the test data that was kept hidden. Finally, an auditing agent checks the activity logs and submitted results for attempts to exploit the evaluation system or other forms of research misconduct.

Figure | The Agent4Sci evaluation process, in which the agents responsible for improvement, evaluation, and auditing work separately from one another

This design directly addresses weaknesses in the evaluation process that appeared in early experiments. The team also used auditing agents from three model families, Claude, Codex, and Gemini, to independently check some of the activity logs. Their conclusions agreed, and none of the runs was flagged. However, separate evaluation and auditing cannot guarantee that every hidden attempt to exploit the system will be detected.

The experiment compared Claude Opus 4.6, Claude Sonnet 4.6, Gemini 3.1 Pro, and Codex (GPT-5.4). Each model ran through its official command-line tool using the highest inference setting. Every experiment had access to 8 H200 GPUs and a 12-hour time limit. No human made decisions during the individual experiments. The four agents achieved average relative improvements of 10.4%, 9.6%, 6.8%, and 4.4%, respectively. No single agent improved every benchmark, but at least one agent improved each of the 20 benchmarks. Individual improvements ranged from 1.1% to 64.4%.

Figure | Average relative improvements achieved by four advanced agents on Agent4Sci

Examples show that improvements can come from both general model tuning and changes designed for a particular task. In the PFDelta task, which estimates power flow, Opus reduced the average power balance loss by 64.4%. It did this by adjusting extra loss terms, giving greater weight to the mean squared error loss, and using cosine annealing with a warm-up period to adjust the learning rate. In the cone detection task from Mars-Bench, Opus replaced the original RetinaNet model with Faster R-CNN, which better suited the task. This raised the mean average precision (mAP) from 16.5 to 30.4.

However, examining the agents' actions revealed limits behind these gains. The agents often used established techniques, such as changing learning rates, making models deeper or wider, replacing backbone networks, and adjusting weight decay. They rarely made changes aimed directly at the specific scientific problem. The agents also worked in noticeably different ways. Opus repeatedly checked logs and ran further experiments. Codex tended to submit long-running scripts early and then wait for them to finish. Gemini's approach fell between these two styles.

To address repeated problems found in the agents' work, the researchers created three simple plugin skills: one for developing research ideas, one for managing experiments, and one for using scheduled checklists. These skills nearly doubled the average improvement of the two agents. Opus improved 18 of the 20 benchmarks. Its improvement on PFDelta rose from 64.4% to 91.0%, while its improvements on MassSpecGym and CarbonGlobe reached 39.5% and 35.3%, respectively. The new PFDelta solution showed more physical reasoning. The agent recognized that the power balance loss could be calculated from Kirchhoff's law without needing labeled data. It made this loss the main training objective, used predictions from a graph neural network (GNN) to provide a starting point for solving the alternating-current (AC) power flow equation during inference, and gradually refined the voltage variables to bring the physical error closer to zero.

Figure | Comparison of improvement rates for each benchmark, with and without the added skills

The researchers said that although most performance gains came from machine learning engineering rather than scientific reasoning, Agent4Sci turns the ability of automated research agents to improve models into something that can be tested again and independently checked. The framework measures not only whether agents improve performance scores, but also tries to tell whether those gains come from established engineering techniques or from understanding the scientific problem itself.

In the future, an important way to measure automated scientific research will be to see whether coding agents can move beyond running experiments more efficiently and start proposing research ideas that offer greater value.

For more technical details, please refer to the original paper.

This article is from the WeChat Official Account "Academic Headlines" (ID: SciTouTiao), written by ZHANG Jiayi and published with permission from 36Kr.