OpenAI’s COBOL migration score has made a striking leap

This week, OpenAI put COBOL in the spotlight. It announced test results suggesting that its newest models are getting much better at moving COBOL programs to other programming languages and building software in languages where models have less code and documentation to learn from.

In OpenAI’s own COBOL migration test, GPT-6.1 Sol reached 86.6% accuracy when using little reasoning effort. That was up from 5.4% for GPT-5.6 Sol, while costing 39% less. When using more reasoning effort, the newer model reached 97.9% accuracy.

But the evidence behind the two tests is not equally strong.

OpenAI has not published the details of its COBOL evaluation. COBOL stands for “Common Business-Oriented Language.” It is a programming language that AI coding tools may help people maintain or move to other languages. However, OpenAI has released the setup and test suite for MultiLangBench on GitHub. This test checks whether models can build a database engine that works with SQLite from scratch in 20 programming languages.

OpenAI’s announcement describes the results as “substantial improvements across both evaluations, delivering much higher accuracy at lower cost.”

These results could matter to teams that maintain and update older software. But because OpenAI has not made its COBOL test cases public, developers cannot examine that evaluation and check the results for themselves.

Is there finally some hope for modernizing COBOL?

OpenAI says its new COBOL migration test “tests whether a model can migrate existing programs to another language while preserving their behavior.” More specifically, it checks whether programs can be moved from COBOL to C++, Java, Kotlin, Python, and Rust.

To pass, each converted program must succeed in a series of hidden tests. These tests require the program’s observable outputs to match “byte-for-byte.” In other words, the outputs must be exactly the same. GPT-6.1 Sol is much more accurate than the model before it, showing major progress.

The measured improvements are striking, although OpenAI is not the first company to use AI to update old COBOL software.

COBOL supports many of the world’s financial transactions and business records. IBM reports that COBOL supports more than 40% of all online banking systems, handles 80% of in-person credit card transactions, and processes 95% of all ATM transactions. Despite its importance, COBOL remains a difficult language for AI coding tools to work with.

Compared with languages such as Python and JavaScript, there is less COBOL code and documentation for AI models to learn from. Experienced COBOL developers can also be difficult to find, and the documentation often fails to keep up with the systems they maintain.

AI coding gains across 20 languages

Along with its COBOL migration tests, OpenAI created a second test called MultiLangBench. It measures how well models can build software from scratch. This benchmark “tests whether a model can build a SQLite-compatible database engine across 20 different programming languages.”

First, the program receives a database and an SQL query. It then creates a changed version of the database. The standard SQLite implementation is used as the reference to check whether the resulting database contains the expected data.

Across all 20 programming languages, GPT-6 Astra — OpenAI’s leading model — passed an average of 87.4% of the tests. That was a large increase from GPT-5.6 Sol’s 26.8%. GPT-6 Astra scored above 83% in 19 of the 20 languages and improved on every language tested.

GPT-6 Luna did not perform as well as Astra, as expected. This lower-cost option passed an average of 40.3% of the tests. However, it performed better than GPT-5.6 Sol in every language tested, with an estimated API cost that was about 78% lower.

All right, then. Keep your secrets.

OpenAI reports promising results from its COBOL migration test and MultiLangBench, but benchmark scores do not tell the whole story. As of Thursday, Real-SWE 1.1, a test that uses private codebases, ranked Claude Opus 5.5 first, with a 56.8% resolution rate. An earlier version of the Real-SWE test, covered by The New Stack, had a different leader and lower scores.

Developers still cannot examine OpenAI’s COBOL migration test. OpenAI says feedback from developers and enterprise customers helped guide its work. But without the actual tests and dataset, people outside the company cannot independently check the results.

OpenAI recognizes an important limitation. It notes that “applying it to production software still requires expert review and testing against the system’s business rules and dependencies.”

OpenAI is entering an area where other companies have already been working. Anthropic explained how Claude Code could help modernize COBOL on February 23. As The New Stack reported, AWS introduced AI agents to make mainframe migrations easier in 2024.