OpenAI's GPT-6.1 Sol Scores 86.6% on COBOL Migration Test

OpenAI released test results on Oct. 19 showing that its newer coding models performed much better on tasks involving COBOL and other older programming languages. These languages are important to companies that maintain software used to manage financial records.

In MultiLangBench, the models had to build a database engine that worked with SQLite in each of 20 programming languages. When allowed to use their highest level of reasoning, GPT-6 Astra passed an average of 87.4% of the tests. GPT-5.6 Sol passed 26.8%.

The difference was also clear in three of the languages tested. In COBOL, GPT-6 Astra passed 72.1% of the tests, while GPT-5.6 Sol passed 10.9%. In Fortran, their scores were 83.5% and 21.4%, respectively. In Visual Basic, they scored 89.4% and 27.5%.

A separate test checked whether models could change COBOL programs into other programming languages without changing how the programs worked. GPT-6.1 Sol passed 86.6% of these tasks using a low level of reasoning effort. GPT-5.6 Sol passed just 5.4%. The estimated cost of using the API for the newer model was 39% lower in this test. When GPT-6.1 Sol used a higher level of reasoning effort, its pass rate rose to 97.9%.

To pass a migration task, the new program had to produce exactly the same results as the original program in every hidden test. This is a stricter standard than simply checking whether the translated code runs. A program may run without errors but still give a different answer.

These differences matter in software that handles financial records. For example, changes in rounding numbers or handling missing records can change the results, even if the replacement program seems to work. The original program may also rely on business rules and other systems that a coding test does not check.

The 39% cost difference is based on estimated API use for the COBOL test. It does not show the full cost of replacing a system that is already in use. Before putting a replacement into real-world use, experts must check the software's business rules, its connections to other systems, and how it behaves.

IBM offers a tool that changes COBOL programs into Java and creates tests to compare the new programs with the originals. Anthropic has also described a way to examine and move COBOL systems using Claude Code. OpenAI has released the MultiLangBench setup and tests used for its defined coding tasks.