OpenAI reports 87% coding benchmark pass rate across 20 programming languages

OpenAI says its AI models have become much better at writing and updating software in programming languages that are not used as widely. In tests across 20 languages, GPT-6 Astra passed an average of 87.4% of tests, compared with 26.8% for GPT-5.6 Sol.

The results come from MultiLangBench, a new test created by OpenAI to check whether its models can build working software in different programming languages. A separate test checked whether the models could move existing COBOL programs into newer languages while keeping their behavior the same.

The work looks in part at languages such as COBOL, Fortran and Visual Basic. AI models have less publicly available code and documentation to learn from for these languages than they do for widely used languages such as Python and JavaScript.

Even though these languages are old, they are still used in important systems. Figures cited by OpenAI say COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions and 95% of ATM transactions. Fortran is still used for tasks such as weather forecasting and checking nuclear reactor safety.

Updating these systems can be difficult. Documents may be missing, the people who first wrote the software may no longer be available, and important business rules may exist only in the code itself.

GPT-6 Astra improves in all 20 languages tested

Instead of asking AI models to solve small, separate coding tasks, MultiLangBench asks them to build a database engine that works with SQLite from scratch, using a specified programming language.

The models must organize the code, manage information, deal with errors and add features without breaking things that already work. OpenAI then checks the database files they create against its reference version of SQLite.

When using its highest reasoning setting, GPT-6 Astra passed an average of 87.4% of the tests. It improved in every language included in the test.

Some of the biggest improvements were in languages where the earlier model had struggled:

• COBOL rose from 10.9% to 72.1%.

• Fortran rose from 21.4% to 83.5%.

• Visual Basic rose from 27.5% to 89.4%.

• OCaml rose from 25.6% to 85.8%.

Astra scored above 83% in 19 of the 20 languages. COBOL was still the hardest language for it.

OpenAI also tested GPT-6 Luna, a lower-cost option. It passed an average of 40.3% of tests, compared with 26.8% for GPT-5.6 Sol. The estimated average API cost was $0.77 per sample for Luna, compared with $3.48 for GPT-5.6 Sol.

This means the estimated cost was 78% lower for the lower-cost model comparison. However, Luna's overall test results were still well below Astra's.

A separate test finds 97.9% accuracy for COBOL migration

OpenAI's second test looked at a different task: whether AI could convert existing COBOL programs into C++, Java, Kotlin, Python or Rust without changing what they do.

This means more than making sure the new code runs. For example, in financial systems, changes to rounding rules or the way missing records are handled can change transaction results, even if the rewritten software runs without errors.

To test this, OpenAI used hidden tests that required the converted programs to produce exactly the same observable output, byte for byte. A conversion counted as successful only if it passed every test.

With its low reasoning setting, GPT-6.1 Sol achieved 86.6% accuracy, compared with 5.4% for GPT-5.6 Sol using the same setting. It also cut estimated API costs by 39%.

With a higher reasoning setting, GPT-6.1 Sol reached 97.9% accuracy.

These numbers show how the models performed on OpenAI's own tests. They do not prove that the models can successfully update real banking systems or other critical systems. OpenAI says experts must still review and test any updated software against existing business rules and system dependencies.

OpenAI is publishing the MultiLangBench test setup, test suite and results for each language on GitHub. This lets developers inspect the tests and check performance in the languages they use.

OpenAI's latest models are available through its API for coding tasks. The company is continuing to ask developers and businesses for feedback about languages and workflows that need further improvement.