AI COBOL to Java migration: the bugs come with it

If you are watching AI COBOL to Java migration and waiting for the moment it becomes safe, the interesting news this week is not a better translator. It is a validator. A paper posted to arXiv on 30 July 2026 describes an agentic loop that proves a generated Java program behaves the same as the COBOL it came from.

Same. Not correct. That distinction is the whole story, and most people reading the headline are getting it backwards.

---

🔍 What the paper actually built

The paper is Agentic Method for Deterministic Validation of Legacy Code Migration by Andras Ferenczi, Jordan Docherty, Mariya Bessonov, Matthew Findlay and Krishna Lingamneni. I am commenting on it, not reproducing it, so go read the abstract yourself.

Their method is called the Locksmith Loop. The shape of it:

1. Stand up two runtime environments — the original COBOL and the generated Java — both instrumented with mocks.
2. Run both off-mainframe, on commodity hardware. No z/OS licence required to test.
3. Run a Witness Search over the mocked inputs, hunting for values that push execution into unvisited branches.
4. Apply parity-preserving mutations to those inputs to widen coverage without changing expected behaviour.
5. When the search stalls, an analyser identifies a Locked Paragraph — the specific condition blocking deeper exploration.

The clever part is step 5. Most coverage tooling tells you that you plateaued at some percentage. This tells you which gate is holding the door shut.

---

🐛 Bug-for-bug parity is the goal, not the failure

The Hacker News headline that carried this paper around framed it as "bugs included," as if the AI fumbled. That framing is the submitter's, not the authors'. And it inverts the engineering reality.

When you migrate a forty-year-old batch job that posts interest to savings accounts, the COBOL is the specification. There is no other one. The analyst who wrote the requirements retired in 2003. If the COBOL rounds a half-cent down in one obscure branch, and your shiny Java rounds it up, you have not fixed a bug. You have created a reconciliation break that some poor operations team will chase for three weeks.

Migration goal	What you validate against	Failure mode
Behavioural parity	The old system's actual output	You inherit legacy quirks
"Correctness"	A written spec nobody has	You silently change business logic
Rewrite from scratch	New requirements	Multi-year project, high abandon rate

Key takeaway: In legacy migration, the old program is the test oracle. Preserving its bugs is a feature of the process, and you fix them after the cutover, as separate, reviewable changes.

---

📊 The numbers, and what they do not cover

Here is everything the paper actually reports, so you can calibrate the hype yourself:

Metric	Reported
Case studies	3
Open-source programs	2
Internal production-like program	1
Program size range	430 to 4,114 source lines
Branch coverage (internal program)	91.90%
Coverage (open-source programs)	"nearly complete"

Three programs. The largest is roughly 4,114 lines. A real bank core is measured in millions of lines across thousands of programs, with copybooks, VSAM files, CICS transactions and JCL glue that this evaluation does not touch.

That is not a criticism of the paper. It is a paper, not a product. But if a vendor walks into your office next quarter waving "91.9% verified migration," ask them which of those three numbers they are borrowing from.

---

🛠️ How to steal this method without owning a mainframe

You almost certainly do not have COBOL. You almost certainly do have a rewrite: a PHP monolith going to Node, a jQuery admin panel going to React, a stored-procedure pile going to application code. The method transfers directly, and it costs nothing but discipline.

• Keep the old system runnable. The single most common mistake I see in local rewrites is decommissioning the old code before the new one is proven. Without the original running, you have no oracle.
• Mock the boundaries, not the logic. Database, clock, file system, network. Then both versions become deterministic functions of their inputs.
• Diff the outputs, not the code. Feed identical inputs to both, capture stdout / JSON / row sets, and compare. A plain text diff is a legitimate regression suite. Ours is free if you want one: text diff checker.
• Chase the uncovered branch, not the coverage number. The paper's Locked Paragraph idea is really just: when coverage stalls, name the blocking condition out loud and hand-craft an input for it.
• Log every parity break as a ticket, not a fix. Decide deliberately whether each difference is a bug you are inheriting or a bug you are correcting.

If you are learning Java as part of a modernisation job and want somewhere to test snippets without setting up a JDK on a slow laptop, our online Java compiler runs in the browser.

---

💡 What this means for you

There is real money in this for Sri Lankan teams, and it is not in writing the translator.

Local banks, insurers and government departments run systems old enough to vote, and the offshore modernisation work that flows through Colombo IT services firms is mostly the same shape: someone has a legacy system, an LLM can produce plausible target code in an afternoon, and nobody can sign off that it is safe. The bottleneck moved. Generation got cheap; assurance did not.

The billable skill in 2026 is not "I can convert COBOL to Java." It is "I can prove this conversion did not change behaviour, and here is the evidence."

Three things worth doing this month:

1. Learn differential testing properly. It is old, unglamorous and applies to every rewrite you will ever touch.
2. Get comfortable reading branch coverage reports in whatever stack you use. The paper's whole contribution is a smarter way to attack a coverage plateau.
3. Stop treating "the AI wrote it" as a review status. The output of a migration agent is a patch from an unfamiliar contributor. Review it that way.

The headline says AI moved the bugs across. Good. Moving the bugs across, on purpose, with proof, is the hard part.