ScarfBench: Benchmarking Autonomous Agents on Enterprise Java Framework Migration

The Enterprise Legacy Impasse

The modern Fortune 500 balance sheet carries an invisible, compound liability: tens of billions of lines of legacy Java. While frontier AI models achieve high pass rates on synthetic Python snippets and isolated algorithmic puzzles, the core infrastructure of global financial systems, telecommunications, and industrial logistics remains anchored to archaic runtimes—predominantly Java 8 and 11, running on end-of-life frameworks like Spring Boot 2.x and historical Java EE namespaces.

Upgrading these codebases is not an exercise in syntax translation. It requires navigating multi-module Maven and Gradle builds, resolving transitive dependency incompatibilities, rewriting dynamic reflection calls, and managing structural changes dictated by the javax.* to jakarta.* namespace migration. Historically, this technical debt sustained multi-billion-dollar IT services contracts, where systems integrators billed between $150 and $280 per engineering hour to execute mechanical, high-risk refactoring.

IBM Research’s introduction of ScarfBench establishes an empirical benchmark that evaluates autonomous coding agents against the realities of enterprise Java framework migrations. By moving the evaluation horizon away from greenfield algorithmic generation toward stateful, multi-file framework transformations, ScarfBench provides engineering leaders and autonomous enterprise architects with actionable telemetry on what current frontier agents can realistically automate—and where their reasoning breaks down.

Technical Mechanics of ScarfBench

ScarfBench does not rely on unit-test stubs. It instantiates authentic, multi-repository enterprise environments characterized by deeply nested dependency graphs, customized plugins, and heterogeneous architectural patterns.

The benchmark evaluates agents across three primary migration vectors:

1. Runtime Upgrades: Transitioning systems from Java 8/11 to Java 17/21 LTS, requiring language feature modernizations and module-path alignments.
2. Framework Evolution: Migrating from Spring Boot 2.x to Spring Boot 3.x, demanding structural reconfigurations of security filters, observability pipelines, and database connection pools.
3. Namespace Harmonization: Executing the Jakarta EE transition, where the standard package namespace shifted from javax.* to jakarta.*, breaking binary compatibility across hundreds of downstream libraries.

The Agent-Execution Interface

Agents evaluated within ScarfBench operate through a constrained tool harness containing workspace navigation, abstract syntax tree (AST) editing primitives, and terminal access to execution tools (mvn clean compile, mvn test). The interaction budget is bounded by both step counts and token limits.

To succeed, an agent cannot simply search and replace string patterns. A migration patch must satisfy strict build-lifecycle contracts:

P_success = C_build(R') ∧ (T_pass(R') / T_total(R) == 1.0) ∧ (Δ_behavior == 0)

Where R' represents the mutated repository, C_build is the boolean compiler status under target JDK/framework specifications, T_pass represents the preserved passing suite of enterprise regression tests, and Δ_behavior confirms zero unhandled semantic divergence at runtime.

Empirical Realities: The Inversion-of-Control Bottleneck

Data from ScarfBench exposes a profound divergence between syntactic fluency and semantic execution. While leading frontier architectures—including Claude 3.5 Sonnet, GPT-4o, and specialized coding models—demonstrate high competence in localized syntax patching, their performance degrades when confronted with the emergent properties of enterprise Java.

Quantitative Performance Metrics

Empirical evaluations conducted across the ScarfBench repository suite reveal the operational limits of contemporary agent architectures:

• End-to-End Autonomous Resolution Rate: Frontier models achieve complete, unattended migration (clean build, all tests green, zero runtime degradation) on only 31.8% of enterprise repositories. When restricted to simpler, single-module projects, this rate improves to 54.2%, but collapses to 11.6% in multi-module enterprise codebases exceeding 500,000 lines of code (LOC).
• Inference Latency & Budget Burn: The median successful migration episode requires 42.6 minutes of agent-environment REPL loops, consuming an average of 186,000 tokens across context windows, with iterative compiler feedback loops accounting for 73% of total execution time.
• The Silent Semantic Degradation Rate: In 41.2% of cases where an agent achieved a successful compile (BUILD SUCCESS), dynamic integration tests failed due to misconfigured Inversion of Control (IoC) bindings, unmapped configuration properties, or broken reflection targets.

These numbers demonstrate that compilation is a poor proxy for correctness in enterprise software. An agent that relies exclusively on compiler errors as its primary feedback signal produces code that builds cleanly but fails silently under production traffic.

The Unit Economics of Agentic Modernization

The business case for autonomous software refactoring is not about theoretical speed; it is governed by direct unit economics. Understanding the shift from human-executed migrations to agent-orchestrated workflows requires evaluating the operational expenditure per thousand lines of code (KLOC).

For a Tier-1 financial institution maintaining an active footprint of 12 million lines of legacy Java, a comprehensive Spring Boot and Jakarta migration historically required an 18-month engagement costing between $4.5M and $7.2M.

Under an autonomous agentic framework operating at ScarfBench baseline performance:

• Token Compute Cost: Assuming frontier model pricing at $3.00 per million input tokens and $15.00 per million output tokens, the pure inference cost per KLOC averages $1.14.
• Human-in-the-Loop Verification Cost: Given the 31.8% fully autonomous pass rate, human operators spend an average of 12 minutes per KLOC auditing failures, resolving reflection anomalies, and verifying edge cases. This human verification overhead adds $4.20 per KLOC.
• Net Unit Cost: Total migration cost drops to $5.34 per KLOC—an 86% reduction in direct capital expenditure, combined with an order-of-magnitude compression in deployment schedules.

Bridging Deterministic Verification and Algorithmic Governance

The primary operational obstacle highlighted by ScarfBench is not model intelligence—it is the absence of comprehensive verification layers. When an agent modifies an enterprise codebase, it alters not only functional code paths, but also the structural discoverability, metadata hygiene, and runtime footprints of the underlying systems.

This exact dynamic governs autonomous systems across industries. Whether an agent is refactoring thousands of Java classes or an autonomous brand engine is deploying programmatically generated enterprise web assets, unverified agentic generation creates catastrophic latent debt.

In digital infrastructure and search landscape governance, platforms like SEOdiag represent the critical external verification layer required for agentic scale. Just as ScarfBench executes deterministic regression suites to ensure an autonomous coding agent has not broken runtime contracts or corrupted internal dependencies, SEOdiag provides deep diagnostic auditing to ensure programmatic architectures, machine-generated structures, and algorithmic brand assets adhere to deterministic, citation-ready technical standards.

Autonomous enterprises cannot rely on self-reporting generative agents. Without independent, automated verification systems capable of validating structural integrity, semantic accuracy, and performance viability, agentic speed simply accelerates technical entropy.

Architecture for the Enterprise Autonomous Operator

To bridge the performance gaps identified in ScarfBench, enterprise engineering organizations must reject simplistic “single-prompt” migration architectures. Operating autonomous migrations at scale requires a multi-agent harness decoupled into specialized roles:

1. **The Static