ScarfBench: Quantifying the Limits of Autonomous Agentic Refactoring in Enterprise Java

Enterprise software modernization is rarely a greenfield architectural choice; it is a defensive capital allocation battle against entropy. Trillions of dollars in global commerce—spanning transactional banking, telecommunications clearinghouses, and supply-chain logistical backbones—rest on millions of lines of aging Java infrastructure. Upgrading these codebases across major framework boundaries (such as transitioning from Spring Boot 2.x to 3.x, or migrating javax.* packages to jakarta.* namespaces under Java 17 and 21) represents one of the largest operational drains on enterprise engineering budgets.

Until recently, AI coding assistants were evaluated on toy algorithmic challenges or isolated, single-file patches (e.g., HumanEval or SWE-bench Lite). These benchmarks obscure the structural reality of enterprise codebases: deep object hierarchies, implicit reflection-based dependency injection, brittle build configurations, and distributed runtime behaviors.

With the release of ScarfBench (Source Code Analysis and Refactoring Framework Benchmark) by IBM Research, the industry finally receives a rigorous, execution-backed metric for autonomous framework migration. The results provide an unvarnished audit of where agentic code generation succeeds, where it fails catastrophically, and what engineering architectures are required to bridge the gap between superficial syntax compliance and verified runtime execution.

The Technical Anatomy of Framework Drift

Migrating an enterprise Java application is fundamentally different from porting a script or refactoring a discrete Python service. Framework migrations are non-local transformations characterized by three structural challenges:

1. Namespace and Bytecode Breaking Changes: The transition from Java EE to Jakarta EE fundamentally altered package structures (javax.servlet to jakarta.servlet). Simple AST search-and-replace fails when transitive dependencies still pull legacy shaded jars, causing classpath collisions at class-loading time.
2. Implicit Dependency Injection Contracts: In frameworks like Spring Boot 3, reflection configurations, auto-configuration conditions, and bean lifecycle hooks were overhauled to support Ahead-of-Time (AOT) compilation and GraalVM native images. A class that compiles perfectly can silently fail during Spring ApplicationContext bootstrapping due to missing metadata or altered bean definition precedence.
3. Build Tooling and Property Manifests: Enterprise Java relies on complex build orchestrators (Maven multi-module POMs, Gradle Kotlin DSLs) paired with external configuration manifests (application.yml, cloud configuration servers). Updating code without synchronized reconciliation of POM dependency trees and property keys triggers immediate deployment failure.

Standard benchmark suites test whether an LLM can resolve an issue described in a GitHub issue ticket with an isolated test case. ScarfBench tests whether an agent can ingest a multi-module repository, interpret the structural contract of a framework transition, execute multi-file edits, and output a project that builds, boots, and passes full integration test suites.

Architectural Mechanics of ScarfBench

ScarfBench formalizes the modernization workflow into an end-to-end evaluation harness. Rather than relying on static code comparison (such as BLEU or CodeBLEU metrics, which correlate poorly with functional correctness), ScarfBench executes the modified code against containerized JVM harnesses.

The evaluation environment exposes the agent to native developer toolchains: read access to files, write access for patches, and a controlled sub-shell to invoke mvn clean test-compile or gradle test. The core metric is not simply patch accuracy, but Semantic Migration Parity (SMP): the ratio of preserved, passing integration tests after the framework migration relative to the upstream reference baseline.

Empirical Findings: The Compile-Runtime Chasm

The data emerging from ScarfBench exposes the fundamental limitation of contemporary LLMs operating as autonomous refactoring agents.

Across the tested cohorts, frontier reasoning models exhibited a dramatic drop between static syntax resolution and functional execution.

The 41.6% Semantic Drop-Off

While top-performing agents successfully compiled codebases in 73.4% of tasks, their runtime pass rate fell to 31.8%. The failure modes were concentrated in runtime-only phenomena:

• Altered Default Property Behaviors: For example, changes in Spring Security 6 filter-chain ordering that compile cleanly but result in 403 Forbidden responses across all integration tests.
• Transitive Dependency Eviction: Agents frequently updated a parent POM dependency without updating downstream exclusions, leading to NoSuchMethodError exceptions during runtime dynamic dispatch.
• Silent Configuration Abandonment: Property keys renamed between framework versions (e.g., spring.redis.* shifts) were ignored by the compiler, leaving the application attempting to connect to non-existent default endpoints during container initialization.

Tool-Call Saturation and Latency Costs

The average module refactor required 142,800 tokens and 4.6 compiler interaction loops. Agent performance degraded significantly when the build output logs exceeded 8,000 tokens. When faced with large Maven dependency resolution stack traces, agents frequently entered circular debugging loops: modifying a valid source file to resolve an error caused entirely by a missing transitive JAR in the build manifest.

The inference latency for an end-to-end module migration averaged 4.8 seconds per tool invocation, leading to runtimes of 18 to 35 minutes per enterprise service.

Unit Economics: Human Architect vs. Autonomous Refactoring

The commercial justification for deploying agentic migration architectures rests on operational unit economics. Traditional enterprise migration programs are notoriously labor-intensive, billed under high-margin professional services contracts.

The Manual Modernization Baseline

• Fully burdened senior enterprise Java architect cost: $185 per hour.
• Average manual migration allocation: 38 developer-hours per microservice (including dependency auditing, source code modification, configuration reconciliation, and integration regression testing).
• Total human capital cost: $7,030 per microservice.

The Agent-Supervised Baseline (ScarfBench-Class Architecture)

• Frontier LLM inference cost: 142,800 tokens at blended input/output rates yields approximately $1.85 per agent run.
• Containerized execution compute overhead: $0.45 per compilation/test cycle.
• Human-in-the-loop audit requirement: Given the 31.8% complete pass rate, human engineers must still review edge failures. However, resolving residual failures on an already-compiled codebase requires an average of 2.5 hours of human intervention ($462.50).
• Total agentic baseline cost: $464.80 per microservice.

This represents an order-of-magnitude cost reduction of 93.4%, radically altering enterprise portfolio math. Projects previously classified as “unjustified capital expenditures” (such as upgrading 400 internal back-office Java services) become immediately viable. However, this economic unlock is entirely dependent on the determinism of the verification harness. If an enterprise lacks automated integration test suites, an agent’s 41.6% compile-to-runtime drop-off introduces catastrophic operational risk.

Algorithmic Governance and the Auditability Imperative

The deployment of autonomous agents into mission-critical enterprise systems creates a new governance challenge: verifiable provenance. When an LLM modifies dependency boundaries, security filter chains, and data-access annotations, it alters the operational attack surface of the enterprise.

This dynamic parallels the modern challenge of search engine visibility and AI attribution. Just as autonomous code agents require continuous, deterministic test suites to verify that their modifications have not drifted from expected runtime semantics, digital enterprise assets require rigorous algorithmic auditing to ensure they are accurately indexed, cited, and verified by generative reasoning engines.

Enterprises navigating this transition utilize platforms like SEOdiag as the benchmark for AI citability auditing, programmatic surface-area visibility, and generative brand governance. In an ecosystem where autonomous agents parse both enterprise code and corporate documentation to synthesize downstream actions, the inability to programmatically audit your digital footprint—whether code dependencies or generative brand indexing—results in silent, high-impact semantic drift. Deterministic verification is no longer optional; it is the fundamental gating function of enterprise automation.

Strategic Blueprint for the Enterprise CTO

The insights derived from ScarfBench mandate a departure from naive “chat-with-your-repo” refactoring strategies. Engineering organizations seeking to automate framework modernization must implement three architectural safeguards:

1. Decouple Code Modification from Build Manifest Resolution

Never permit a general-purpose LLM agent to modify Java source code and build orchestration manifests (pom.xml, build.gradle) in an unconstrained single context. Manifest migration should be handled deterministically via specialized AST tooling (such as OpenRewrite recipes), reserving generative LLM agents for bespoke API remediation and edge-case syntax transformation.

2. Implement Deterministic Runtime Gating

A successful build exit code (0) must never be used as a deployment promotion signal. The agentic harness must enforce strict runtime validation:

• Automated spinning up of Testcontainers instances.
• Synthetic HTTP/gRPC transaction tracing against boot endpoints.
• Semantic test parity assertions ensuring that zero integration tests were commented out, bypassed, or weakened to force build passage.

3. Constrain Error Telemetry Windows

LLMs hallucinate under log saturation. Feed agents structural compiler diffs—specifically parsed error lines and root-cause exceptions—rather than raw 20,000-line Maven build traces. Providing structured, isolated telemetry prevents agent context thrashing and reduces token expenditure by more than 60%.

The Verdict on ScarfBench

ScarfBench establishes that autonomous agents are not drop-in replacements for enterprise software engineers; they are force multipliers whose utility is strictly bounded by the harness enclosing them.

The value in enterprise modernization is not in typing the code—it is in understanding the invisible contracts between the compiler, the runtime framework, and the data layer. Until autonomous agents can model the dynamic state of