TestEvo-Bench

an executable, live benchmark for test and code co-evolution
mined from real open-source Java projects

About

Tests have to change when the code they check changes. TestEvo-Bench turns real commits from open-source Java (Maven) projects into tasks for coding agents: given a code change, the agent must write or edit the tests that go with it. Every task comes with a buildable repository and is validated by running the developer's own test, and every task records its commit date, so results can be restricted to commits made after a model's training cutoff.

The current snapshot has 3,020 tasks from 196 repositories with commits from January 2019 to June 2026, and a 584-task Lite subset for cheaper evaluation. Beyond pass/fail, each test is scored by the coverage it reaches and the share of mutants it kills. On Lite, the six agent configurations we evaluated (mini-SWE-agent and Claude Code with four models) succeed on 98.3–99.1% of tasks and reach an overall score of 79.8–84.4%, on par with the developers' own tests (81.8%). The leaderboard therefore ranks them by this overall score, which also counts coverage and mutation.

Three tracks — pick one to see a real task from the benchmark

A code change adds or changes behaviour. Write a new test that passes on the new code and fails on the old code.

Example UrlTemplate.put(Map) called putAll on its own argument, so the parameters were silently dropped. The fix passes each entry to put(String, String), and the developer adds a test that checks the rendered URL contains them. Task from FluentLenium/FluentLenium, January 2019 (diff).

production UrlTemplate.java

public UrlTemplate put(

Map<String, String> values) {

values.putAll(values);

values.forEach(this::put);

return this;

}

test generation UrlTemplateTest.java

public class UrlTemplateTest {

// ... existing tests

@Test

public void testSetParametersFromMap() {

UrlTemplate template = new UrlTemplate(

"/abc/{param1}/def{?/param2}/ghi{?/param3}");

template.put(Map.of(

"param1", "test1", "param3", "test3"));

assertThat(template.render())

.isEqualTo("/abc/test1/def/ghi/test3");

}

A code change breaks an existing test. Update it so it passes on the new code and fails on the old code.

Example ExclusionChecker.check now sorts the invalid exclusions it reports, so the existing test fails on the new code, and the developer reorders its expected list. Task from apache/maven-dependency-plugin, April 2024 (diff).

production ExclusionChecker.java

void check(Coordinates artifact,

Set<Coordinates> excludes,

Set<Coordinates> actualDependencies) {

List<Coordinates> invalidExclusions =

excludes.stream()

.filter(exclude -> actualDependencies

.stream().noneMatch(

exclude.getExclusionPattern()))

.sorted()

.collect(toList());

if (!invalidExclusions.isEmpty()) {

violations.put(artifact, invalidExclusions);

}

test update ExclusionCheckerTest.java

@Test

public void shallReportInvalidExclusions() {

// excludes: one, two, three, four

// actual dependencies: one, four

...

checker.check(artifact, excludes,

actualDependencies);

assertThat(checker.getViolations())

.containsEntry(artifact, Arrays.asList(

coordinates("com.example", "two"),

coordinates("com.example", "three")

coordinates("com.example", "three"),

coordinates("com.example", "two")

));

}

A test already passes. Improve it so it covers more of the changed code.

Example AccountBuilder.addEmail now accepts addresses that already carry the mailto: prefix. The old test still passes on the new code but never reaches the new branch; the developer adds a prefixed address, so the test covers all 4 lines of the changed method instead of 3. Task from shred/acme4j, April 2023 (diff).

production AccountBuilder.java

public AccountBuilder addEmail(

String email) {

addContact("mailto:" + email);

if (email.startsWith("mailto:")) {

addContact(email);

} else {

addContact("mailto:" + email);

}

return this;

}

test refinement AccountBuilderTest.java

@Test

public void testEmailAddresses() {

var builder =

Mockito.spy(AccountBuilder.class);

builder.addEmail("foo@example.com");

Mockito.verify(builder).addContact(

Mockito.eq("mailto:foo@example.com"));

builder.addEmail("mailto:bar@example.com");

Mockito.verify(builder).addContact(

Mockito.eq("mailto:bar@example.com"));

}

covered lines of addEmail 3 / 44 / 4

Dataset overview

Loading dataset statistics…

TestEvo-Bench

About

Three tracks — pick one to see a real task from the benchmark

Dataset overview

Leaderboard

Data Explorer