Let’s easily inject random delay to your Java application

Finding race condition issues in tests can be challenging, as they occur only when an operation is unexpectedly executed between other operations. Such race condition issues could happen in production environment despite passing tests many times.

Consider a scenario where we’re implementing code to transfer money between accounts as follows.

package org.komamitsu.database;

public class Account {
    public int balance;

    public Account(int balance) {
        this.balance = balance;
    }

    public int getBalance() {
        return balance;
    }

    public void setBalance(int balance) {
        this.balance = balance;
    }
}

package org.komamitsu.database;

public class Transfer {
    public void exec(Account from, Account to, int amount) {
        int balanceOfFrom = from.getBalance();
        from.setBalance(balanceOfFrom - amount);

        int balanceOfTo = to.getBalance();
        to.setBalance(balanceOfTo + amount);
    }
}

package org.komamitsu;

import org.komamitsu.database.Account;
import org.komamitsu.database.Transfer;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.*;

public class RaceConditionable {
    void transfer() throws ExecutionException, InterruptedException {
        Account a = new Account(2000);
        Account b = new Account(2000);
        Transfer transfer = new Transfer();
        ExecutorService executorService = Executors.newFixedThreadPool(2);
        List<Future<?>> futures = new ArrayList<>(2);

        // 2 concurrent threads execute the transfer.
        futures.add(executorService.submit(() -> transfer.exec(a, b, 100)));
        futures.add(executorService.submit(() -> transfer.exec(a, b, 100)));
        executorService.shutdown();

        for (Future<?> future : futures) {
            future.get();
        }

        int balanceOfA = a.balance;
        int expectedBalanceOfA = 2000 - 2 * 100;
        if (balanceOfA != expectedBalanceOfA) {
            throw new RuntimeException(
                    String.format("Unexpected balance of `a`. Expected: %d, Actual: %d",
                            expectedBalanceOfA, balanceOfA));
        }

        int balanceOfB = b.balance;
        int expectedBalanceOfB = 2000 + 2 * 100;
        if (balanceOfB != expectedBalanceOfB) {
            throw new RuntimeException(
                    String.format("Unexpected balance of `b`. Expected: %d, Actual: %d",
                            expectedBalanceOfB, balanceOfB));
        }
    }

    public static void main(String[] args) throws ExecutionException, InterruptedException {
        RaceConditionable raceConditionable = new RaceConditionable();
        for (int i = 0; i < 1000; i++) {
            raceConditionable.transfer();
        }
    }
}

In this implementation, this application does the following operations:

1. Instantiates two accounts, account a and b, each with an initial balance of 2000
2. Concurrently transfers 100 from account a to account b with two threads
3. Verifies the balances of both accounts a and b. Account a should have 1800 and account b should have 2200
4. Repeats these steps 1000 times

As you may notice, the implementation of Transfer class has a race condition issue (“lost update” anomaly). However, reproducing this issue in tests can be challenging due to the rapid execution of each line of code under light workload, minimizing the chances of thread interleaving. Even executing the above code multiple times may not reliably reproduce the race condition issue.

To address this, I developed a small Java agent library called Chaos Dukey (https://github.com/komamitsu/chaos-dukey), designed to inject random delays into Java application. It internally uses Byte Buddy. Here’s how you can use it:

1. Download the released JAR file (chaos-dukey-x.x.x-all.jar) from https://github.com/komamitsu/chaos-dukey/releases
2. Pass -javaagent option pointing the downloaded JAR file with some parameters to inject delays into the target classes and methods
3. Execute the Java application with the javaagent option. You’ll observe random delays injected to your Java application.

For example, if you execute the following code (class Foo and Bar are empty) without the Chaos Dukey Java agent library, you’ll observe no delays of course.

    Foo foo = new Foo();
    for (int i = 0; i < 4; i++) {
        long start = System.currentTimeMillis();
        int hashcode = foo.hashCode();
        System.out.printf("Foo: DURATION=%d, HASHCODE=%d\n", (System.currentTimeMillis() - start), hashcode);
    }
    
    Bar bar  = new Bar();
    for (int i = 0; i < 4; i++) {
        long start = System.currentTimeMillis();
        int hashcode = bar.hashCode();
        System.out.printf("Bar: DURATION=%d, HASHCODE=%d\n", (System.currentTimeMillis() - start), hashcode);
    }

Foo: DURATION=0, HASHCODE=1262822392
Foo: DURATION=0, HASHCODE=1262822392
Foo: DURATION=0, HASHCODE=1262822392
Foo: DURATION=0, HASHCODE=1262822392
Bar: DURATION=0, HASHCODE=120694604
Bar: DURATION=0, HASHCODE=120694604
Bar: DURATION=0, HASHCODE=120694604
Bar: DURATION=0, HASHCODE=120694604

However, with passing -javaagent option with proper parameters like “-javaagent:/path/to/chaos-dukey-x.x.x-all.jar=typeNamePattern=^foo\..*\.Bar$,methodNamePattern=hashCode,waitMode=RANDOM,percentage=50,maxDelayMillis=1000”, random delays will be injected to the Java application.

Foo: DURATION=0, HASHCODE=1262822392
Foo: DURATION=0, HASHCODE=1262822392
Foo: DURATION=0, HASHCODE=1262822392
Foo: DURATION=0, HASHCODE=1262822392
Bar: DURATION=148, HASHCODE=489349054
Bar: DURATION=729, HASHCODE=489349054
Bar: DURATION=0, HASHCODE=489349054
Bar: DURATION=689, HASHCODE=489349054

Returning to the first example application, injecting random delays into Transfer.getBalance() and Transfer.setBalance() can help to reproduce the race conditoin issue. For instance, using the following option “-javaagent:/path/to/chaos-dukey-x.x.x-all.jar=typeNamePattern=^org\.komamitsu\.database\.Account$,methodNamePattern=.*Balance$,waitMode=RANDOM,percentage=10,maxDelayMillis=200”, the race condition issue would be easily reproduced.

Exception in thread "main" java.lang.RuntimeException: Unexpected balance of `a`. Expected: 1800, Actual: 1900
 at org.komamitsu.RaceConditionable.transfer(RaceConditionable.java:38)
 at org.komamitsu.RaceConditionable.main(RaceConditionable.java:14)

Although there are some other ways to do similar thing (e.g., write a Byteman script and execute a Java application with Byteman JAR file and the script). But I think chaos-dukey is simpler since it focuses on the target use case. I encourage you to try it out and share your feedback!