Mocking External Services

Should your tests mock outside services or not? I keep seeing discussions on this topic. On the one hand, it lets your tests be in control of the testing context. Otherwise it may be very difficult to create a reliable automated test.

  • The external service might return different results at different times.
  • The external service might be slow to respond.
  • Using the external service might require running the test in a particular environment.
  • It may be impossible to generate certain error conditions with the real service.
  • There may be side-effects of using the real service.

On the other hand, our mock of the external service might differ from it in important ways.

  • Our mock service might have a slightly different interface than the real one.
  • Our mock service might accept slightly different parameters than the real one.
  • Our mock service might return slightly different results than the real one.
  • The real service might change behavior, and we won’t notice until we deploy to production.

This leaves us in a quandary. What to do?

Here is a pattern that I’ve found helpful:

1. Isolate my code from the external service using the Adapter Pattern. Less often, I might use the Mediator Pattern. I find this isolation a good idea whether or not I need it for testing. It allows me to wrap the external service with an API that’s custom made for my application, offering just the right affordances in terms that make sense in my context. All of the translation between the terminology of my application and the terminology of the external service are handled within the adapter. Changes in the external service, or changing to another service altogether, should be limited to the confines of the adapter.

2. Test my application using a Mock Adapter. Strictly speaking, it’s not a mock, which implies that it self-validates the usage. Generally it’s a fake or a stub (depending on whose terminology) that provides data to my system, or a spy that captures output data from my system for examination by the test. Testing with the mock adapter demonstrates how my application behaves with the API that is specific to the needs of my application.

3. Test my Adapter with the external system. As I’m building my application, I discover needs that it has of the adapter. For each of those needs, I have a test (or multiple tests) of the application using the mock adapter. When generating those tests, I also write tests that demonstrate how I think the real service, as seen through the real adapter, will behave.

These tests go in a different suite and get run less frequently. They get run often as I’m growing the adapter. They also get run periodically to make sure things still work. And they get run whenever receiving a new version of the external service. This has saved a lot of work when the new version was delivered with regressions in the existing functionality.

Sometimes testing the adapter and service is hard to do, as the external system may have difficult environmental constraints. I’ve had cases where the adapter had to be running in the same application server as the external service in order for things to work. This makes it more difficult to run the tests. The tests either need to be inside the container, too, or they need a proxy inside the container to talk to the adapter. Either is a PITA, but worth the effort if you can’t test the combination of adapter and external service any other way.

What does this look like? Declan Whelan asked this question, and prompted this article.

I’ve got an application that delivers race horse horoscopes to help betters make logical decisions on race day. The top API of the domain is the CrystalBall class:

public class CrystalBall {

    private HoroscopeProvider horoscopeProvider;

    public CrystalBall(HoroscopeProvider horoscopeProvider) {
        this.horoscopeProvider = horoscopeProvider;
    }

    public static CrystalBall instance() {
        return new CrystalBall(new CachingHoroscopeProvider(
                MumblerHoroscopeProvider.instance(),
                DerbyHoroscopeCache.instance()));
    }

    public String requestHoroscope(String horsename, String effectiveDate) {
        return horoscopeProvider.horoscopeFor(horsename, effectiveDate);
    }
}

Note that the factory method, instance(), provides a default HoroscopeProvider. This one happens to cache the results in a Derby database to avoid unnecessary calls to the actual MumblerHoroscopeProvider which is the adapter to the service, Mumbler, which is implemented in a third party library. The adapter looks like this:

public class MumblerHoroscopeProvider implements HoroscopeProvider {

    private static final String DEFAULT_RULES = "{Outlook cloudy, try again later.}";
    private Mumbler mumbler;

    public MumblerHoroscopeProvider(String rules) {
        this();
        mumbler.loadRules(rules);
    }

    private MumblerHoroscopeProvider() {
        mumbler = new Mumbler();
    }

    public MumblerHoroscopeProvider(File file) {
        this();
        try {
            mumbler.loadFile(file.getCanonicalPath());
        } catch (IOException e) {
            mumbler.loadRules(DEFAULT_RULES);
        }
    }

    public MumblerHoroscopeProvider(InputStream stream) {
        this();
        try {
            mumbler.loadStream(stream);
        } catch (IOException e) {
            mumbler.loadRules(DEFAULT_RULES);
        }
    }

    @Override
    public String horoscopeFor(String horsename, String effectiveDate) {
        return mumbler.generate();
    }

    public static HoroscopeProvider instance() {
        String resourceName = "/com/gdinwiddie/equinehoroscope/resources/MumblerHoroscopeRules.g";
        ResourceLoader loader = new ResourceLoader();
        InputStream stream = loader.loadResourceStream(resourceName);
        return new MumblerHoroscopeProvider(stream);
    }

    static class ResourceLoader {
        InputStream loadResourceStream(String name) {
            return getClass().getResourceAsStream(name);
        }
    }
}

This adapter is a little complicated because it has the flexibility of providing the necessary grammar file to Mumbler via a String, a File, or an InputStream. In all cases it meets the needs of the service to have a grammar that describes the horoscopes to be generated. It also provides the method horoscopeFor() that translates to the service API call generate().

The unit tests use a mock adapter, MockHoroscopeProvider, that allows the test to add horoscopes and expect them to be returned in that order.

public class MockHoroscopeProvider implements HoroscopeProvider {
    List horoscopes = new ArrayList();

    public void addHoroscope(String horoscope) {
        horoscopes.add(horoscope);
    }

    public String horoscopeFor(String horsename, String effectiveDate) {
        return horoscopes.remove(0);
    }

}

You can see this being used in the test, ensureWeGetHoroscopeFromProvider():

public class EquineHoroscopeTest {
    private static final String CANNED_HOROSCOPE = "The rain in Spain falls mainly on the plain.";
    private MockHoroscopeProvider mockHoroscopeProvider;

    @Before
    public void setUp() {
        mockHoroscopeProvider = new MockHoroscopeProvider();
    }
    @Test
    public void ensureWeGetHoroscopeFromProvider() {
        mockHoroscopeProvider.addHoroscope(CANNED_HOROSCOPE);
        CrystalBall forecaster = new CrystalBall(mockHoroscopeProvider);
        assertEquals(CANNED_HOROSCOPE,
                forecaster.requestHoroscope("Doesn't Matter", "today"));
    }

}

The test queues up a horoscope and then verifies that it is returned by the CrystalBall. In this manner, we check that our system works properly as long as the HoroscopeProvider provides a horoscope when expected. There are other tests for other classes verifying the the horoscope is properly cached when the CachingHoroscopeProvider is used.

What about our real horoscope service? We have tests for our real adapter calling the real service:

public class MumblerHoroscopeProviderTest {
    @Test
    public void canGenerateHoroscopeFromString() {
        HoroscopeProvider provider = new MumblerHoroscopeProvider(
                "{Better go back to bed.}");
        assertEquals("Better go back to bed.",
                provider.horoscopeFor("any horse", "any date"));
    }

    @Test
    @Ignore
    public void canGenerateHoroscopeFromFile() {
        // TODO need to handle when run from common build
        String fileName = "test/com/gdinwiddie/equinehoroscope/resources/dummyHoroscope.g";
        HoroscopeProvider provider = new MumblerHoroscopeProvider(new File(
                fileName));
        assertEquals("Simple sentence.",
                provider.horoscopeFor("doesn't matter", "yesterday"));
    }

    @Test
    public void handleMissingRulesFile() {
        HoroscopeProvider provider = new MumblerHoroscopeProvider(new File(
                "invalid/path/to/rules.g"));
        assertEquals("Outlook cloudy, try again later.",
                provider.horoscopeFor("any horse", "any date"));
    }

    @Test
    public void canGenerateHoroscopeFromResourceFile() {
        String resourceName = "/com/gdinwiddie/equinehoroscope/resources/dummyHoroscope.g";
        InputStream stream = getClass().getResourceAsStream(resourceName);
        HoroscopeProvider provider = new MumblerHoroscopeProvider(stream);
        assertEquals("Simple sentence.",
                provider.horoscopeFor("doesn't matter", "yesterday"));
    }

    @Test
    public void defaultRulesFile() {
        HoroscopeProvider provider = MumblerHoroscopeProvider.instance();
        assertNotSame("Outlook cloudy, try again later.",
                provider.horoscopeFor("any horse", "any date"));
    }

    @Test
    public void printSomeHoroscopes() {
        HoroscopeProvider provider = MumblerHoroscopeProvider.instance();
        for (int i = 0; i < 10; i++) {
            System.out.println(provider.horoscopeFor("", ""));
        }
    }
}

As I look at these tests, I see that I left them in less-than-perfect condition. The test to generate a horoscope from a rules file is ignored because the build file for a full system that includes this component runs from a different directory, and cannot find the file. This test was never fixed because that system depends on a resource rather than a file. I note now that I’m missing a test for the behavior when the resource is specified but missing. The last test is unusual in that it has no assertion. It merely prints a few horoscopes. This allows visual inspection of the output, which has significant randomness. This is a quick-and-dirty method of testing the “interestingness” of the generated output given the current state of the rules file. I don’t yet know a way to assert “interestingness.”

TL;DR

Isolate your system from an external service using an adapter.

System isolated from External Service using Adapter

Use a mock adapter to test your system. Also test the real adapter and external service to verify your assumptions.

Test in two parts.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.