Last time, we talked about how to write tests that don’t impede our continuous delivery pipeline. Today, we’ll start the discussion on how to make our pipeline effective for all stakeholders.
Different types of tests have different audiences, and those audiences have different needs. The audience not only defines what tests we should write but in the case of continuous delivery testing, when they are run.
Some audiences need faster feedback than others, so we want to execute tests for those audiences at different stages in our pipeline. We’ll talk about those stages later, but to do so we need to apply another practice first:
Practice #2: Make each test appropriate to its audience
If our tests serve many audiences, it makes them difficult to group. Above, we see developer-produced tests divided into unit, integration, and functional acceptance. We ignore other types of testing, most notably manual testing.
The figure shows us that unit tests should support our developers, while functional acceptance should be meaningful to business stakeholders. Integration tests fall somewhere in the middle, making them particularly interesting. Here are four specific ways you can make your tests target their audience well.
1. Don’t use unit tests to capture requirements
Unit tests serve developers and provide feedback at a level far below any meaningful requirement.
Good unit tests are clear in what they demonstrate, run extremely fast, and are easy to diagnose when they fail. Meeting these requirements means we fake, stub, or mock a great deal in unit tests. Since we control what the fake objects do, the interactions we define might differ from the interactions that occur in production.
This makes unit tests a poor choice for capturing requirements as our unit tests may not capture reality. Let unit tests serve your developers by providing ultra-fast feedback on design. And let other tests worry about functional requirements.
2. Use functional acceptance tests to capture requirements
Functional tests are where we demonstrate our application meets user needs. Thus, they should be written in a way that makes the requirements very clear. When business stakeholders want proof our application does what it’s supposed to, we should be able to show them these tests, and they should be able to understand them.
A Behavior Driven Development style comes in handy here, as its Given, When, Then syntax is more readable to non-technical audiences. Tools like Cucumber offer a way to include business stakeholders into the test writing process. This facilitates communication and trust throughout the whole team.
3. Mock/stub nothing in your functional tests
When using functional acceptance tests to demonstrate requirements, we only want to test against what is actually deployed. Except for third party services, using mocks and stubs produces a test that is part functional acceptance and part unit. In doing so, we have a test that serves neither audience well. It’s too slow in providing feedback for developers, and may provide false assurances on requirements to business stakeholders.
4. Use integration tests for contracts and heartbeats
Integration tests are in the middle ground, and an astute reader will note we’ve already said that’s a dangerous place to be. Why then would we write any integration tests?
We use integration tests to verify how our code interacts with services we don’t control – e.g., a third party service or library. These integration points are typically critical to our application and need a special kind of test. Developers want to know their code interacts properly; business stakeholders want to know our application can interface with a service.
The key here is to write only as many integration tests as absolutely necessary. In particular, we’re not verifying functional requirements here. That’s the role of functional tests. Instead, we follow the advice Neal Ford shared in his workshop of testing for contracts and heartbeats. We answer the question of does the service respond the way we expect.
By making our tests serve their audience, we’ve set the stage for a continuous delivery pipeline that is effective for the whole team. We’ll see how in our final entry in this series.