Part I - Microservices testing alphabet soup
Contract testing vs functional testing vs service testing
The testing world is flush with a dizzying array of test types. The various test types are not all clearly defined, and as a result different people have different outlooks on what the same term encompasses. Today we are going to take a look at three test types associated with microservices APIs, and explain them from our perspective. As stated above, the definitions are not universal – this is just our point of view. However, regardless of the terminology used, there is widespread support for the need of these types of testing.
Contract test is best exemplified by Pact.
From Matt Fellows article: “Contract testing is a methodology for ensuring that two separate systems (such as two microservices) are compatible with one other. It captures the interactions that are exchanged between each service, storing them in a contract, which can then be used to verify that both parties adhere to it.”
From Alex Pruss: “Contract testing tests that any pair of dependent services can properly send and decode messages between each other, but doesn’t test the services’ internal logic.”
Specifically, the consumer of an API makes a request that contains a required set of elements, and in turn the consumer expects certain data elements to be present in the response from the producer. Contract testing validates that the required elements are present in both the request and the response.
Contract testing can be consumer driven or producer driven. Pact supports the consumer-driven model. With Pact, a consumer can manage the contract such that it can:
Validate that expected items are in the response
Inform producers when they stop consuming certain elements in the response
Inform producers when new elements are required
Prevent deployment of incompatible API versions
A major benefit of Pact is that it is integrated with the code, and ensures that there is no divergence between the intended contract and the actual implementation.
The limitations of the Pact approach are:
The contract management is effort intensive since you must create the code to stub every contract.
For every contract, you need create stubs for all possible variations (different status codes, response structures, etc.)
Deploying Pact effectively requires a team buy in. Unless the entire team adopts the process, it does not deliver value.
While the contract specifies the data elements that are expected, it does not include validation of the items. The producer can meet the contract spec but still be functionally incorrect. You still need to write all the functional tests to validate functionality. So why not include the contract validation in the functional tests?
That takes us to functional testing.
(API) Functional testing without service isolation
These tests are run with all the producer services live. Functional testing goes beyond contract testing to validate that not only does the consumer and producer pair satisfy the contracts, but the data returned by the producer meets the consumer’s functional requirements as well. The correctness can be verified with test cases and appropriate assertions.
Producer generates a ‘blueprint’ collection which defines the contract for an API.
The tool generates API documentation from the blueprint collection for the consumers.
Producer creates an example mock for the API for consumers to use during development.
Once the service is available, consumers can create collections of their own based on the blueprint and the API in the dev cluster.
Consumers create assertions to validate the required functionality.
The advantages of this process are:
We think that adoption of this workflow is simpler than adopting Pact. The process is loosely coupled to the development process, and should be less disruptive to roll out incrementally across the organization.
API functionality are validated with live API services.
The contract validation is independent of the code. That means other people like the test engineers can create the tests.
Some of the limitations are:
Contract validation is not explicitly supported as in Pact. However, functional tests and assertions implicitly encompass contract testing since the test will fail unless the contract, as well as the functionality, is satisfied.
API validation is a combination of producer and consumer driven processes. Not a clear single owner.
The tests are not coupled to the actual code, as with Pact. The ‘contract’ and actual API behavior can diverge unless the teams are diligent about keeping the test collections up to date.
The producer services must be live to do the functional testing. Dependency on the producer service availability is built into the process.
Functional testing with service isolation (aka Service testing)
An alternative to the functional testing approach above is to test the consumer service in isolation – i.e., with all producer services mocked. Service testing can validate the contract and functionality just like the functional testing approach above – but with a much more efficient footprint. This approach provides a significant advantage as the number of services increases and service dependency graphs become more dense.
However, this approach requires accurate mocks of the producer services to validate the interaction of the producer and consumer services, and the API functionality. Using this approach requires tools that can support the process of creating mocks for producer services with very little effort.
The advantages of this approach include:
The test setup is lightweight -- it does not require all the producer services to be available. Since the service is tested in isolation using mocks for producer services, the footprint of the test system is very small.
Since the service is tested in isolation, the tests can be run both locally during pre-submit as well as in CI.
Its easy to test compatibility with different versions of the producer services. Testing with a different version of a producer service is as simple as switching to a different set of mocks.
Similarly, it is easy to test compatibility with the consumer services by using the mock contracts used by the consumer as test cases for the producers.
The tests are fast compared to end-to-end tests. In the context of the test pyramid, service tests are just above the unit tests layer, execute fast like unit tests, and keep the test pyramid balanced.
The API validation process is a combination of producer and consumer driven approaches – similar to the functional testing without isolation approach above.
The validation process is similar to the process outlined above for Postman, and involves both the consumer and producer.
The contract implicitly is represented by the result returned by the mock service, and not tightly coupled to the actual code. The mocks can diverge from the code changes unless the teams are diligent about keeping the tests up to date. The divergence can be prevented by enforcing a process using automation of tests that are derived from the consumers (just like Pact).
Scaling the creation and maintenance of accurate mocks does not scale unless the process of mock creation itself can be automated.
Thus, service testing in isolation can deliver all the benefits of functional testing. It can validate the contract and functionality efficiently. In addition, it has the advantage of running the test during pre-submit and identifying problems early in the process. But it creates a new challenge – it requires that there is a scalable way to create functionally accurate mocks for producer services.
In the next part of the article, we will go into how we are addressing these challenges with service testing at Mesh Dynamics.