• Venky Ganti

API Tracing Accelerates Microservices Debugging and Testing

Testing and debugging distributed and microservices applications has become effort-intensive as modern applications rely on complex interactions among services and APIs. Developers must therefore cope with the additional effort necessary in this context and create different types of tests to validate both functionality and the inter-dependence between services.

To detect bugs early, developers must now create the additional types of API tests in the context of microservices & distributed services architectures besides E2E API tests that they are traditionally used to creating.

  • Service testing: Testing each service with its producers and consumers early helps detect functional (including integration) issues early. Service testing is usually done by isolating the service under test, i.e., all the producer services are mocked during the test. Service tests can therefore be run by developers pre-submit or run in the CI and thus enable shift-left.

  • Component testing: Testing the functionality and integration of a group of closely related services which constitute a component of the application functionality. This is a natural extension to service testing. However, all services in the component must be ready for debugging and testing before. Hence, these ad hoc and automated component tests tend to be run a later stage than service tests in the development cycle.

Both these types of API tests require developers to create and maintain extensive API mocks. The following figure illustrates a service test: test requests are driven against the service under test while all the producers the service requires are mocked.

Developers, therefore, need to:

  • observe the relationship between ingress and all the egress requests it generates to producers. This visibility is critical for debugging.

  • create and maintain extensive API mocks.

Most of the API design tools like Insomnia, Postman, BloomRPC focus only on the endpoint and do not offer any visibility into the request execution – i.e., the connection between egress requests generated by the service as a result of the ingress request.

  • Even with these tools, it is tedious for developers to debug the interactions between their service and the producer services they depend on.

  • Developers must put in a bunch of effort to prepare egress API mocks.

Mesh Dynamics: Built-in API Tracing

Mesh Dynamics is the only API testing tool which builds upon API tracing to enable developers:

  • easily debug functionality and interactions between services

  • create and maintain service tests as easily as unit tests with auto-created API mocks

Request tracing helps developer debugging

While designing and developing APIs, Mesh Dynamics enables developers to visualize the dependence between an API request and response of the developer’s service and the egress/outgoing API requests & responses her service makes to its producers. An “API request trace” in Mesh Dynamics consists of the API request & response against the endpoint and all egress requests & responses to producer services.

The above screenshot of the Mesh Dynamics API editor illustrates the API request trace as well as the comparison with an expected baseline trace.

  • The left-hand pane is the baseline API request trace along with all the request/response details. This baseline is likely the expected behavior by the service version in production. If no such baseline is available and the API request is being created for the first time, the left-hand side would be empty.

  • The right-hand pane is the actual API request trace of the API request against the service under development and the producer services it uses. In this example, the API request trace made two additional outgoing calls than what used to happen in the baseline. A developer is thus easily able to dig into each of those outgoing requests/responses and figure out what change may have caused this difference. Without MeshD, the developer must track the code execution and review all the egress API calls being made by the service being modified.

Create service tests as easily as unit tests with auto-created API mocks

After the change is validated with ad hoc testing, the developer must create or update a service test. As discussed above, a service test enables the service to be tested in isolation without provisioning producer/consumer services. The typical approach, even with all other API development tools, requests the developer to invest a significant amount of time to create API mocks manually.

API tracing of each request execution enables MeshD to create API mocks for all egress requests with no additional effort from the developers. MeshD saves the entire API request trace as one unit. The following figure illustrates the trace for the API request “/minfo/listmovies” endpoint of the Movieinfo service. The trace consists of the request and response details of the /minfo/listmovies API request as well all the outgoing requests made by the Movieinfo service.

When the developer needs mocks for this instance of the /minfo/listmovies API request, she has them readily available without any effort. She just has to save the API trace, and the egress requests & responses will become the relevant API mocks. Further, she can even configure assertions through the UI to compare the expected and actual responses whenever the service test is run. (More information on the configuration of assertions is available online.)

Auto-created API Mocks are Customizable

In the following figure, we illustrate editing the response body (the “ratings_source” field) of the outgoing request to the Reviews service. Once saved, the mock response would be updated for this particular API request hitting the “movieinfo/listmovies” endpoint. We could equally easily update the request to any of the outgoing services including params and headers.


Modern API-driven applications add a costly layer of effort-intensive API testing tasks, especially service testing and component testing. These add an additional burden of creating an extensive set of API mocks. We illustrated how MeshD leverages API tracing to alleviate this burden by making it easy to create and maintain service tests with auto-created API mocks.

Component testing is a natural extension of service testing. However, all services comprising a component may be running in a remote dev/staging cluster. So, we need a different mechanism to listen to the API traffic in such clusters. We will discuss this capability of Mesh Dynamics in the second part of this blog post.