Mocking AWS SDK APIs using Mesh Dynamics
Many cloud-native applications build on AWS services using AWS SDK. For example, applications storing and retrieving files from AWS S3 use the AWS S3 SDK. The developers of these applications need to protect the functionality from regression by creating automated tests that run for every PR, or at least periodically.
In these tests, you often do not want to make real API calls to S3. Solutions like the following enable developers to write such tests using emulators/mocks:
Most of these options follow a similar approach where there is an emulated S3 server running, and it is backed by either the local file system or an in-memory backend.
The tools mentioned above are great if you want the initial S3 state to be simple like below:
Either no buckets initially, or
S3 configured with a few buckets and a few files. In this case, you need to prepare this manually in your local filesystem backed by the emulated S3.
The above approach becomes cumbersome if you want to mimic a real S3 account where you might have many buckets and many files under different buckets. You would either need to prepare the desired setup manually, or you would need to sync and download the entire content locally.
Mesh Dynamics approach
Mesh Dynamics takes a different approach to this problem. The main intent of the test is to validate the application that uses the AWS SDK. If the mocked S3 API call returns responses like the real S3, it is good enough to test the application. It is not necessary to create all the buckets and the real files in the S3 emulator.
For example, we need to test an application that makes the following calls to an S3 account that has 100 buckets, with 100 files per bucket.
List files in a bucket
Get a file from a bucket
With existing solutions, if you want to test in an environment that resembles the actual operating conditions, you would need to create all those buckets locally (in-memory or in local file system), then create 100 files under the specific bucket, and place a real file in that bucket to be fetched. This involves a lot of manual effort.
With Mesh Dynamics, these manual preparatory steps are not needed. We can record the reality that we want to simulate by letting all the calls to the real S3 go through Mesh Dynamics proxy. We can record the requests and responses once, and auto-create exact mocks from the recordings.
Steps to mock S3 using Mesh Dynamics
Let us see how we can do the above using Mesh Dynamics.
Overview of the steps
Build S3 client to send outgoing traffic through the Mesh Dynamics proxy.
Configure Mesh Dynamics to capture the requests to S3, and the responses from S3.
Configure the collection where to save the requests and responses.
Configure Mesh Dynamics to unencrypt https traffic through the proxy.
When Mesh Dynamics gets the data from the configuration above, it will auto-create mocks for S3.
1. Build S3 client to send outgoing traffic through Mesh Dynamics proxy
Let us take the following example where we assume that there is an existing bucket in S3 called “bucketmdtest”, and we are putting an object in that bucket. Here is the GitHub link to the full code https://github.com/cube-io-corp/public/tree/main/awss3mock.
The code below does the following actions:
Setup an HTTP client factory with the proxy to send all calls through the proxy.
Build S3 client object using the HTTP client factory.
Put an object in a predefined S3 bucket.
Close S3 Client.
The only change here from the default client configuration is that we are building an S3 client with the proxy pointing to the Mesh Dynamics proxy running on port 9002 on the local machine.
//Step 1: Setup an HTTP client factory with the proxy to send all calls through the proxy. SdkHttpClient apacheClientFactory = ApacheHttpClient.builder() .proxyConfiguration(ProxyConfiguration.builder() .endpoint(URI.create("http://127.0.0.1:9002")).build()).build(); Region region = Region.US_WEST_2; //Step 2: Build S3 client object using the HTTP client factory. S3Client s3 = S3Client.builder() .httpClient(apacheClientFactory).region(region).build(); //The following commented line is the one used without any proxy //S3Client s3 = S3Client.builder().region(region).build(); String bucket = "bucketmdtest"; String key = "key"; System.out.println("Uploading object..."); // Step 3: Put an object in a predefined S3 bucket. s3.putObject(PutObjectRequest.builder().bucket(bucket).key(key) .build(), RequestBody.fromString("Testing with the AWS SDK for Java")); System.out.println("Upload complete"); System.out.printf("%n"); //Step 4: Close S3 Client s3.close();
2. Capture traffic to and from S3
We need to configure Mesh Dynamics proxy in live mode to allow capturing the traffic to S3. The proxy forwards the requests to the S3 service, and captures the requests and responses. The captured information is used for mocking S3 in subsequent steps.
Note: The target URL in the above configuration dialog is not used by this use case. However, it must be filled in when configuring the setup.
3. Save S3 requests and responses in a collection
Before starting the capture, we also need to configure the collection in which the captured S3 API requests would be saved.
4. Configure Mesh Dynamics to unencrypt https traffic through the proxy
As the calls from AWS S3 SDK to the AWS cloud are https calls, we need to decrypt the https traffic to store the data in a readable format. To achieve this, Mesh Dynamics proxy can generate a self-signed certificate. To generate the certificate, first open the proxy configurations panel by clicking on App:
Next, select Settings:
Next, select the Generate Root Certificate checkbox at the bottom of the Settings page.
When we set the above configuration and restart Mesh Dynamics, it creates a root certificate. We need to add them to java trust store or alternatively, we can also configure the java client to trust all certificates like below (of course this is not a secured option).
SdkHttpClient apacheClientFactory = ApacheHttpClient.builder() .proxyConfiguration(ProxyConfiguration.builder() .endpoint(URI.create("http://127.0.0.1:9002")).build()) .buildWithDefaults(AttributeMap.builder().put(TRUST_ALL_CERTIFICATES, Boolean.TRUE).build()); Region region = Region.US_WEST_2; S3Client s3 = S3Client.builder() .httpClient(apacheClientFactory).region(region).build();
The following is an example of how to add the generated certificate to the java trust store using the key tool command in Mac:
/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/keytool -importcert -alias anyproxy_cert --file ~/.anyproxy/certificates/rootCA.crt --keystore /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/lib/security/cacerts
5. Use Mesh Dynamics to auto-create mocks for S3
Once all the above configuration settings are done, the Mesh Dynamics API Studio is ready to capture S3 calls made by the client program using the AWS SDK. All captured requests and responses are automatically converted to mocks.
Before running the same client program next time, we can simply configure the AWS service to be mocked by selecting the Mock check box as shown below:
This time, the client program will not make a call to the real AWS s3 service -- rather it is served by mock.