Assessment API: Load & Stress Testing
This page describes the plan for the load and stress tests related to the Assessment API, its execution, and results
Motivation
The Asessment API is a component that will be receiving requests from every product that uses the AssessmentModel directly or indirectly (through the Controller). This includes (but is not limited to):
RapidScan
SnowConvert (all "flavors")
SnowSpark
These products are widely used and combined, have many daily executions. That is why our team is very concerned about the performance of this API.
Test Plan
Introduction
We gathered information about the execution of these tools. For instance, we have this numbers:
RapidScan has been executed as much as 23 times on a single day
Snow (all "flavors") has been executed as much as 209 times on a single day (looking at the data from August 2021)
Based on this data, we created the following worst case scenario for a day:
RapidScan will be executed 23 times
SnowConvert (all "flavors") will be executed 209 times
After adding that number, we got 232 executions on the same day. If we duplicate this number (to account for the requests we would have from other products), we get 464 executions a day. If we assume that all these executions take place during an 8 hour span*, we would have 60 (rounding up) executions per hour, which would translate into 15 executions in 15 minutes.
*This assumption is based on the fact that many of the requests we receive seem to fit with the schedule of a person that is working 8 hours a day in a timezone similar to UTC-6. This assumption is also a worst case: if the executions were evenly divided between all 24 hours a day, that would mean it is less likely for the API to receive two simultaneous requests.
Summary of important metrics
15 executions in 15 minutes
The average execution time is 5 minutes
There is only 1 request for execution
At most 25 executions in 15 minutes
Using a Poisson distribution we can tell that, if the average is 30 executions in 15 minutes, there is a 99.6% chance that there won't be more than 25 executions in 15 minutes. Read more about this calculation in the Appendix A.
Purpose of Load Test
To Identify whether the application can handle the peak load
Observe the behavior of the application in terms of response time
To check whether the resources (CPU, Memory and Disk) do not breach the defined performance limit
To identify if there is any bottleneck
Purpose of Stress Test
To Identify whether the application can handle the future load
Note the response time at future load
To check whether the behavior of resources (CPU, Memory and Disk). They should not breach the defined performance limit
To identify if there is any bottleneck
Note down the error percentage
Test Plan
With these numbers, we can extract multiple scenarios for testing. An explanation for each case is included further in this document.
Test ID | Executions | Time (min) between requests** | API Method | Ramp Up |
1 | 15 | 1 | POST | 2s |
2 | 25 | 1 | POST | 2s |
3 | 15 | 1 | POST | 2s |
4 | 25 | 1 | POST | 2s |
5 | 15 | 1 | POST | |
6 | 25 | 1 | POST |
**Time between two requests of the same client
Test Case #1 (Load Testing) - RapidScan
This case was designed thinking in:
The max number of executions: 15
We will test the POST method of the Assessment API
In this case the max number of executions is an average of the peak load base on the data provided by the Application Insights.
Test Case #2 (Stress Testing) - RapidScan
This case was designed thinking in:
The max number of executions: 25
We will test the POST method of the Assessment API
In this case the max number of executions is the max number calculated using a Poisson distribution. By doing this we validate that the API can handle the future load.
Test Case #3(Load Testing) - SnowConvert
Identical to Test Case #1, but with the AssessmentModel of SnowConvert – Oracle execution.
Test Case #4 (Stress Testing) - SnowConvert
Identical to Test Case #2, but with the AssessmentModel of SnowConvert – Oracle execution.
Test Case #5(Load Testing) - SnowCnvert
This case was designed thinking in:
The max number of executions: 15
We will test the POST method of the Assessment API for the GenerateDataset endpoint
Test Case #6 (Stress Testing) - SnowConvert
The max number of executions: 25
We will test the POST method of the Assessment API for the GenerateDataset endpoint
Execution
We executed these tests with Apache JMeter. The file for the test plan is here (you can open it with Apache JMeter. Please do not run it without authorization):
You can enable/disable the different nodes in the left panel to choose which test will you run, and in which environment will you run it. You can also tweak the Thread Group configuration to choose the number of clients (Number of Threads) and number of requests (Loop count). The Sleep Action must be modified to change the time between two requests of the same client.
The results of an execution are shown when clicking on the three bottom nodes. Graph Results show a graph indicating throughput, average response time, standard deviation for response time, and more data (over time). View Results Tree shows every request that was performed during the test. Summary Report shows summary data for the executions (average response time for all requests, max response time for all requests, etc...).
Results
Average, Min, Max and Std Dev for the response time of all requests in each test case (in milliseconds)
Test ID | Average | Min | Max | Std Dev |
1 | 466 | 305 | 663 | 121.95 |
2 | 1283 | 314 | 3045 | 791.19 |
3 | 320 | 99 | 1076 | 243.59 |
4 | 1026 | 71 | 2451 | 663.46 |
Overall performance
Even in the worst scenarios, the average of time for a request to be completed was a fraction of a second.
There were no errors, and all the data was uploaded correctly, and is consistent in the database and the application insights resource.
Behavior of the API during the worst scenarios
In these graphs
The average response time is represented by the blue line
The median response time is represented by the purple line
Conclusions
The average and max response times are good enough for their respective scenarios, considering that:
There will be only 1 request for each execution of the different tools.
All the requests were processed correctly, and the integrity of the data was preserved without any errors.
The performance of the API its good mostly because of the implementation of a queue to process the different jobs.
The resource can handle both the peak and the future load base on the information from the different scenarios.
Appendix A: Calculating the max number of executions for a 15 minutes time span
Calculations
The Cumulative Distribution Function for the Poisson Distribution takes two parameters k and lambda
Cumulative Distribution Function for the Poisson Distribution
And tells us the probability of the following event: "An event will happen at most k times in an interval of time", given that the same event happens lambda times (in average) in the same interval of time.
Using lambda = 30 and k = 45, we got the value 0.996. This indicate that, there is a 99.6% chance that there won't be more than 45 new executions in a 15 minute timespan, given that the average amount of new executions in a 15 minutes time span is 30 (lambda).
Reasoning
The Poisson distribution is usually used to determine the probability of an event happening k times during a time interval. It works best when the events are independent. We also took some considerations (like thinking that maybe there was an 8 hour time span in which the number of requests was much more than during the rest of the day). We considered that the event we wanted to analyze was the first request performed by a session.
Apache JMeter has a number of Timers we can use to simulate the apparition of new events:
Timers proviede by Apache JMeter
However, we did not want to commit to an specific tool, since we are considering automating this tests using Azure DevOps in the future.
Last updated
