Performance Test Results¶
The performance tests were conducted for streaming APIs of WSO2 API Manager. Tests were conducted for the three protocols listed below and the subsequent sections present the results and key observations.
- Server Sent Event APIs (SSE)
- WebSub / WebHook APIs
- WebSocket APIs
Deployment used for the test¶
| Name | EC2 Instance Type | vCPU | Mem(GiB) |
|---|---|---|---|
| Apache JMeter Client | c5.large | 2 | 4 |
| Apache JMeter Server 01 | c5.xlarge | 4 | 4 |
| Apache JMeter Server 02 | c5.xlarge | 4 | 4 |
| MySQL | db.m5.large | 2 | 8 |
| Netty HTTP Backend | c5.xlarge | 4 | 4 |
| WSO2 API Manager | c5.large | 2 | 4 |
- Token type: JWT
- The operating system is Ubuntu 18.04
- MySQL version in RDS instance is 8.0
- Java version is Temurin JDK 11
Performance test scripts¶
All scripts used to run the performance tests and analyze results are in the following repositories.
Results¶
WebSub¶
Test for WebSub API was done for varying number of subscribers and measuring the associated error, CPU and Memory % consumed.
The following graph depicts the variation of Error, CPU and Memory % with concurrent users.
Key observations:
-
10,000 subscribers can be handled with zero error rate serving a payload of average size of 8kB.
-
CPU spike is expected during the processing time as higher number of requests are server simultaneously, however this will settle when the events are delivered.
Note
10,000 concurrent users mean a lot, and it is not very common. It is recommended to scale horizontally or vertically to support more concurrent users with acceptable response times. When scaling horizontally, two or more Gateway nodes need to be used with a load balancer. Another load test must be carried out to measure the performance after scaling.
WebSocket¶
Test for WebSocket API was done using a WebSocket server, which sends back the reversed message received from the client. The CPU and memory usages for concurrent connections were recorded during that time. The tests were run for 30 seconds.
The following table shows the number of concurrent connections with error rate. A 500 bytes string payload was used as the message.
| Events/second | Message Size | Attempted Concurrent Connections | Failed Connections | Error Rate |
|---|---|---|---|---|
| 1 | 500 bytes | 100 | 0 | 0% |
| 200 | 0 | 0% | ||
| 300 | 0 | 0% | ||
| 400 | 0 | 0% | ||
| 600 | 0 | 0% | ||
| 800 | 0 | 0% | ||
| 1000 | 0 | 0% | ||
| 5 | 500 bytes | 100 | 0 | 0% |
| 200 | 0 | 0% | ||
| 300 | 0 | 0% | ||
| 400 | 0 | 0% | ||
| 600 | 0 | 3.24% | ||
| 800 | 0 | 44.54% | ||
| 1000 | 0 | 45.79% | ||
| 10 | 500 bytes | 100 | 0 | 0% |
| 200 | 0 | 0% | ||
| 300 | 0 | 0% | ||
| 400 | 0 | 16.72% | ||
| 600 | 0 | 57.60% |
Note
The backend used here is a Tornado backend. The error rates observed in the above graph are due to limitations in the Tornado backend. API Manager was able to succesfully handle upto 1000 concurrent connections even in the 10 events per second scenario.
Key observations:
- 1000 concurrent connections can be made without any failed connections, at ten events per second.
The following graphs depict the variation of CPU and Memory % with concurrent connections.
