Performance Test Results

The performance tests were conducted for streaming APIs of WSO2 API Manager. Tests were conducted for the three protocols listed below and the subsequent sections present the results and key observations.

  1. Server Sent Event APIs (SSE)
  2. WebSub / WebHook APIs
  3. WebSocket APIs

Deployment used for the test

API-M performance test all-in-one deployment

Name EC2 Instance Type vCPU Mem(GiB)
Apache JMeter Client c5.large 2 2
Apache JMeter Server 01 c5.xlarge 4 4
Apache JMeter Server 02 c5.xlarge 4 4
MySQL db.m5.large 2 8
Netty HTTP Backend c5.xlarge 4 4
WSO2 API Manager c5.large 2 2
  • Token type: JWT
  • The operating system is Ubuntu 18.04
  • MySQL version in RDS instance is 5.7
  • Java version is JDK 11

Performance test scripts

All scripts used to run the performance tests and analyze results are in the following repositories.

Results

Server Sent Events

Test for SSE API was done by using an SSE back end which emitted events in different frequencies. The error rate with concurrent users and the CPU and memory percentage consumed was recorded at that time.

The following graph depicts the Error %, CPU % and Memory % variation in the server during the time in which the request was processed.

Server Sent Events Test Results

Key observations:

  • Five thousand (5000) concurrent connections can be handled with zero errors, with an event rate of 10 Event/sec (Average size of an Event used was 82B)

  • CPU spike is expected during the processing time, however it will settle once the requests are served.

WebSub

Test for WebSub API was done for varying number of subscribers and measuring the associated error, CPU and Memory % consumed.

The following graph depicts the variation of Error, CPU and Memory % with concurrent users.

WebSub Test Results

Key observations:

  • 10,000 subscribers can be handled with zero error rate serving a payload of average size of 8kB.

  • CPU spike is expected during the processing time as higher number of requests are server simultaneously, however this will settle when the events are delivered.

Note

10,000 concurrent users mean a lot, and it is not very common. It is recommended to scale horizontally or vertically to support more concurrent users with acceptable response times. When scaling horizontally, two or more Gateway nodes need to be used with a load balancer. Another load test must be carried out to measure the performance after scaling.

WebSocket

Test for WebSocket API was done using a WebSocket server, which sends back the reversed message received from the client. The ramp up period, error rate, concurrent connections, and CPU and memory usages were recorded during that time.

The following graph depicts the ramp up period with error rate.

websocket_ramp_up_period_vs_error_rate

The following table shows the number of concurrent connections with error rate. A 500 bytes string payload was used as the message. Size of the payload was reduced when non-zero error rates were observed without any failed connections.

Events/second Message Size Attempted Concurrent Connections Failed Connections Error Rate
1 500 bytes 100 0 0%
200 0 0%
300 0 0%
400 0 0%
600 48 0%
800 78 0%
1000 74 0%
5 500 bytes 100 0 0%
200 0 0%
300 0 0%
400 0 0%
600 42 32.90%
800 64 65.25%
1000 62 70.14%
10 500 bytes 100 0 0%
200 0 0%
300 0 46.10%
250 bytes 300 0 14.29%
125 bytes 300 0 12.70%
60 bytes 300 0 29.12%
400 4 62.17%
600 56 85.14%

Key observations:

  • 400 concurrent connections can be made with zero error rate and without any failed connections, at five events per second.
  • 200 concurrent connections can be made with zero error rate and without any failed connections, at ten events per second.

The following graphs depict the variation of CPU and Memory % with concurrent connections.

100_connections

200_connections

300_connections

400_connections

600_connections

800_connections

1000_connections

Top