Now you may be thinking the main focus should be on measuring ESME performance. True. However, the ESME version that we used is based on a older Scala library with a known memory bug. Thus, the test results are not really representative of the current ESME code base.

If you look at other microblogging platforms (including Twitter) – irregardless of whether they are focused on the enterprise or not, you’ll be hard pressed to find any load test results that are published and available to potential users. However, such tests are usually mandatory for IT projects, especially for larger companies with many potential users.  Thus, we would like start publishing our results so that those interested in ESME can make better sizing decisions.

We will soon be starting a new round of load tests based on the Apache code base and a new Scala library, so expect a blog in the future with these results.

For those interested in performing the tests themselves, Daniel has also made the test scripts available

Last night, we performed our first load test of the ESME core. We used the cloud-based installation that is running on the Amazon cloud via Stax.  We used a test suite with nine parallel Java applications threads that made  calls to the REST-API and performed actions such as logging-on and sending a message. This occurred for each of the threads about 250-500 times.  At the test “peak”, we had approx. 1000 concurrent connections.  The source code was based on the old Google Code-based version which includes an older version of the Scala with a known memory bug. This was our first test and we didn’t use the clustering functionality of Stax. We also didn’t run in dedicated server pools available via Stax which would probably improve our performance as well.

Right now, we are analyzing the results and will publish them later.

Why is the load test interesting:

• Anyone in an enterprise who is thinking of going productive with an application (ESME or any other) usually goes to the vendor and asks about sizing. This load test and the ones that follow will help the ESME team make suggestions about what sort of hardware sizing is necessary for installations.  Thus, performing such tests and making the results public gives ESME real credibility in the marketplace.
• Since ESME is focused on integration scenarios where machine-based and human-based messaging comes into question, we have to be able to handle more load.  Note: The fact that we are very aware of scaling issues doesn’t mean that we are suggesting that ESME is some sort Enterprise Service Bus. ESME might not meet the requirements associated with such an architecture.
• For anyone who is thinking about a cloud-based ESME installation, the lessons learned can help them create an optimal Stax-based configuration.
• Publishing the results of the load test is another sign of the “open-source” characteristic of the project.
• Load tests will help us improve the performance of ESME and avoid future problems  (as exemplified by the “failed whales” that were present during Twitter’s rapid expansion).

Next steps:

• Currently, the Stax environment just supports Tomcat. Use of Jetty in load tests might be another option inasmuch as it should improve performance. I’d have to check to see if Stax supports this setup
• Testing the new Apache-based code with the new Scala library
• Try to increase performance by using features available in the Stax environment (such as clustering)
• Expanding the test suite by 1) including a test step to simulate a call of the the Web UI to get the complete picture of ESME performance, 2) creating more complicate networks inasmuch as the first test was just based on one user who currently has no followers and 3)testing the receipt of messages as well.

Thanks to Daniel Koller for performing the test.