ESME Blog

In the past, we’ve blogged about our performance tests but in the chaos of our various development cycles we never really institutionalized such tests.

We are now reaching the point as we push towards our first release that we need to assure that performance considerations are met. One move in this direction was our use of Hudson as a base for building ESME on a daily basis. Now, we’ve started examining our performance in detail by planning a series of performance tests to examine ESME in various configurations (cloud-based, on-premise, etc.). The goal is to establish a performance testing component as part of our continuous integration efforts. There is now a Apache wiki page where the various performance test configurations and the performance test results will be listed.

We already started with this increased focus on performance tests with an initial test conducted by Markus Kohler on 2009-11-19. Here is a screenshot from this test.

The results of this test are described in full detail here.

For us, the most important thing is to establish a structure and format for such tests that makes it easy for us and ESME users to be able to compare results in a logical fashion. It is only so that users can decide what sort of infrastructure is necessary for their respective ESME installations.

In a previous blog, I described our use of the Stax Cloud to perform load tests for ESME. Well, Daniel Koller has just finished his analysis of the results of this initial test series and has made a few interesting discoveries. For the ESME team, these initial tests were primarily focused on gaining experience with Stax as an environment to perform such tests (including the use of clusters) as well as creating a test bed for future ESME tests.  The tests used both the REST-API and the Web UI.

Now you may be thinking the main focus should be on measuring ESME performance. True. However, the ESME version that we used is based on a older Scala library with a known memory bug. Thus, the test results are not really representative of the current ESME code base.

If you look at other microblogging platforms (including Twitter) – irregardless of whether they are focused on the enterprise or not, you’ll be hard pressed to find any load test results that are published and available to potential users. However, such tests are usually mandatory for IT projects, especially for larger companies with many potential users.  Thus, we would like start publishing our results so that those interested in ESME can make better sizing decisions.

We will soon be starting a new round of load tests based on the Apache code base and a new Scala library, so expect a blog in the future with these results.

For those interested in performing the tests themselves, Daniel has also made the test scripts available