ESME Blog

In the past, we’ve blogged about our performance tests but in the chaos of our various development cycles we never really institutionalized such tests.

We are now reaching the point as we push towards our first release that we need to assure that performance considerations are met. One move in this direction was our use of Hudson as a base for building ESME on a daily basis. Now, we’ve started examining our performance in detail by planning a series of performance tests to examine ESME in various configurations (cloud-based, on-premise, etc.). The goal is to establish a performance testing component as part of our continuous integration efforts. There is now a Apache wiki page where the various performance test configurations and the performance test results will be listed.

We already started with this increased focus on performance tests with an initial test conducted by Markus Kohler on 2009-11-19. Here is a screenshot from this test.

The results of this test are described in full detail here.

For us, the most important thing is to establish a structure and format for such tests that makes it easy for us and ESME users to be able to compare results in a logical fashion. It is only so that users can decide what sort of infrastructure is necessary for their respective ESME installations.

We’ve waited far too long.

After our move to Apache, we were always saying that our Web UI was too primitive and a new one was top priority. Well, the design for the new web UI is finished. The web designer Joy Reyes has been working on the new UI design.

The next step is to take this design and bring it to life with our existing code base.

It is now possible to add formatting to your ESME messages. Through the use of the Textile formatting language, you can create messages with bold, italic, etc.

For example, the messages in the top message were created with the following lines:

_emphasis_
*strong*
__italic__ **bold** ??citation??
-deleted text-
+inserted text+
^superscript^
~subscript~
%span%
@code@

We haven’t gotten all the Textile possibilitites  to work but we are trying to get as many as possible to be available for users.

Have fun.

We’ve been cleaning up our code base to enable more international versions of the application. We also have some new contributors who are helping with the translation work.  Xuefeng Wu is a new contributor from China who is helping us with various aspects of the project. With his help, we are working on bringing out a Chinese language version of ESME.

Don’t be shocked by the UI. This picture was taken with the old UI (Our new UI is currently being designed! More soon).

One of the advantages  of having an international team is being able to crowd-source the translation work. Expect ESME in different languages once we get our new UI up and running.

We are now using Apache Hudson for daily builds.   Every day, our code base is extracted from the subversion-based code repository, built and tests performed. This allows us to make sure that we have code base that is viable and assists in our quality control measures. JIRA items that were built also receive comments describing the successful build.

If you want to keep track of our builds, here is the RSS feed.

ESME has a REST API that is necessary for clients to access data / functionality. There was some degree  of debate / criticism – some of it rather critical  – of this initial interface. Ethan Jewett has blogged about a suggestion for a new REST API for ESME which not only provides an examination of main points of REST-related debate but also the foundation for a new REST API.  Ethan is planning on moving the discussion over to the Apache wiki for further discussion.

Please join us in creating the best possible REST API for ESME.

Note: I don’t currently have the link for the page in Apache wiki with the discussion but I will update this blog when I have the link.

One of the main tasks for the first sprints since we joined Apache has been support of the Twitter API .  The main reason for this functionality is the ability to use existing twitter clients to access ESME. Well, Vassil Dichev  has been working hard to this important task up and running. Using Twitter clients, it is now possible to send messages, receive the public and private timelines as well as view friends/followers.

The use of other Twitter clients to access ESME isn’t always possible, because the client’s configuration must be changed so that the application accesses the Esme server rather than the main Twitter server. Unfortunately, this switch isn’t always possible, because the server which is the basis for messages is usually hardwired.  Those applications that can also access laconi.ca  servers can also be made to access ESME servers

I’ve tried it using Twhirl against the Stax deplyoment with the latest Apache code drop and was pleased to see that it works.

If you want to try it yourself, then you to do the following steps.
1. Create a user on the ESME server at Stax
2. Click on the manage token link and create a token. This is is your password for using the Twitter client
3. Create a new account with twhirl using the laconi.ca type not the identica.ca type.

Use this synatx “apachedick@esmecloudserverapache.dickhirsch.staxapps.net”. Instead of “apachedick” use your own user_id.
4. Click on ”Connect” and set password using your token from step 2.

5. You should now see the public timeline.

I’ve only tried accessing ESME via twhirl. If anyone has success using other clients, then please add a comment.

Please note that we have not yet done everything in the Twitter API, so some things still aren’t working (for example following users). Other functional characteristics of Twitter like DMs aren’t yet part of the ESME architecture.

Please also note that the Web UI at Stax doesn’t work properly, so bear with us. The Stax deployment is the latest code drop from Apache which means it might include bugs.

Daniel Koller has released a few preliminary details on his latest load test based on the latest Apache-based source.  This time he focused on the REST API and compared a one-server configuration to a configuration with three server.

The first picture shows the environment with one server:

The second picture shows three servers:

What is interesting is the comparison of these results with the initial load tests that were based on an older version of Scala with a notorious memory bug. In the first tests, the performance curve was very chaotic. The new version of ESME is much more stable and shows more expected performance behavior.

We’ll publish more details on these latest performance test soon.

Darren Hague has moved the ESME Client based on the Java Api from the Google Code repository to our Apache repository. He also added additional methods including the critical ”getMessages” method which was previously missing. It is now possible to easily create bots that integrate with existing environments.  Darren has also provided a test application (with code) that shows how to use the API.

The JAVA API is still a work in progress but the most important parts are now present.

One of the most powerful features of ESME are the so-called actions which allow users to individually respond to certain event types (previously primarily associated with message contents) . For example, you could send yourself an email when messages with a certain tag were created. Well, Vassil Dichev has finally merged his new actions into the main development trunk and there are some cool new features. What Vassil has done is to look at other types of microblogging events (those that aren’t necessarily associated with messages) and has created “hooks” so that users can respond to them.

Here are a few examples:

Vassil also added other functionality to the “http” action so that it is now possible to create twitter and identi.ca messages as well via actions. I’ve tried it and it is pretty awesome. Here is how it is done:

http://[User]:[password]@identi.ca/api/statuses/update.xml
status=%s

(Make sure that "status=%" is on a new line)

The idea was to look at microblogging tools that supply functionality that is missing in Twitter and embed this provided functionality in ESME in the form of actions.  Therefore, it wouldn’t be necessary to have additional tools but users could have everything in one place.

The full description of the new functionality is present in the Apache wiki .

We are currently playing with ideas regarding a plug-in architecture so that others could also develop actions.

Every week, we deployed the latest version of ESME on our server in the Stax cloud. If you want to play with the actions yourself, you can login to that server and test the actions out.

Note: Please be aware that this server has a version of ESME that reflects the latest code base. Since some things are still being developed, certain functionality will not work. For example, the current release after our code restructure still can’t display actual messages and the UI is horrible. This should all change in the next few weeks.

If you have ideas for new actions, then feel free to add a feature request at our Apache Jira site.