ESME Blog

In a previous blog, I described our use of the Stax Cloud to perform load tests for ESME. Well, Daniel Koller has just finished his analysis of the results of this initial test series and has made a few interesting discoveries. For the ESME team, these initial tests were primarily focused on gaining experience with Stax as an environment to perform such tests (including the use of clusters) as well as creating a test bed for future ESME tests.  The tests used both the REST-API and the Web UI.

Now you may be thinking the main focus should be on measuring ESME performance. True. However, the ESME version that we used is based on a older Scala library with a known memory bug. Thus, the test results are not really representative of the current ESME code base.

If you look at other microblogging platforms (including Twitter) – irregardless of whether they are focused on the enterprise or not, you’ll be hard pressed to find any load test results that are published and available to potential users. However, such tests are usually mandatory for IT projects, especially for larger companies with many potential users.  Thus, we would like start publishing our results so that those interested in ESME can make better sizing decisions.

We will soon be starting a new round of load tests based on the Apache code base and a new Scala library, so expect a blog in the future with these results.

For those interested in performing the tests themselves, Daniel has also made the test scripts available

One of the most powerful features of ESME are the so-called actions which allow users to individually respond to certain event types (previously primarily associated with message contents) . For example, you could send yourself an email when messages with a certain tag were created. Well, Vassil Dichev has finally merged his new actions into the main development trunk and there are some cool new features. What Vassil has done is to look at other types of microblogging events (those that aren’t necessarily associated with messages) and has created “hooks” so that users can respond to them.

Here are a few examples:

Vassil also added other functionality to the “http” action so that it is now possible to create twitter and identi.ca messages as well via actions. I’ve tried it and it is pretty awesome. Here is how it is done:

http://[User]:[password]@identi.ca/api/statuses/update.xml
status=%s

(Make sure that "status=%" is on a new line)

The idea was to look at microblogging tools that supply functionality that is missing in Twitter and embed this provided functionality in ESME in the form of actions.  Therefore, it wouldn’t be necessary to have additional tools but users could have everything in one place.

The full description of the new functionality is present in the Apache wiki .

We are currently playing with ideas regarding a plug-in architecture so that others could also develop actions.

Every week, we deployed the latest version of ESME on our server in the Stax cloud. If you want to play with the actions yourself, you can login to that server and test the actions out.

Note: Please be aware that this server has a version of ESME that reflects the latest code base. Since some things are still being developed, certain functionality will not work. For example, the current release after our code restructure still can’t display actual messages and the UI is horrible. This should all change in the next few weeks.

If you have ideas for new actions, then feel free to add a feature request at our Apache Jira site.

We’ve always used the Scrum methodology as the basis for our development work. Last week, we started our first iteration / sprint since we joined the Apache Incubator and I wanted to create a quick blog describing what we are planning.

Note: All this material is also present on the esme-dev mailing list at Apache as well as on the Apache Jira site for ESME but I wanted to save people the effort of foresting through all the posts to find the details.  The post of this summary is also another indication of the openness of the project. If you want details for each task, then go to Jira and look at the individual tasks

Scrum completion date: Feb 6, 2009.

Teams:

• Idea is that there will be two teams that focus on particular tasks. Each team will have between 5-10 tasks for each sprint / iteration.
• Server team: David, Daniel Koller, Darren, Vassil
• UI team: Bill, Mrinal, Anne, (Dick – Use cases).

User Interface:

• General User Interface (UI) roadmap with order of goals and what needs to be accomplished
• Adequate description of User Interface  architecture
• Creation of first sprint specification
• Partial implementation  (Web Interface)

Server:

• Generic code clean-up
• Unifying server calls (JSON-related)
• Support for Twitter API
• Document how compass can use other persistency types
• Merge Action branch into official trunk
• Further work on java API (GetMessages)

Note: Some of these tasks have already been accomplished.

I’ve been thinking about how to integrate various collaboration tools into ESME based on the use case that users wish to exploit a particular ESME conversation (starting from a single ESME message to series of related messages) as the basis for another type of collaboration (discussion, wiki entry, document, etc.).  By the way, I’ve discussed such interactions in other contexts  in SAP SDN blogs.

I’ve come up with two basic options:

1. The ESME UI is adapted to add new functionality via a new icon on each message panel.
2.  Use of an ESME bot  that listens for messages with certain tags and then performs various activities on the part of users. 

Of the two options, the first involves changes in the ESME core that might cause problems inasmuch as additional functionality might lead to a cluttered UI which would reduce the ease of use associated with a simple UI. The second option appears to be more applicable to a variety of use cases and doesn’t require UI changes based on every new use case. 

A Concrete Use Case 

I’d like to sketch out the steps associated with the use of such a bot. Let’s focus on the use case where an ESME conversation is being used to create a wiki page with the contents of the conversation

Read it all..

One idea that we’ve kicking around is an ESME client that is context-sensitive.  The idea would be that just ESME messages are displayed that reflect the particular contents on the page on which the ESME client is running.

For example, an ESME client is running on an internal blog page that deals with sales problems with a particular customer “Patriot Chairs”. The ideal situation would be that the ESME client shows all messages that are tagged with the words “sales”, “Patriot Chairs”. The idea is those reading the page could get a feeling for the corporate “heart-beat” dealing with these areas.  The blog reader would be able to see other messages relating to the particular customer in question and sales issues in general.  If desired, the blog reader could then interact with those individuals talking about these matters.

To meet these requirements, the ESME client would have to respond to how a particular blog page is tagged. Whether this tag is based on URL-based tags or another mechanism would be implementation-specific and would in all likelihood be based on the details of the integration environment.

This use case would also probably require new REST API functions (for example, “GetFilteredMessages” http://esme.example.com/messages?tag=sales) to be able to search the public timeline for particular tags.  Another option would be to have a client that performs the filtering actions in the JSON transformation.

Last night, we performed our first load test of the ESME core. We used the cloud-based installation that is running on the Amazon cloud via Stax.  We used a test suite with nine parallel Java applications threads that made  calls to the REST-API and performed actions such as logging-on and sending a message. This occurred for each of the threads about 250-500 times.  At the test “peak”, we had approx. 1000 concurrent connections.  The source code was based on the old Google Code-based version which includes an older version of the Scala with a known memory bug. This was our first test and we didn’t use the clustering functionality of Stax. We also didn’t run in dedicated server pools available via Stax which would probably improve our performance as well.

Read it all..

Daniel Koller has created a  ESME Mailet is a plugin for the Open source Mail Server Apache James which allows users of an ESME infrastructure (both humans and technical systems) to send email messages to an ESME instance using their own identity. Messages sent through the mailet show up as personal messages from these end users, so people following you get them in the respective blogging timeline.  The install guide is available on Google Docs.

This new functionality is especially useful for integrating legacy applications into the ESME environment inasmuch as many such applications might not be able to integrate via the REST API but usually have the ability to send emails. Furthermore, an email interface also allows users to post ESME messages with mobile devices. This is a useful functionality until native ESME clients for mobile platforms are available.

Now that ESME works in the cloud, I was thinking about ways to take this existing installation and going viral with it so that a variety of cloud-based ESME  installations could be easily created.   The following steps would be necessary:

1. Download existing ESME source code from Stax
2. Create new application in Stax via the application console
3. Download the new application into the local installation
4. Create new database in Stax via the application console for the new ESME application
5. Copy the existing ESME source into the deployment directories of the new application
6. Change the source code to meet the basic style configurations of the new ESME installation
7. Change data-source configuration of the new application to point to new database
8. Upload the new application

 One shortcoming with this description is that as owner of the original ESME application, only I have the ability to create copies of the current installation.  At the current time,  just individuals who I invite to join my Stax application team have access to the ESME source deployment. Ideal would be the ability to create public clones of Stax-based deployments.  Then, other Stax users could very easily create new cloud-based ESMEs.  Although ESME is open-source and the source is available to all via the Apache repository, it would be easier if this was possible via Stax itself.

As a ESME supplier (someone who creates, maintains and/or supports cloud-based ESME installations for others), these steps could be patched together to create a process to very rapidly create micro-blogging sites for customers. The only task that requires more effort would be the customer-specific adaptions regarding styles, etc.

In a recent blog about Enterprise Twitter,  Mike Giotta of the Burton Group described 10 questions that enterprises will ask regarding the use of microblogging tools. One question was “Integration with Microsoft Communicator and IBM Sametime”. At first, I disregarded the idea and thought “who needs an integration? They are two tools based on two different collaborative metaphors”.  After a round of sauna, I started to think about ways that the two might complement one another.  Based on this “heated” discussion with myself, I set out and tried to create use cases for this integration.

Read it all..

Dennis Howlett recently sent the ESME team a link to a long article about VAXNotes which was a collaboration tool that was active in the 1980’s (!) at  Digital Equipment Corporation (DEC). I was reading this article and had a real case of Deja Vue.  The article provides an excellent description of the use cases involved in the use of the tool as well as the corporate culture that was necessary for its widespread usage within DEC.

As I read this article, I realized that many of the use cases that micro-blogging tools are meant to solve are actually problems that have been around a long time – for example, the need for employees in distributed companies to be able to collaborate and discover individuals (those old “weak ties”) who can solve their problems. The high level of employee involvement at DEC regarding VAXNotes (irregardless of job title, location, etc.) shows that the assumption that active enterprise collaboration is just restricted to the generation who have grown up with Web 2.0 technology is wrong. VAXNotes emerged in an environment that was pre-Web 2.0 – indeed it was pre-Web 1.0.  Thus, the assumption that it is just Generation Y employees who will be the primary users of such micro-blogging tools is misleading. If the corporate culture (as evidenced by the DEC article) supports such tools, users from all generations will use them.  

Read it all..