ESME Blog

The ESME Project has come a long way in only five months, from its genesis in a conversation between people who had never met, to a demo-grade prototype which has been shown on a live stage to a crowds of thousands in Las Vegas and Berlin. This has been achieved in the spare time of a fantastic team of people, but what I want to talk about here is the support we have had along the way. The people and companies mentioned here deserve our thanks for being there when we needed them on our journey. In alphabetical order, they are as follows:

Adobe

An early, though temporary, member of the ESME team was Matthias Zeller of Adobe. Matthias was instrumental in introducing the team to Scrum, a project management technique which revolves around multi-week implementation “sprints” and daily quickfire team status meetings. Matthias’ great gift to the team, via Adobe, was the use of an Adobe Connect Pro room and a daily slot on a conference call number. Adobe Connect Pro allows us to gather as a team in a shared space where we can share live desktops, make notes and draw on a whiteboard together – key tools in the early phases of design. Adobe’s help has enabled us to use Scrum to function as a globally-distributed agile team, able to resolve in minutes issues which might have taken days to resolve via email exchange.

Assembla

While Adobe provided us with real-time collaboration tools, the good folks at Assembla provided us with all the asycnhronous collaboration tools that are essential to running a software project: source control, bug tracking, project documentation wiki and file upload/download. On top of this, they providing their own functionality to help with Scrum-based projects, and they even do this for free for modestly-sized projects such as our own. In our case, we just checked the box labelled “Trac with Subversion” and off we went. In their own words:

Trac is a popular open source ticketing system, with the mission to “help developers write great software while staying out of the way.”, if you are familiar with Trac, this package is perfect for you. In this configuration, you will get a Subversion repository you can easily manage and other enhancements like a simplified team management, HTML alerts (called “notifications” in trac), and hourly and weekly alert summaries. You can upload your existing Trac directory and import your existing Subversion repository.

Subversion is the most popular centralized source code repository and version control system. Our subversion tool includes email alerts on commit, code browsing, linking of comments to tickets, and a post-commit hook to trigger your own actions. We know that reliability is important for subversion users, so we backup to failover servers.

If Trac and/or Subversion are not your particular tools of choice, they also offer source control using Git & Mercurial, and their own Wiki and ticketing system. In their latest offering there is a Mylyn connector for Eclipse (Eclipse-integrated source & ticket control with Mylyn is awesome) and even a Twitter tool so you can feed your project activity into Twitter. Hmmm… must talk to them about ESME integration.

Families & Friends

Everyone in the ESME team has worked on the project in their own time. As we have approached our various internal deadlines, this has meant spending significant amounts of time away from our loved ones, who have supported us amazingly well in our endeavour and complained only when we have been truly overdoing it. One strength of our team is that we are always able to carry on if individual team members need time and space, ready to pick up again when they come back to the project, but even so we are fortunate to have got this far with personal relationships still intact.

Google

Our primary means of team communication in between daily Scrum calls has been a private Google group for the core team. We are slowly moving most of the traffic from this group to the public esme-dev* Google group, but we still use the private group for organisational discussions, as well as plotting how to win the next Demo Jam away from the eyes of our competitors.   In addition, since we took the project open source in September, we now use Google Code* to host the project source code, issue log & documentation. I don’t think Google have made much (if any) money from us in this way – there aren’t too many advertisers who want to be associated with an open-source enterprise micro-messaging platform just yet – and so Google deserve our thanks for providing these services.

Joyent

Joyent provide cloud-based hosting on heavy-duty Sun servers running OpenSolaris. In our earlier days, they generously offered to host an ESME instance for us, running on SAP’s NetWeaver platform. We would really have loved to take them up on the offer, but we couldn’t figure out any way of getting a NetWeaver license from SAP to do this. We’re still enaging with SAP on this issue via the SAP Mentor programme (several of the ESME team are SAP Mentors), but right now SAP just don’t have a licensing or operational model for working with community-based teams. They are definitely thinking about it – in some quarters at least – but we are not there yet, which is a shame. There is a lot to be gained from showing how SAP can work in the cloud, and Joyent provide the kind of heavy-duty infrastructure that SAP installations can often require.

Plurk

Plurk was the service that really started us thinking about the potential for micromessaging in the enterprise. During a Twitter outage, a bunch of us congregated their for a while. The differences between Twitter and Plurk got us talking about what we in the enterprise software world would want from such a service, and that conversation led directly to where we are now.

RedMonk

Open-source analyst firm RedMonk is made up of the smartest quartet of people this side of MIT, and they have been enaging with us in small but important ways ever since we started. Among many other contributions from them, this article was prompted by James Governor quite rightly pointing out that we’d been using a lot of free help and not thanking anyone publicly for it. James tell it like it is, and that is a highly valuable attribute. Michael Coté did a great video interview with some of the team, Stephen O’Grady has helped us with some thinking about open source business models, and Tom Raftery has us thinking about how to make software green – in our case, designing a messaging system with awesome performance and scalability, meaning fewer servers generating heat in datacentres.

SAP

Most of the people on the ESME team are involved in the SAP ecosystem in one way or another – working for SAP partners, SAP customers, even for SAP itself, but also passionate and dedicated to what SAP is about. Many of us are SAP Mentors, and of course our public demonstrations of ESME to date have been at the SAP Tech Ed conferences. SAP has paid for our entry to these conferences, and has even loaned us a server hosted at the SAP Co-Innovation Lab so that we can run ESME on the SAP NetWeaver platform. Most of the enterprise-oriented thinking around ESME is based on our experience working in the SAP arena. We hope that SAP sees the value in this, and is able to come up with some kind of community licensing scheme that will make projects like ours viable.

Siemens SIS

One of our team works for Siemens SIS, and the Siemens name is one of the factors that got us an entry into Demo Jam – there’s nothing like having a big-name customer reference on the entry. More recently, Siemens SIS have been a really useful sounding board for discussing such things as cloud-hosted SAP NetWeaver and some of the large-enterprise-scale business scenarios for ESME.

Twitter

Last on the list, but reallt the first: Twitter is really where this all began. The core team behind ESME started off as a Twitter-based network of people from the SAP community, and we still regularly use Twitter alongside ESME. We do not compete with Twitter at all – in fact we are completely complementary. Twitter is the PSTN, we are the PBX – or to translate that from telecomms acronyms, Twitter runs a public, accessible-to-all communications service, whereas we focus on running within and between the boundaries of companies. There are overlaps in functionality, but a completely different emphasis on features.

The following diagram gives a big picture of the basic usage scenarios in  ESME. 

The ESME message cloud contains all messages that are created.

Note: This simplistic description is based in the assumption that there is just one ESME server instance. In reality, a federated environment may be present increasing the environment’s complexity as well its architectural possibilities to meet more sophisticated enterprise requirements 

The following users are present:

• Bob: This user creates messages. Bob is following no one and has created no actions to filter messages based on tags
• Sue: This user just receives messages. She is following Bob; therefore she receives all his messages
• CRM System Zeus: This user sends messages into the ESME message cloud based on certain business events. The messages’ contents and tags are set according to the particular business event that originated them.
• John: This user has created an action based on the tag “Acme”. John performs a certain activity (for example, sending him an email) when messages with the tag “Acme” are present in the cloud.  Although John does not know Bob and is not following him, John receives a message from Bob that is tagged with the word “Acme”.  Although John does not the CRM System Zeus and is not following it, John receives a message from this user that is tagged with the word “Acme”.  John also creates messages.
• Bot Timesheet: This user watches the ESME message cloud for messages with the tag “Hours” and receives messages that are tagged in this way. This user receives a message from John, processes the message, converts into a more appropriate format and passes it on to a back-end system – in this case, a xRPM system.

There are two possible basic scenarios dealing with integration patterns.

• Sending ESME messages
• Receiving ESME messages

Of course, there are some applications (Web Client, ESME Desktop Client, etc.) that represent both patterns but I’d like to concentrate on those applications that fulfill one pattern but not the other. 

Sending ESME Messages 

Applications that fulfill this pattern are those that supply the ESME cloud with information in the form of messages. These applications may either be based on human-created content via UI or machine-based content. Examples of this integration pattern are:

• Javascript Client
• VBA Client
• Java Logger Component(presented at the DemoJam in Berlin)
• ABAP Development Lifecycle Watcher (presented at the DemoJam in Berlin)
•  A watcher process that deals document management-related (“check-in”, “check-out”, etc.) events. In Sharepoint, this functionality might be linked to the Microsoft Workflow Foundation. In SAP’s Knowledge Management & Collaboration (KMC), this could easily be implemented via the KM IResourceEventReceiver

Receiving ESME Messages

Applications that fulfill this pattern are those that receive ESME messages and then process the information in some manner.. One of the main requirements for such applications are the ability to deal with long polling (ala Comet-style ) communication in order to avoid performance problems on ESME server instances. 

Possible implementations of this pattern include:

• ESME Bots
• Light-weight applications that just show certain messages based on tag-based filters. An integration via javascript-based scripts (for example, dojo) would allow use in most browser-based scenarios.

One of the interesting exercises when considering integration patterns for ESME is to distinguish between human and machine-based scenarios.

I wanted to share a picture that summarizes the interesting possibilities of both types.

At the Demo Jam at the SAP TechEd in Berlin, we presented a NetWeaver-based logger that sent ESME messages in response to exceptions in the Java Stack. The code is available on our Google Code repository. Just like our ABAP-based integrations, we have a created a JAVA API class that wraps the REST-API.  The wrapper class isn’t complete yet but it contains enough functionality to be able to send ESME messages via Java.

The method below is the “heart” of the ESME functionality and sends message via ESME.

public void handleEvent(MessageEvent event)
{
   LogRecord rec = event.getLogRecord();
   Message esmeMsg = new Message();
        
    int msgLength = rec.getMessage().length();
        
    String message = rec.getMessage().substring(0,(msgLength<255)?msgLength:255);
       
    String[] lines = message.split(”\n”);
    StringBuffer wrappedMsg = new StringBuffer(255);
    for (int i=0; i<lines.length; i++)
    {      

       wrappedMsg.append(wrap(lines[i], 55));
     }
     esmeMsg.setText(wrappedMsg.toString());
        
     String[] tags = new String[4];
     tags[0] = Severity.toString(rec.getSeverity());
     tags[1] = rec.getApplication().replace(’.',’ ‘);
     tags[2] = rec.getLocationName().replace(’.',’ ‘);
     tags[3] = rec.getUser();
     esmeMsg.setTags(tags);
  
     esmeMsg.setVia(”NetWeaverLogger”);
  
     esme.sendMsg(esmeMsg);
 }

With this code, it should be relatively easy to integrate other message sources into ESME.

Thanks to Darren for contributing the code.

Last night, there was an interesting twitter discussion (including @dan_mcweeney, @mrinalwadhwa, @dhague and myself)  about bots that process ESME messages. Originally, the conversation started with ideas concerning an integration of SAP’s xRPM into ESME.

The following ideas originate from this conversation

• A bot is a ESME client without a UI that is specialized to deal with the processing of one or more tags.
• A bot may be developed in any language that has the ability to perform “long polling”.  Although Adobe AIR might be one choice, ABAP (still untested regarding comet-like “long-polling”) and java appear to be better choices.
• Although actions – set by individuals in their respective ESME clients – might be one option to perform this functionality, if a large number of individuals are using similar actions, then there be a performance impact on the central ESME server instance.  A bot that is just another ESME client may be a better option.
• One example of this functionality would be a bot that tracks a particular tag in messages sent by developers. When messages with this tag appear, the bot submits the messages to xRPM  to track their project accounting for this development team.
• There might be a distinction between light-weight tag processing via actions and heavy-duty tag processing via bots.
• The bot would have a better integration possibilities (via web-services, RFCs, etc) with back-ends than the existing ESME clients with their primitive ”Post” actions”. The bots would also be able to deal more effectively with authentication issues with back-ends.
• One another idea is @dhague’s “ESME federation based on ESME using its own REST API as a client could work for server-to-server. xRPM just implements ESME send_msg API.” This idea probably has to be examined in more detail but definitely has potential.

One thing that the discussion demonstrated to me is the importance of integrating microsharing applications into the entire enterprise landscape.

DemoJam only gives each team six minutes to present their innovations on stage.
This is not much, so when we were given the opportunity to give a 45-minute “Meet the Jammers” in-depth session the following day in the Clubhouse Theater, we happily took it.

This session was streamed live on Ustream.tv, and gives you a deep dive into what makes ESME tick.

Thanks to Twan van den Broek for providing the picture

Introduction

“We are all part of teams. Java teams, ABAP teams. Communication in these teams can be a challenge”

This is how we opened our Demo Jam entry last week at the SAP TechEd in Berlin – the complete script is here. Based on my experience in Berlin, lately I have been thinking a lot about communication and its relationship to a team’s success.   

The story behind the Demo Jam in Berlin

How do you define “success”? This one word has a multitude of meanings – many of which are subjective in character. For me, “success” means overcoming adversity and the ESME demo in Berlin is an excellent example of a team dealing with enormous pressure and not crumbling under it.

Not many people know the “real” story behind our participation in Demo Jam and what happened during the Demo Jam itself. Craig said that we had a problem with one of our servers. What really happened was that there were network problems with our main ESME server. The reason for this problem has been discovered but is largely irrelevant to the point I want to make in this blog. We discovered the problem at the final technical check 5 minutes before the Demo Jam started. Darren Hague, Anne Petteroe and I took Darren’s laptop back stage and tried –along with the technical staff associated with the Demo Jam – to localize the problem.  After realizing that there was a problem with our initial server, we first tried to correct the problem. When we realized that this wouldn’t work, we decided to use a secondary server.

You have to remember that all this time the clock is ticking and Craig Cmehil, Jeff Word and other DemoJam staff are getting increasingly worried.  Are we going to make it on time?

Since the Java logger code was created by Darren, he could make the changes himself. The ABAP code, however, was contributed by Thomas Jung and Athavan Raja Durairaj  Thomas Jung was somewhere in the audience of over 2,500 screaming fans. We had to find him and fast. Anne went out and found him and brought him behind stage to fix the ABAP code. At this point, we had about 15 minutes before we were supposed to go on. 

At some point, Dennis Howlett  came back stage to see if he could help us. During this time, he sent off a tweet on Twitter with one word “backstage”. For all those who follow Dennis on twitter and know his wit, such one word tweets are exceedingly rare. This one word was proof of the strain that he (as well as the entire team) was experiencing.

Originally, we were the fifth from six teams. Realizing that we needed more time, we asked if we could go last.  We were granted this request. We had gained 6 minutes. Finally, after changing the code to communicate with the new server, we had to test our demos.  At this point, Craig came back stage and asked how long we needed. We said “10 minutes”. Craig then said “You have 10 minutes”.  We struggled and tested everything as much as we could. At two minutes before we were to go on stage, Jeff came and asked “Are you ready? Yes or no?” We said “Yes” and went on stage. As Anne was setting up the laptop on stage, we realized that we didn’t have a password to logon to the remote desktop. I had to race back stage, grab Darren and race back on stage to unlock the Remote Desktop. After doing this, we were ready but were still very worried. Moving from back stage to center stage meant disconnecting from the network and then reconnecting. We had no idea if everything would work.  When the lights focused on us and Craig started talking about sacrifices to the Demo Jam God, I thought to myself “We’ve sacrificed enough in the last hour: 3 kilos and 40 new grey hairs.“

For the Demo Jam,  I just had to give the introduction to ESME and provide the transitions between demos, Anne was doing the actual demos. We were both under unbelievable pressure. Speaking in front of a crowd of 2500 is tough. Combine this with uncertainty whether the technology is even going to work is even worse.

Although we would have liked to won the Demo Jam in Berlin, it was really of secondary importance. If you look at how Anne and I responded after our six minutes is over, we don’t ask the audience to clap. The first thing we do is to give each other a high five. At this moment, we were overwhelmed that everything had worked. If other team members were on stage, we would have included them as well. 

Lessons from the Demo Jam and ESME

Some might say that this is a personal story and too emotional and really doesn’t belong as a blog on a technical site. That is just my point; it is this emotion – this passion – that enabled us – the ESME team – to deal with a crisis.

Remember, the ESME project is composed of about 20 members spread across the globe.

This dispersion means that really have / had no possibility for face-to-face personal interaction. Although many of us had never met each other in person we had contact on a daily basis. We communicated with each other via twitter, ESME, emails, telephone calls etc. Our network was active. We were connected.

I consider these my individuals more than just “project” members -I consider them my friends.  We all have the same passion not only for ESME but also for collaboration in general. It is something we thrive on as evidenced by our exceedingly active communication with others – our networks.  When we communicate, we just don’t communicate about ESME or technology or our jobs. We communicate about our lives. It is the resulting bonds that give us our strength and our ability to master thorny situations such as the technical difficulty associated with our participation in the Demo Jam.

A note: Many of those in ESME (but not all) are SAP Mentors. If you look at the characteristics needed to be an SAP Mentor:

• Hands-On Expert in an SAP Product or Service
• Collaborative attitude
• Good Communicator
• Preferably working at a Partner or Customer of SAP
• Interested in Improving product and services of SAP and the relationship of SAP with customers, partners and prospects

you will see that are interesting parallels to the topics that I’ve just discussed.

The current state of software development

Now, you might be thinking to yourself – “Give me a break. Where is my handkerchief? I’m starting to get teary-eyed. What relevance does this have to software?”

If you look at the how software development or projects in the current enterprise environment are typically structured, there are usually short-lived projects based on a global team that come together for the first time in the project and then disengage when the project is over. 

How often do we hear about failure in such global teams? When the distance is so great, you usually can’t really get to know one another. There might also be cultural differences which lead to misunderstandings at a personal level. Unfortunately, to achieve optimal performance, it is often critical to bond the team members as rapidly and efficiently as possible. If a problem occurs in a software project and it is necessary that team members go the extra mile to meet a deadline or solve a difficult problem, it is usually the more cohesive team that will be successful.  I’m much more willing to help some one whom I know than a complete stranger.

There are currently a number of new community projects that are being formed in the SAP communities.  Community projects are project that take place in the community – that means usually outside of work hours and usually individual participation is voluntary. What I have discovered is that what keeps those individuals involved is their passion and the relationships to others in the team. 

Conclusion

I think it was these personal bonds that allowed us to deal with the stress / pressure involved with the Demo Jam in Berlin. We didn’t want to let each other down.  We supported one another.  This experience has taught me a lot about what makes teams successful – not only those in community projects – but in other contexts as well.  It is the personal relationships – the web that connects us with others – that is critical. As I have mentioned, these bonds were largely created by communication at various levels – twitter being the probably most important, because we were a distributed team.

Thus, my experience in this community project demonstrates the importance that such microsharing tools (such as Twitter or ESME) can play in teams. Although our script in the Demo Jam focused more on interactions of a technical nature (Java logger, ABAP watch, etc.), ESME is one example of a tool that creates bonds of a more personal nature.  Such platforms are instrumental in building strong teams that act well in times of adversary or stress which are unfortunately commonplace in the current corporate environment.

Note: I also posted this blog on SDN.

Although the script below doesn’t contain the exact words that we said at the Demo Jam in Berlin, it was our basis. 

Dick

We are all part of teams. Java teams, ABAP teams. Communication in these teams can be a challenge, and is most often not as rapid as we would like it to be.

Every day we are flooded with emails and hand on your heart, do you read every line in all your emails? When you need to communicate with teams, how do you usually do it? You send an email to a distribution list. Spray and pray.

With ESME, introduces a new concept in communication, which lets you filter out the noise! You decide which information you would like to receive and respond to.

Let’s look at three common problems:

1. How do I get answers to technical questions? Today, I use that old email distribution list and hope I find the right person. 

2. How do I deal with problems that occur in the development lifecycle? Today, I have to dig through various systems to find out what happened.

3. How do I deal with developers that change code without the telling the team? Today, I have to wait for my code to break before responding.

Let’s look at a simple problem: getting answers to technical questions

Anne

I have a CE installation problem. I have ESME open in my browser. I am going to type in my message here which goes to my followers and add these extra words, which are called tags. What are tags? Tags are special keys that other users in my team or company can listen or subscribe to. Because I add these words I make sure that my message goes to those people who are interested in this subject.

Darren is from another part of my company and has subscribed to tags that interest him. Remember I don’t know Darren. He gets a message about my problem and sends me a message in return referring to the SAP Note that helped him and solves the problem.

I don’t find an answer. I find the right answer.

Dick

With ESME, I can communicate with people whom I don’t immediately know. I can tap into tribal knowledge. Now what if we extend this so that the system talks to the same group of people?

Let’s look at a more complicated problem: how do I deal with issues that occur in the development lifecycle?

Imagine this, I am developing a WebDynpro application. Usually, when an exception occurs, I have to log onto the NetWeaver Administrator and open up the log viewer and find my exception amongst the 1000s of other exceptions. †Wouldn’t it be cool if this exception was sent to me?. When they have problems, maybe I will be informed but maybe not. Wouldn’t it be nice if I was sent this information.? Better yet, wouldn’t it be useful if my entire network was informed about this problem?

Let’s see how this works.

Anne

I am going to create an exception with my WebDynpro application.

See, a message just appeared in ESME. This message is created with tags. Those individuals subscribing to these tags will get the message.

You might think that people would be flooded with information but because, they choose to receive messages on certain topics or individuals, they decide what they receive.

Dick

Let’s look at a real complex problem: how do I deal with developers that change code without the telling the team?

Picture this: someone goes in and changes your code or one of your development objects. What happens – your code breaks. I bet this never happens to you. Now you have to find who broke the code and why.  Wouldn’t it be cool, if the system informed you when others changed their code?

Let’s see how this works.

Anne

I will now pretend I am that developer down the hall, and I will change one of my function modules in his name. Because this function module is one of my development objects, I subscribe to any changes which happen to it.

I just activated the function module and as you can see the system immediately sent a message to me, which arrived in my inbox in ESME.

The message tells me which object was changed, who changed it and when the change happened.

I can now respond before code breaks.

Dick

We have just shown you three ways to solve problems you experience on a daily basis.

1. How do I get answers to technical questions?

2. How do I deal with problems that occur in the development lifecycle?

3. How do I deal with developers that change code without the telling the team?

All with one tool – ESME.