Monday, August 23, 2010

SEDA in Streametics

Recently, I came across Matt Welsh's "A Retrospective on SEDA". Matt writes
If I were to design SEDA today, I would decouple stages (i.e., code modules) from queues and thread pools (i.e., concurrency boundaries). Stages are still useful as a structuring primitive, but it is probably best to group multiple stages within a single "thread pool domain" where latency is critical.

SEDA was one of the research projects that had influenced Streametics' technology. Streametics was the company I had co-founded in 2006. To jump start on the SEDA based implementation, I had started with the code base of Commons Pipeline and made several modifications to it. One change relevant to the above discussion was that of an application handler. This handler was introduced to separate out the application logic from a queue and thread pool of the stage. The stage driver would call out to the application and pass on the event.

Second modification was about allowing the stage designer to receive events synchronously from the previous stage without the queue -> thread pool route. This was done to allow the application designer to group multiple stages such that the processing of the event passing through these stages happens in the same thread without any context switch.

We had also externalized the pipeline (or route) configuration in XML. One could design the whole Streametics application by designing pipelines of stages and connecting these with endpoints and adapters. This way, one could form a graph of stages in the application using a tool. We had a library of stages, endpoints and adapters. The application developers could also use simple APIs to write these components on their own. We used typed XML-Java binding (XMLBeans) at the time of deployment. In a hindsight, we could have used Spring beans framework to wire-in these components.

When we were developing Streametics' event engine, Apache Camel was in very early stage.Camel supports SEDA. However, it required writing routes in Java then. Since then it now allows to do the wiring using Spring Beans. Oh well, there goes a little retrospective on Streametics...


No comments:

Post a Comment