Monday, August 23, 2010

SEDA in Streametics

Recently, I came across Matt Welsh's "A Retrospective on SEDA". Matt writes
If I were to design SEDA today, I would decouple stages (i.e., code modules) from queues and thread pools (i.e., concurrency boundaries). Stages are still useful as a structuring primitive, but it is probably best to group multiple stages within a single "thread pool domain" where latency is critical.

SEDA was one of the research projects that had influenced Streametics' technology. Streametics was the company I had co-founded in 2006. To jump start on the SEDA based implementation, I had started with the code base of Commons Pipeline and made several modifications to it. One change relevant to the above discussion was that of an application handler. This handler was introduced to separate out the application logic from a queue and thread pool of the stage. The stage driver would call out to the application and pass on the event.

Second modification was about allowing the stage designer to receive events synchronously from the previous stage without the queue -> thread pool route. This was done to allow the application designer to group multiple stages such that the processing of the event passing through these stages happens in the same thread without any context switch.

We had also externalized the pipeline (or route) configuration in XML. One could design the whole Streametics application by designing pipelines of stages and connecting these with endpoints and adapters. This way, one could form a graph of stages in the application using a tool. We had a library of stages, endpoints and adapters. The application developers could also use simple APIs to write these components on their own. We used typed XML-Java binding (XMLBeans) at the time of deployment. In a hindsight, we could have used Spring beans framework to wire-in these components.

When we were developing Streametics' event engine, Apache Camel was in very early stage.Camel supports SEDA. However, it required writing routes in Java then. Since then it now allows to do the wiring using Spring Beans. Oh well, there goes a little retrospective on Streametics...


Wednesday, August 4, 2010

RESTful web services in CollectionSpace

I have left the CollectionSpace project. However, I occasionally check out what my former colleagues there are up to. We had started work on an early draft for RESTful API documentation a few months ago. Subsequent versions are getting better, you could find it over at here. Because we had to support schema extensibility, you may find multi part payloads in examples cited. However, the draft does a nice job in describing the request and response payloads as well as error codes.

Thanks to Aron Roberts who has been diligently maintaining and updating it regularly and to Patrick Schmitz in shepherding the work. I hope it becomes a good reference for other projects using the RESTful web services.