At POSSCON 2011 I gave a talk titled “Data Processing @ bit.ly” in which I shared about some of the open source tools we have written at bit.ly, and the approaches we use in data processing.
Much of this talk covers 3 core applications we have developed @bitly: sortdb, simplequeue, and pubsub; all of which can be found on github http://github.com/bitly/simplehttp