2011-07-07
MapReduce Made Easy
As a nice little side project while here in San Francisco, I took an increased interest in the MapReduce API that Google App Engine offers - disclosure: I'm here on an internship at Google :) So with that completely biased goal in mind, I spruced up the MapReduce API home page and the sample app that comes with it (source to be released soon). In the meanwhile, however, here's a video of me showing the new MapReduce Made Easy app in action - enjoy!
2011-05-04
Pipeline API on AppScale
Update (5/25/11): AppScale 1.5 adds transactional task support, so don't you don't need to make the changes listed here to get the Pipeline API up and going if you have that version or anything newer. Otherwise, read on!
Recently I've been hacking away at the super-cool Pipeline API that Google App Engine puts out. It's an easy way to run otherwise complex workflows over large datasets, and of course I wanted to get it working on AppScale as fast as possible. Since we don't have transactional tasks working in AppScale right now, if you want to play with the Pipeline API in AppScale, all you have to do is open up the Pipeline API's pipeline.py file, and replace this:
task.add(queue_name=self.queue_name, transactional=True)
with
task.add(queue_name=self.queue_name, transactional=False)
and you're good to go! Here's a screenshot showing it running on our AppScale deployment:
Of course, we'll have a version of the API with these patches (and eventually, transactional task support so you won't need a separate API in the first place) out with the 1.5 release. Enjoy!
2011-04-25
Talk at LA.rb and SoCal
Just gave a talk last week at LA.rb about the AppScale project (essentially a shortened version of my SBonRails talk) - here are the slides:
I also gave a talk at the SoCal Programming Languages and Systems Workshop on Neptune, our metaprogramming extensions to Ruby that make high performance computing super simple - here are the slides to that:
I also got the video of it, which I'll post a link to when I get it on the YouTube.
Enjoy!
