Byzantine Reality

Searching for Byzantine failures in the world around us

Pipeline API on AppScale

Update (5/25/11): AppScale 1.5 adds transactional task support, so don’t you don’t need to make the changes listed here to get the Pipeline API up and going if you have that version or anything newer. Otherwise, read on!

Recently I’ve been hacking away at the super-cool Pipeline API that Google App Engine puts out. It’s an easy way to run otherwise complex workflows over large datasets, and of course I wanted to get it working on AppScale as fast as possible. Since we don’t have transactional tasks working in AppScale right now, if you want to play with the Pipeline API in AppScale, all you have to do is open up the Pipeline API’s pipeline.py file, and replace this:

1
task.add(queue_name=self.queue_name, transactional=True)

with

1
task.add(queue_name=self.queue_name, transactional=False)

and you’re good to go! Here’s a screenshot showing it running on our AppScale deployment:

Of course, we’ll have a version of the API with these patches (and eventually, transactional task support so you won’t need a separate API in the first place) out with the 1.5 release. Enjoy!