July 18th, 2009 chris
A while ago, I compared the C programming language to a miniature nuclear weapon from Fallout 3. It’s still an apt metaphor, as the language does give you more than enough power to take care of the biggest tasks ahead of you and blow yourself to kingdom come in the process. It takes an expert to really use it but once you figure it out, you can avoid a lot of the pitfalls involved.
Ruby, on the other hand, is an entirely different beast. It relates more to one of DC’s most famous supervillains, the Joker.
Read the rest of this entry »
Posted in Uncategorized | 5 Comments »
May 1st, 2009 chris
…if you’re a sysadmin, of course. pconsole is a pretty cool utility that lets you enter in your *nix commands from one machine and have it run wherever you need to. It connects to terminal windows that are already open and connected to the machines you want to run it on, so the output is broken up every nicely. So if there’s a problem running a command on one machine, it’s really easy to know which machine it is. And it works on Mac OS X out of the box (almost)! All you have to do is run “chmod +x /usr/local/bin/pconsole” once you’ve installed it. I think the obligatory picture from the pconsole site pretty much sums up how cool it is (although it’s hard to really know it until you’ve racked your brains copying and pasting to many many machines)
Thanks pconsole and thanks to Server Fault! It’s the new sister site to programming-QA site StackOverflow and looks pretty nice so far. Since I’m more in programmer-land than sysadmin-land I don’t get quite as much use of Server Fault as StackOverflow, but it did lead me to this gem called pconsole!
Posted in Uncategorized | No Comments »
April 23rd, 2009 chris
Recently for one of my classes I got the chance to research a parallel programming technique and it’s applications. Being a bona-fide MapReduce whore-in-training, I naturally had an easy choice. Since this is “Part 2″, we cover the era I’ll dub the “Return of MapReduce”, in which Google re-introduces and rebrands MapReduce as the technique we love to hate and love to love (choose whichever suits you best) and new advancements are made with it. This covers about the 2004-2008 time period, in contrast to Part 1 (the beginning of existence-2004), where map-reduce exists as part of Lisp, MPI, and other environments but is not particularly accessible to the average programmer, and Part 3 (2009-present), which I will leave as a surprise for those of you diligently following along at home.
With that said, here’s part 2, beginning with the paper that changed MapReduce and revived mainstream interest in the technique. Enjoy!
Read the rest of this entry »
Posted in Uncategorized | No Comments »
March 25th, 2009 chris
One of the most viewed posts we’ve written about here is our post on MapReduce from a little while ago, so when picking out the next paper to look over, I thought something related to it would be optimal. With that in mind, Yahoo! Research has a relatively new paper that they published in SIGMOD ’08, titled Pig Latin: A Not-So-Foreign Language for Data Processing. Pig Latin bills itself as a natural progression of MapReduce in several ways, and indeed looks pretty interesting.
Read the rest of this entry »
Posted in Uncategorized | No Comments »
February 26th, 2009 chris
Now that our coverage on the Big 5 Google papers has concluded, I thought it may be handy to aggregate the links to them all in one handy spot. That being said, here’s the links to our series of reviews (sorted by post date / chronological publishing of paper):
- Thoughts on the Google File System: How one constructs a fault-tolerant file system with which to build everything else on that performs optimally on huge data sets.
- Thoughts on MapReduce: How to write programs to process huge amounts of data as quickly as possible given the huge amount of resources available.
- Thoughts on BigTable: How to store data in a more structured format yet keep replication and high speed processing.
- Thoughts on Chubby: How to locate specific nodes serving a specific purpose though the use of a highly available naming service.
- Thoughts on Paxos: How to implement a consensus algorithm to ensure agreement amongst nodes in the presence of failures and keep high performance in the meanwhile.
So check them out if you haven’t, and be sure to come back next time for our review of Amazon’s Dynamo!
Posted in Uncategorized | No Comments »
February 26th, 2009 chris
After a month of Google papers, we finally come to the last paper in our Google reading list: Paxos Made Live. Like the Chubby paper from last week, this paper doesn’t present any new research, but instead tells us how to build a system from the current research out there such that it can withstand incredible load. It also aims to show how we could possibly be confident in a non-trivial amount of code as well as optimizations that can be made to improve the performance and reliability of the system. So how did they do and how did it turn out? Read on!
Read the rest of this entry »
Posted in Uncategorized | No Comments »
February 20th, 2009 chris
The last few times around we’ve been talking about performance. Extreme performance. We’ve talked about how fast the Google File System is, how fast MapReduce is, and how fast BigTable is. Yet all three rely on a common service we hinted upon last time, a lock service (amongst other things) named Chubby, the topic of today’s discussion. Chubby doesn’t bill itself as high performance like the other services we’ve mentioned, but is designed with high availability in mind. Let’s dive in further and see what exactly makes up Chubby and why it’s so special in the Google world.
Read the rest of this entry »
Posted in Uncategorized | No Comments »
February 13th, 2009 chris
Now that we’ve looked at the Google File System and MapReduce, we come to the next paper on our reading list: BigTable. It comes to us two years after the MapReduce paper and presents an interesting new step in databases. Like MapReduce, it is another notable paper and takes a very familiar programming construct and revamps it to new heights. Let’s see if I can whet your appetite with a spoiler:
Maybe a better name for it is Big Hash Table.
Read the rest of this entry »
Posted in Uncategorized | 1 Comment »
February 4th, 2009 chris
The first paper on the extremely interesting list of distributed systems readings is the Google File System. It is not a particularly new paper, published way back in 2003 at the Symposium on Operating Systems Principles (SOSP), but is still a great read. I’ve chosen to start with this paper over the other, more recent papers, since this paper is both the earliest chronologically and the lowest layer of abstraction. If you have never heard of it before this point and have the vaguest interest in file systems, then read on!
Read the rest of this entry »
Posted in Uncategorized | 2 Comments »
February 4th, 2009 chris
So now I’ve finally moved onto my own hosting and now that I’ve moved the blog over, I figured I’d finally take of it the way I should be. I always hate to say that I’ll be posting more often, since I always say it and so far I haven’t been great about it. However, a while ago I did read a good post on one of my favorite programming sites about how to fix up WordPress and make it “sing”. Check out how you can modify your MySQL conf file to work better with WordPress.
Read the rest of this entry »
Posted in Uncategorized | No Comments »