Byzantine Reality

Searching for Byzantine failures in the world around us

Our Byzantine Reality

This blog attempts to catalog my experiences, and as such focuses on books I’ve read and software I’ve written (albeit heavily biased towards the former). As such, I feel this is a perfect time to talk about why I came across the particular name of this blog that I did, “Byzantine Reality”. Of course, if you’ve read the “About Me” page, you’ll know that Byzantine refers to software of a certain quality and not the empire of the same name. So let’s dive a little deeper and talk about what we really mean when I say we live in a Byzantine Reality.

There are many ways that a computer component, hardware or software, can fail. Most of us are familiar with a computer suddenly dying one day, or a piece of software that just doesn’t work when you click on a certain button. But there is another type of behavior that a component can exhibit: being Byzantine. A Byzantine component, at its core, is inconsistently inconsistent. It can lie or be malicious or meddle with other components that it shouldn’t be talking to. The original Lamport paper describes Byzantine failures as follows:

A failed component may exhibit a type of behavior that is often overlooked–namely, sending conflicting information to different parts of the system.

Examples are simple enough to find: your computer tells you that you have the most up-to-date software on your computer while at the same time harasses you to update your software. You’re playing a video game and you’re at a loading screen that shows a progress of 99% forever because some vital component behind the scenes has crashed. You tell a friend to meet you at the mall at 7pm and another friend to meet you at the movie theater at 7pm and the three of you expect to all meet up at the same place at 7pm. And so on.

My argument is a simple one. The same three failures that apply to computer hardware and software also apply to people. The first two are simple enough to show. People can suddenly die without warning, just like computer components. Some people just can’t do certain things when they’re told to. Just like clicking on the “Quit” button in a program that has crashed and nothing happens, people also are told to do things that they can’t do for some unknown reason (to distinguish it from being Byzantine, they don’t tell you if they could do the task or not).

Yet while the first two types of faults are simple enough to show apply to people, Fortune has been kind enough to give me a real world example to show how we can be Byzantine. Here in the Unites States, quite a bit of money has been given out for the various stimulus packages and the government wants to let the citizens know that it actually had a positive impact. So far so good. They thus decide to make a web siteshowing every part of America and how many jobs were saved or created, as well as other things relating to this money. So after the government spends eighteen million dollars on this web site, you would think that this would be the most amazing web site ever. But no, it is certainly not the greatest web site ever. In fact, the chairman of the Recovery Accountability and Transparency Board told Congressman Issa that:

Your letter specifically asks if I am able to certify that the number of jobs reported as created/saved on is accurate and auditable. No, I am not able to make this certification.

So the new web site from the government whose sole function is to inform the citizens of jobs created and saved cannot be certified as being accurate or auditable. And to put the final nail in the coffin:

In Oklahoma, lists more than $19 million in spending — and 15 jobs created — in yet more congressional districts that don’t exist.

In Iowa, it shows $10.6 million spent – and 39 jobs created — in nonexistent districts.

In Connecticut’s 42nd district (which also does not exist), the Web site claims 25 jobs created with zero stimulus dollars.

The list of spending and job creation in fictional congressional districts extends to U.S. territories as well.

 $68.3 million spent and 72.2 million spent in the 1st congressional district of the U.S. Virgin Islands.

 $8.4 million spent and 40.3 jobs created in the 99th congressional district of the U.S. Virgin Islands.

 $1.5 million spent and .3 jobs created in the 69th district and $35 million for 142 jobs in the 99th district of the Northern Mariana Islands.

 $47.7 million spent and 291 jobs created in Puerto Rico’s 99th congressional district.

To sum up: Obama promises greater government accountability, so (as part of this goal) a web site is made to track jobs saved and created and other signs that the economic depression is over. The web site costs an outlandish amount of money to taxpayers and consists of phony data. Furthermore, since the data itself isn’t auditable, the numbers can’t be verified, so we don’t know which numbers are true and which are false. On top of that, there’s evidence of blatant falsehoods in the data that are provably wrong (e.g., money spent and created in areas that don’t exist). And finally, there’s talk of another stimulus, when we don’t even know what the last one did.

This is truly Byzantine. We do a stimulus package, have no idea of the results of it and at the same time know the numbers are completely botched, and then are convinced that the stimulus worked and should do another one.

Come on. This is completely ridiculous. Computers at least have the excuse that they don’t know any better and that they’re just doing what they’re told to do, but I would like to think that people have just a little more brainpower in their heads.

This effortlessly brings me to a delicious quote from Carl Sagan:

The methods of science – with all its imperfections – can be used to improve social, political, and economic systems, and this is, I think, true no matter what criterion of improvement is adopted. How is this possible if science is based on experiment? Humans are not electrons or laboratory rats. But every act of Congress, every Supreme Court decision, every Presidential National Security Directive, every change in the Prime Rate is an experiment. Every shift in economic policy, every increase of decrease in funding for Head Start, every toughening of criminal sentences is an experiment. […] In almost all of these cases, adequate control experiments are not performed, or variables are insufficiently separated. nevertheless, to a certain and often useful degree, policy ideas can be tested. The great waste would be to ignore the results of social experiments because they seem to be ideologically unpalatable.

So while politics is not as rigorous as science is, we can still see something like the stimulus package as an experiment. There was a problem, we tried the stimulus to fix it, we got some data, evaluate whether it worked or not, and go from there (ideally). But what really happened is more like this: there was a problem, we tried the stimulus to fix it, ???, try it again. Simply put, we have no idea what the stimulus did right now. The logical step then, is to go get the data and talk about it before we throw more money at the problem. The counter-argument is usually of the form “we don’t have time for that! we need to fix the problem now!”. But if we’re not even sure that the stimulus is working, why do it? And furthermore, if the stimulus is hurting us, doing more of it is going to hurt even more!

This is why skepticism is necessary, and why the book I just finished reading, is such a necessity. Carl Sagan’s The Demon-Haunted World: Science as a Candle in the Dark, is truly a masterpiece, from no less than the master himself.

The book is essentially a case study on the need for science, skepticism, and wonder. Science is necessary since it, unlike what religion and politics traditionally are, is self-correcting. It is open and needs as many eyes on the problems in order to figure out what’s really going on. It needs skeptics who doubt conclusions and ask “why?” even when it’s unpopular or unsafe to do so, like in the case of Galileo. And finally, it needs wonder: a sense of curiosity and amazement about the universe we live in. Otherwise, why do science at all? If it’s just as boring and dull as your standard day-job, then why do it? The legends of science were both skeptics and dreamers, and knew how to make both of these work for them and the world.

Although this book is more than ten years old, it is surprisingly relevant to today. It put together the basis of my previous argument about the stimulus package and that experiments can work outside of scientific mediums. And all the problems Sagan said were huge ten years ago are still looming over us (climate change, anyone?).

Sagan says that the primary problem in America as of his time (and I’d argue now as well) is that people don’t really “get” science, skepticism, wonder, or why they’re important (and all related). In fact, even people I normally look up to, like Deepak Chopra, shows he has no idea what skepticism is about:

It never occurs to skeptics that a sense of wonder is paramount, even for scientists. Especially for scientists. Einstein insisted, in fact, that no great discovery can be made without a sense of awe before the mysteries of the universe. Skeptics know in advance — or think they know — what right thought is. Right thought is materialistic, statistical, data-driven, and always, always, conformist. Wrong thought is imaginative, provisional, often fantastic, and no respecter of fixed beliefs.

Carl Sagan is pretty high up there as far as skeptics goes, and in this book (and the famous Cosmos television program) he makes it clear that science, skepticism, and wonder, all must go hand in hand for any of them to really work in the real world. That’s why it’s so baffling that every sentence in that Chopra quote is so wrong (except for the second about Einstein). Let’s break it down:

  1. It never occurs to skeptics that a sense of wonder is paramount, even for scientists. – We’ve already proven that wrong, and that in fact it certainly does occur to skeptics like yours truly and Sagan that wonder is needed.
  2. Especially for scientists. – See previous sentence.
  3. Einstein insisted, in fact, that no great discovery can be made without a sense of awe before the mysteries of the universe. - Agreed, as stated before.
  4. Skeptics know in advance — or think they know — what right thought is. – Completely wrong. The idea of skepticism itself is that we don’t know what the right idea is and need to find out what it is, even when it relates to thought itself.
  5. Right thought is materialistic, statistical, data-driven, and always, always, conformist. – No. Right thought may be statistical and data-driven, but it is not always conformist. Copernicus, Kepler, and Einstein all had wacky ideas (sun-centered solar system, planets with elliptical orbits, and relativity) that are not common sense, but are true, and that’s what counts.
  6. Wrong thought is imaginative, provisional, often fantastic, and no respecter of fixed beliefs. - Wrong once again. The three geniuses I just mentioned were all extremely imaginative, fantastic, and did not care about fixed beliefs whatsoever (although it may have taken them a while to come to grips with what they found).

So I feel like I’m in a weird situation here. I like Deepak Chopra and really got into the Tao of Physics so long ago, but find his current statements to be ridiculous (Postscript 12/24/09: Chopra didn’t write the Tao of Physics, Capra did. Sorry about that!). I will have to give him the benefit of the doubt and hope by “skeptics” he means “skeptics he talks to today” and not “skeptics across the world”.

At the same time, I’m a person who was not really into politics or skepticism a few years ago but now is starting to come along in these fields. And it very much makes me feel like a character in an H.P. Lovecraft story, who starts off ignorant of the world around him but through some sheer accident, sees the world around him for the first time and is driven completely mad (see The Call of Cthulu or The Haunter of the Dark). Certainly I am not saying I am a genius of any kind, but I think it’s that the sense of wonder I had as a child is coming back to me. To quote Sagan again:

Every now and then, I’m lucky enough to teach a kindergarten or first-grade class. Many of these children are natural-born scientists – although heavy on the wonder side and light on skepticism. They’re curious, intellectually vigorous. Provocative and insightful questions bubble out of them. They exhibit enormous enthusiasm. I’m asked follow-up questions. They’ve never heard of the notion of a “dumb question.”

But when I talk to high school seniors, I find something different. They memorize “facts.” By and large, though, the joy of discovery, the life behind those facts, has gone out of them. They’ve lost much of the wonder, and gained very little skepticism. They’re worried about asking “dumb” questions; they’re willing to accept inadequate answers; they don’t pose follow-up questions; the room is awash with sidelong glances to judge, second-by-second, the approval of their peers.

Sagan takes care of answering why this happens as well as selling the necessity of science, skepticism, and wonder, and how they’re all tied together, so I won’t bother answering the obvious “what happened to kill all the wonder” question that arises. And while greats like Sagan and Carlin (who I’ve mentioned numerous times already) will shatter your world and leave you dazed, confused, and mad like a character out of a Lovecraft story, I think it comes down to whether you believe that “ignorance is bliss.” If you do, go about your day as usual. If not, do something about it. Read Carl Sagan or watch him on Cosmos. Listen to Dan Carlin and learn that science, skepticism, and wonder do work in the political world. But just like a character in a Lovecraft story, once you see into the abyss and learn the truth, you can never unlearn it.