Have you head of the Swiss Cheese model? You see it sometimes in descriptions of how complex systems fail. The visual usually goes like this:

The whole idea is: even if you have multiple layers of safety – like many slices of cheese – there are always holes in each slice. Typically, a hole in any slice is covered by a non-hole in the previous one or the next one. (For example, a car might not allow you to grab your keys if you have not shifted to park – or, if you start driving with a handbrake on by accident, the car might yell at you.) But occasionally, the holes just happen to line up, and a larger disaster strikes.

The model is used in analyses of past accidents, and prevention of future ones. It has proponents and detractors. It’s hard to talk about its applications because the most common examples are horrific. In my book, I wrote about Therac-25, and that was a really unpleasant chapter to research and to write. Other go-to case studies are equally bleak: Chernobyl, Challenger, the Tenerife airport disaster, the Deepwater Horizon explosion.

But I wanted to share it because in my head it applies to UI design also, and sometimes helps me think of how small details add up to a larger whole.

In this first part, let’s start with a more traditional example, although a non-drastic one. Here’s a story of Knight Capital Group, a financial services and trading firm. I’m going to hand off the summary of the accident to Henrico Dolfing:

On the morning of August 1, 2012, Knight Capital Group opened its systems for what should have been a routine trading day, yet within minutes the firm began sending a flood of unintended orders into the U.S. equity market, buying high and selling low across dozens of stocks in a pattern that made no economic sense and could not be stopped through normal controls. What initially appeared as unusual market activity quickly escalated into a systemic failure inside one of the largest market makers in the United States, with algorithms behaving in ways that neither traders nor engineers could fully understand in real time. […]

By the time the issue was identified and the system shut down roughly 45 minutes later, Knight had generated more than 4 million executions across 154 stocks, covering approximately 397 million shares, and accumulated positions worth billions of dollars, resulting in losses of more than $460 million […]. The scale of the incident was not only financial but structural, as a single deployment failure had propagated through a system responsible for a meaningful share of U.S. equity trading. […]

Here’s what happened.

Long ago, the firm built a pretty boring function called Power Peg to automate some transactions. The function used a standard shared limiter that made it stop executing when all the required transactions were fulfilled. After some years in use, the function was deprecated in 2003 and stopped being used then, but crucially, the code was never actually removed.

Some time in between 2003 and 2012, the limiter functionality was upgraded, and the old code stopped being compatible with it. All code in production was rewritten to use the new limiter, but the Power Peg feature wasn’t, as it was already deprecated and not in use.

In 2012, the firm started writing a new program for automated transactions. Its creators decided to reuse the same software flag that previously activated Power Peg. The existence of the old code was known at the time, and the idea was that the code would be overwritten by the new program, so the reused flag would only trigger new code.

In July 2012, the new code was finished and the firm started to install it on all the servers, overwriting Power Peg. The firm intended to deploy the new code to all eight servers, but a mistake resulted in it only arriving on seven servers.

No one caught that mistake.

At this point, you can piece it all together. At this point, I imagine many of you have been wincing more and more with each passing paragraph.

On August 1, the flag for the new functionality was turned on. Everything was fine on seven servers, but on the eighth one, the flag reawakened dormant code that immediately started executing. As the old code was not compatible with the new limiter, it was never limited, cascading into millions of transactions in less than an hour, and a lot of collateral damage; the subsequent market reaction to the news of the firm losing over $400 million and caused its own stock to tank.

Oh yeah, I didn’t mention this yet – the failure killed the firm. (Well, it resulted in a merger, but that seemed to be a way to save face after Knight Capital Group almost went under.)

How does the Swiss Cheese apply to this? You can see it as five holes in five different slices of cheese:

• it was a mistake to not actually remove the old code
• it was a mistake to reuse the same flag
• it was a mistake to not deploy to all 8 servers
• it was a mistake to not have a procedure for someone to double check the deployment
• it was a mistake to not have a way of auto-detecting (and perhaps auto-stopping) the runaway processes when the regular limiter failed

What’s important to understand about this model is that either of these in isolation would objectively be a small mistake, and caught by the other slides. As a matter of fact, any four of these happening would still not add up to a catastrophe.

But in this case, the five mistakes lined up perfectly.

Many analyses of such accidents blame a single event in the chain – in this case often the sysadmin that didn’t deploy the code to eight servers properly – but this is primarily because we like stories of individual agency, and are not well equipped to understand stories of systems. (Even Star Trek added a Borg Queen, after all.)

That’s the accident eventually dubbed “a Knightmare.” More examples of systems closer to our hearts in following parts.