“Everything in the digital universe was going to have to get patched.”
Computers spend an enormous amount of time pretending numbers can be other things, and internet infrastructure does the same, constantly working hard to translate impenetrable IP addresses such as 104.21.49.121 into friendly strings like unsung.aresluna.org.
The details of the infrastructure are above my pay grade, but in principle it works like this: You ask your browser to visit a URL on a domain it hasn’t seen before, so it doesn’t know its IP address yet. Your computer asks a DNS server for that IP address, and the server sends it over. Now, your browser knows where to go, and fires the request directly to the website using its just-learned address.
Since you don’t want to bother the DNS server with too many questions, your operating system remembers the answer in its cache for a while, so next time you visit the same server, it can retrieve the IP address from memory. The DNS server in its response even handily provides a few parent addresses to cache too (e.g. aresluna.org), to further save on future network traffic.
Now, I glossed over one part we gloss over every day just to say sane. “The server responds” is its own complex challenge. The Internet’s series of tubes are not those rubber hoses where one opening ends up next to someone’s mouth, and another next to the other person’s ear. No, on the internet, things are divvied up in packets that all travel together side by side, squeezed like packages on an Amazon truck, and they are only dispatched to the right precise destination at the end of their journey.
Even though the DNS logic was created during an early era of the internet where computers trusted each other more, the creators recognized the risk of someone else impersonating the DNS server, and sending their own invalid response to your “translate this domain into the IP address, pls” query. So, a simple authentication system was built. When asking the server, you’d include a number only you knew – in effect, a simple password. Then, when the DNS responded, it included the same number. If you saw your number and the domain you asked about, you could have a pretty high confidence that who was responding was the server you originally asked. A wrong number? Means a mangled transmission or an impersonator, so just throw the whole thing away.
This seems like a primitive system, but this sort of security by unguessability happens all the time. When you start a new Medium post, you will see this in its URL bar:
The e091fb25af43 part is just a number. That number is the entire security. There’s nothing preventing you from guessing that number to get to my draft, or methodically start testing each number one by one – except the space extended by 12 hexadecimal digits, at over 281 trillion numbers, is basically astronomical, and you are not. If you checked one URL per second, it would take you 9 million years. If you checked a hundred a second, it would be 60 millennia. 10,000 per second? Still over 600 years of work. This is effectively a monkeys with typewriters scenario, let alone that at some point during those 600 years Medium would also recognize what you’re trying to do, and their systems would stop responding to your efforts and perhaps send an appropriate three-letter agency your way.
This system works because the number space is incredibly vast. But for the DNS query we talked about it was only 65,536 numbers.
Enter Dan Kaminsky, a security researcher with a reputation for tackling the unfun, pedestrian issues no one else cared about. In 2008, Kaminsky found himself poking at the fringes of DNS, trying to find problems and vulnerabilities.
He was specifically interested in helper DNS servers. See, the caching I mentioned doesn’t just happen on your computer. DNS servers are arranged in a small hierarchy and also have a bunch of helper servers (called “public resolvers”), popular for their speed. If a helper server doesn’t know something, it asks a server closer to the truth, starting with one of the supreme court of few “root servers” at the core of the whole system, and then down the chain, sometimes a few more times. Then, after receiving the answer, it sends it over to whomever asked – but it then caches the answer for a while, too, to prevent all this effort in case the same computer or another one asks about it again, too.
This means that if you could convince that helper server to cache a wrong answer, you could trick a potentially significant swath of the internet to go to a wrong IP address – sending, say, all amazon.com traffic your way, and then stealing people’s passwords and credit card numbers.
But you’d still have to know the one right magic number out of 65,536.
This is where Kaminsky tried something different. 65K numbers is a lot for a person, but not for a computer. What if you could, indeed, monkey-and-typewriter this whole situation? What if you did it by brute force?
Say, you ask a DNS server for an uncommon version of a common URL (e.g. test123.amazon.com) so you know it won’t have it cached. No cache means that server has to ask a server closer to the core, which takes a bit of time. But before that happens, you fake an answer from that parent server to the server you just asked.
No, scratch that, you fake 65,536 answers.
Now, you know the domain you asked about, so that part will match for sure. You don’t know what secret number the server communicated to its parent server, but you know that out of all the answers you generate, one will have to match. Now, you just need some luck of timing. If your answer will arrive before the official answer, it will be treated as the official one and cached – and the official one coming after yours will be rejected.
(Here’s a pretty horrible real-life example: Imagine someone wanting to hijack you as you land at the airport. They know where you land, but they don’t know your name – so instead of trying to guess, they just deploy a few dozen limo drivers, each one holding a card with a different popular first name. If they time it well, you will see the fake limo driver before your limo driver – and that’s the driver you will follow to your doom.)
And… the real server wouldn’t just respond with an IP address test123.amazon.com. It would also, as was customary, thoughtfully include an IP address for the entire amazon.com domain while you’re at it, just to help you in the future. If your fake answer included that too, you could theoretically insert yourself in front of Amazon’s traffic.
Kaminsky tried it… and saw it succeed. The consequences were huge. He later recalled:
Once I saw it work, my stomach dropped. I thought, “What the heck am I going to do about this? This affects everything.”
An article by Erica Naone written around that time adds:
Kaminsky’s technique could be used to direct Web surfers to any Web page an attacker chose. The most obvious use is to send people to phishing sites (websites designed to trick people into entering banking passwords and other personal information, allowing an attacker to steal their identities) or other fake versions of Web pages. But the danger is even worse: protocols such as those used to deliver e-mail or for secure communications over the Internet ultimately rely on DNS. A creative attacker could use Kaminsky’s technique to intercept sensitive e-mail, or to create forged versions of the certificates that ensure secure transactions between users and banking websites. “Every day I find another domino,” Kaminsky says. “Another thing falls over if DNS is bad… I mean, literally, you look around and see anything that’s using a network–anything that’s using a network–and it’s probably using DNS.”
And here’s Wired with its typical, exaggerated take:
The problem had been coded into Internet architecture in 1983. It was 2008. Somebody must have fixed it by now. He typed a quick series of commands and pressed enter. When he tried to access the Fortune 500 company’s Web site, he was redirected to an address he himself had specified.
“Oh shit,” he mumbled. “I just broke the Internet.”
Kaminsky asked his friend and DNS expert Paul Vixie, who grasped the severity of the situation:
I realized we were looking down the gun barrel of history. It meant everything in the digital universe was going to have to get patched.
Together with Vixie, they alerted Department of Homeland Security, and then execs at networking companies. The plan was hatched: he’d work with biggest DNS vendors to quietly patch the system, and only a month after that, once everything was already corked up, he’d release the details of the finding to the public.
The patch was deployed on July 8, 2008. It was as cheap as possible, basically vastly increasing the number space from 65,536 to billions (Vixie compared it to dogshit, but there wasn’t time for anything else), and Kaminsky planned to announce the details on August 6.
But there was an unexpected last-minute kink in the plan, which was… someone’s curiosity. Apparently, one security expert saw the patch, and from pondering that patch figured out the problem, and exposed it prematurely on July 21. Luckily, by then things were already relatively patched up, and no one could nefariously use the original flaw.
This cache poisoning problem was a nightmare some compared to Y2K, and likewise avoided – but this one without any of the limelight.
Kaminsky was inducted to an Internet Hall of Fame for detecting the problem and eventual successful effort in containing it. Unfortunately, this happened only a few months after his untimely death at 42, in early 2021.