À propos the previous post, wanted to share a 2019 article by Gregory Travis about the two Boeing 737 Max disasters that shook the world in late 2018 and early 2019. It’s titled How the Boeing 737 Max Disaster Looks to a Software Developer, and I found it fascinating, well-written, and filled with great details.

The Max variant of the 737 was redesigned to allow larger, more efficient engines, but this made it less predictable, which the company decided to paper over in software:

The airframe, the hardware, should get it right the first time and not need a lot of added bells and whistles to fly predictably. This has been an aviation canon from the day the Wright brothers first flew at Kitty Hawk.

Apparently the 737 Max pitched up a bit too much for comfort on power application as well as at already-high angles of attack. It violated that most ancient of aviation canons and probably violated the certification criteria of the U.S. Federal Aviation Administration. But instead of going back to the drawing board and getting the airframe hardware right (more on that below), Boeing relied on something called the “Maneuvering Characteristics Augmentation System,” or MCAS.

Boeing’s solution to its hardware problem was software.

This feels like an interesting extension to the fly-by-wire story of Air France Flight 447, where Boeing seemed to be on the other side of the fence to the Airbus. Perhaps no more:

[…] It seems that with the 737 Max, Boeing has changed philosophies about human/​machine interaction as quietly as they’ve changed their aircraft operating manuals.

Travis argues that people building the software did not fully understand the shift, and particularly didn’t really understand how pilots fly the airplane during more challenging moments:

Those lines of code were no doubt created by people at the direction of managers. Neither such coders nor their managers are as in touch with the particular culture and mores of the aviation world as much as the people who are down on the factory floor, riveting wings on, designing control yokes, and fitting landing gears. Those people have decades of institutional memory about what has worked in the past and what has not worked. Software people do not.

The real reason? Money.

MCAS is certainly much less expensive than extensively modifying the airframe to accommodate the larger engines. […] What’s worse, those changes could be extensive enough to require not only that the FAA recertify the 737 but that Boeing build an entirely new aircraft. Now we’re talking real money, both for the manufacturer as well as the manufacturer’s customers.

That’s because the major selling point of the 737 Max is that it is just a 737, and any pilot who has flown other 737s can fly a 737 Max without expensive training, without recertification, without another type of rating. Airlines—Southwest is a prominent example—tend to go for one “standard” airplane. They want to have one airplane that all their pilots can fly because that makes both pilots and airplanes fungible, maximizing flexibility and minimizing costs.

It all comes down to money, and in this case, MCAS was the way for both Boeing and its customers to keep the money flowing in the right direction. The necessity to insist that the 737 Max was no different in flying characteristics, no different in systems, from any other 737 was the key to the 737 Max’s fleet fungibility. That’s probably also the reason why the documentation about the MCAS system was kept on the down-low.

Put in a change with too much visibility, particularly a change to the aircraft’s operating handbook or to pilot training, and someone—probably a pilot—would have piped up and said, “Hey. This doesn’t look like a 737 anymore.” And then the money would flow the wrong way.

There are some harsh words toward the entire software industry toward the end. This feels directionally accurate to me, and resonated even before the arrival of generative AI:

I believe the relative ease—not to mention the lack of tangible cost—of software updates has created a cultural laziness within the software engineering community. Moreover, because more and more of the hardware that we create is monitored and controlled by software, that cultural laziness is now creeping into hardware engineering—like building airliners. Less thought is now given to getting a design correct and simple up front because it’s so easy to fix what you didn’t get right later.

Three nice moments in Buttondown’s (pretty comprehensive) release notes:

• There is a filter and tags to search more easily.
• Blog entries are linked throughout, complete with cover images – it’s a nice way to distinguish small changes from big improvements.

• The fixes are attributed to specific people. (This part doesn’t appear on mobile.) Particularly nice to see support for more than one person collaborating on a fix – I’ve had fun memories working at various companies of pairing up with someone else to debug a particularly tricky problem.

★★★★☆ (as a book)
★★★☆☆ (for the purposes of this blog)

There are as many books about Steve Jobs as there were Quadra models, but they focus mostly on two phases:

• 1955–1985 – Steve Jobs befriending Wozniak, the early days of Apple, Lisa, and the Mac
• 1997–2011 – Steve Jobs’s “second act” at Apple, and the creation of the iMac, iPod, iPhone, and so on

Steve Jobs in Exile by Geoffrey Cain is a just-released, rare volume that focuses on the “in-between years” – starting with Steve Jobs founding NeXT and Pixar after his Apple ouster, and ending with him coming back to Apple under the absolutely strangest of circumstances. It’s a doubly interesting phase, both because we see Jobs maturing as a leader and actually learning from his many mistakes, and because the early technical NeXT decisions eventually became underpinnings for modern macOS and iOS.

I do not see this as a book of new immense insight, technical depth, or design details, but that doesn’t mean it doesn’t go beyond surface level. What I appreciated most was Cain not shying away from pointing at some of Steve Jobs’s mistakes: hiring wrong people he happened to like, almost driving the company to the ground through obstinance, inability to focus on things he considered uninteresting, and a profound dose of duplicity coming into the NeXT/​Apple merger.

Other things that stood out: focus on people around Jobs, spotlight on Jobs’s disappointing moral flexibility around working with government (or befriending Larry Ellison, for that matter), and a really fun pizza ordering story that serves as a prelude to the Starbucks call during the iPhone 2007 keynote.

Some learnings:

• Craft and taste alone are not enough; you can spend your talents and energy on things that don’t “matter” given some definition of the word. That could be okay if that’s a choice you make – “impact” is ill-defined and often overrated, anyway – but you need to approach it clear-eyed, which Jobs didn’t initially know how to do.
• Confidence, like everything, needs to be practiced, and focused, and influenced back by feedback and reactions. (Witness the negotiating acumen of a certain Jean-Louis Gassée!)
• It’s really hard to create a culture of hard and honest and deep conversations that’s also not a culture of abuse and toxicity.

The one thing I didn’t like about the book was that the few photos inside are only perfunctory; there’s a lot of chatter about a beautiful, symbolic NeXT lobby staircase, top-of-the-landline phones, and expensive chairs, but we never get to see them. Many of the photos are by Doug Menuez – which you can also see online – but the problem is that those photos are generally not that interesting.

That aside, it’s still a breezy and entertaining read that filled in some gaps and provoked new thoughts.

• The UI for filing bugs is inscrutable and has too many hoops to jump through.
• No one does anything unless every field has been filed meticulously and there is a clear repro.
• The designer is ridiculed if the thing isn’t actually a bug, is a duplicate, or if it was filed in the wrong place.
• Front-end bugs are automatically “minor” or “nice to have”s without listening (as there is no loss of functionality, and no data loss).
• The designer is always responsible for stating how it should work, without being able to say “I am not sure why, but this started feeling off and it’s in an important place. Can we investigate?”
• “This is as designed” is an automatic conversation ender.
• The tiniest of external reports, social posts, or blog posts, immediately are prioritized higher than in-house experience.
• Once every few years, a designer gets 20+ demotivating automated emails saying 20+ bugs they filed over the years have been closed automatically during a purge, without any word of explanation.
• Simple human touches like “thanks for filing!” or “nice catch!” never enter the picture.
• Engineers never file design bugs themselves.

If you’re an engineer, I can sense you might be getting frustrated, as most bullet points I listed look like extra work. I agree with you. It is. This post is as much about process, as it is about culture and the incentives it establishes. The best places I’ve worked were filled with shared trust and treated bugs as a joined responsibility of everyone, rather than a black-and-white division into “filers“ and “fixers,” with the ultimate end goal always being user’s experience – nothing else.

I also understand this dives right into an age-old tension between manufacture and craft. Bug-fixing processes have to be well-oiled bureaucracies with very specific rules so that they don’t turn into a pile of vibes and Brownian motions. But design (and, by extension, a lot of front-end) doesn’t work like that. Design needs room for taste, for careful exceptions, for escalation of immesurable things, and for a certain flexibility in even the basic definitions.

If it’s a tiny, but embarrassing bug, or a flow killer, or a thing that bothers your most valuable group of users, or something appearing in a well-trafficked place – it is no longer tiny. If it’s working as intended, but it feels buggy to the user – it ought to be a bug. If it’s a long-standing bug, it should be considered as cumulative damage already done, not “oh, this has been like this for a long time, no one cares.” If there’s a shaky repro, but the bug feels important, you need to work from principles or analyze the code. If it’s something no one mentioned externally (ergo: why fix it?), consider a lot of bugs rankle but never get reported, particularly if your company doesn’t project an external presence of caring about feedback and acting upon it. cough cough Apple cough cough cough cough cough dies coughing

Of course, designers have responsibilities in the process also, among them mutual respect and understanding of engineering, clarity of communication (particularly about things that are hard to reason about mathematically), seeing patterns that could be grouped into bigger bug bundles to make fixing more efficient, (occasionally!) helping figure out a fix if the obvious fix isn’t available, and shared understanding with their team about what actually matters. There is always a thousand details that could be better, but for every thousand only a hundred might actually be worthwhile. Flooding the bug process with irrelevant minutiae that won’t realistically ever be fixed is not very helpful.

This is the only way I know of to capture the full spectrum of bugs that ruin software – from front-end to back-end, from visual/​interactive quality to works-or-not functionality, from what can be measured to what never will be. And this is not just about designers, of course. It’s not even about any non-engineering function. Design serves everyone; if your bug-filing UI or your process or your definitions are not well-designed or -balanced, I strongly believe you’re also hurting engineers on your team. And you’re definitely hurting your users.

Complementing my previous post, a lot of great thoughts in this post about invisible work from Hardik Pandya:

When the project succeeded, her work had dissolved into the project’s infrastructure. The doc was just “the doc.” The tracker was just “the tracker.” The alignment was just how things were. People forgot it had ever been otherwise. That’s the thing about good coordination. I’ve realized that when it works, it disappears. You can’t see it precisely because it worked.

Even though Pandya didn’t call that out, it’s worth highlighting that his “founder friend” example wasn’t a woman by pure chance; often the invisible work becomes the second shift of women in the workplace. And then:

The problem is that recognition follows narrative. When a project succeeds, credit flows to the people whose contributions are easy to describe. The person who presented to the board. The person whose name is on the launch email. The person who shipped the final feature. These contributions are real, I’m not diminishing them. But they’re not more real than the work that made them possible. They’re just easier to point at. Easier to put in a slide. And I think that’s where the unfairness starts, slowly, without people really noticing.

However, I disagreed with these parts:

There’s no framework that fixes this. You can’t design a rubric that captures “held the project together.”

Wait, why not? This is a similar challenge to quantifying design contributions (some of which might not clearly map to KPIs or sometimes even OKRs). You can’t measure being in the flow, true user satisfaction and frustration, or world-class-adjacency of taste. But it doesn’t mean you cannot design a system or a rubric that recognizes and talks about them.

From Jeff Veen:

It reinforces my belief that teams need a culture that values attention to detail when building products. Tiny annoyances so often get neglected as we rush to ship, but the consequences accumulate, souring the whole brand. It’s not a long journey from “Ugh, these AirTags…” to “Apple has lost their way…”

But in my experience, those rough edges seldom go unnoticed by someone, somewhere, who was unable to stop the momentum of a product release for such an “insignificant” flaw. Or, even more consequentially, they did not feel it was safe to do so.

I want to quote so much of this essay, so I’m going to do just that.

I’ve always felt that culture is made of the accumulation of small acts of gracious leadership: acknowledging moments of bravery during a retro, teasing out a reticent comment during a product review, and on and on. It can come from other places too, but it is most effective when it comes from the top.

If you’re leading a team remember: Never criminalize pride in craft.