The theme: What does it take to build interfaces that truly allow for fast operation – and why that matters.
If you like the interactive details posts here on Unsung, the essay is kind of a concentrated dose of all that. You can technically read it on the phone, but it’s so much better on a computer (or a big tablet). It has ~40 interactive playgrounds, and sounds, and a glossary, and all sorts of fun stuff I’m doing for the first time.
I wanted to share some things I learned over the years, and nod toward mostly anonymous creators of UX inventions I’ve long admired. I also thought it could be interesting to make interfaces appear as machinery – you’ll see what I mean.
There are tons of small bugs around. Occasionally, there is a really big one. In the “oh my god, I can’t believe it” category, here’s a story of the game Yakuza 6.
A few months ahead of launch, in February 2018, Sega released a free playable demo of the game. The demo gave access to the first level, and then blocked the progress with a barrier:
Somewhat unusually, the demo actually contained the entire game in a hefty 35GB+ download. The idea was apparently that after the game release and player’s purchase, they could “unlock” the game with a code and resume as if nothing happened, instead of downloading it separately and likely losing their progress. However, that made this barrier a pretty load-bearing one, as finding a way to circumvent it would mean people could play the entire game for free.
But before speedrunning hackers laid their hands on this challenge, it turned out no circumvention was even necessary – a bug in the game code itself made the barrier simply not appear on American PlayStation consoles.
Sega realized it quickly, pulled the demo, and blocked the installed copies via DRM, but not before some players got access to the complete game and finished more than the expected first chapter.
Alas, there was no public post mortem or an explanation; I’d love to understand what happened on a technical level. Either way, it’s wild trying to imagine the moment people at the company realized what they’ve done.
In iPhone’s accessibility settings you can choose the allowed speed of double- and triple-clicks on its side button (why is it important? we talked about it once), and the interface does something nice – after you make your choice, it shows the expected speed in the same place a sort of a preview:
To be honest with you, I was surprised that I liked it. This feels like it’d be a perfect example of cheapness, especially given the iPhone has this delightful animation that could be reused here:
But, I don’t know. Somehow, this one feels like it’d be too complicated. Maybe cheap is okay if one cannot think of a better “bespoke” interface?
Cheap here also has an added benefit of reusing existing patterns, which might feel nicer in the more utilitarian surroundings of settings.
But my favourite thing that elevated this was that with each visual blink there is also an accompanying haptic buzz. I think this is really clever. A haptic buzz is much “closer” to your fingers than onscreen blinking, and can help you feel the speed rather than just see it.
Unfortunately, the same clever preview is not present here in the otherwise very similar AirPods menu…
…and I also found myself wondering what would it take for it to make its way here as well:
You might have seen Bret Victor’s 55-minute Inventing On Principle talk soon after he gave it in 2012. If not, you should check it out. If you did, you should check it out again and see how it makes you feel today:
It is about interactions but in the service of something grander, which (if I’m doing my job well) you might recognize as Unsung’s core theme.
Victor – a designer, researcher, and computing historian – gave a few other talks in the few years since, and I thought a little guide might be helpful:
Drawing Dynamic Visualizations (35 mins) is specifically about information visualization, chiefly a demo of the “Illustrator, but programmatic” tool showed briefly in the above talk. There’s also a bit more theory.
Similarly, Stop Drawing Dead Fish (53 mins) is a demonstration of a different programmatic tool to make animations.
I love this blend of theory and practice, inspiration and pragmatism, high- and low-level. The tools look surprisingly professional for research projects, but underlying their microinteractions is a deep philosophical stance. It all reminds me a bit of Jef Raskin and Doug Engelbart.
Victor’s last talk of this era is Seeing Spaces (15 mins) from 2015, serving as a sort of introduction of him moving toward computing in physical spaces. As far as I understand, Victor has been spending time on Dynamicland since, which is definitely more physical computing, but also a lot more academic and scrappy, and as such out of range for this blog.
(His website is worth checking out, especially if you’re not in the mood for talks and would like to get to know his work in a different way.)
I thought about this the other day, and I thought it’d be fun to share this internal tool I made over a decade ago to aid with exploring options for Medium’s typographical redesign.
The motivation for building Fontificator came from two observations:
font previews on type foundry sites were generally too limited to get a real sense of how a certain typeface feels, and it was best to see a font in situ,
often an extremely tiny nuance – like adding some letter spacing, or messing with line height – was what separated something that was promising from something that seemed very far from working.
With Fontificator, I was aiming at this Doug Engelbart-esque notion of one hand on the keyboard + one hand on the mouse, and the UI where it was only necessary to point to an element, and the keys under your other hand would start working immediately – no clicking needed:
F and G to change the font,
– and + for font size,
← and → for letter spacing,
↑ and ↓ for line height,
< and > for opacity (for all the above you can hold Shift for bigger moves),
and, there are a few more shortcuts you can see at the top.
This way, we could move really, really fast. To accommodate that, Fontificator always tried to keep the current item under the cursor by counter-adjusting scroll position as needed.
On top of it all, a few more shortcuts:
⇥ and ⇧⇥ move very quickly between different types of stories so you can preview that,
Space compares to the original/current version,
1–9 allow you to switch to different “slots” so you can have various presets ready to compare,
Esc hides the toolbar for maximum immersion,
⇧R resets.
You can also edit any text if you are so inclined, and also drag in any font file from your computer onto a paragraph – then that font becomes part of the F/G stack. (Bernino Sans and Freight Text were the starting fonts before the redesign.) On the left, you can also see a naïve mobile preview – there was also more sophisticated on-smartphone preview, but I removed it from this restored version.
Fontificator was literally made for an audience of 2–3 designers (and perhaps 1–2 stakeholders in read-only mode), and it was surprising to me how quickly one could master this strange tool, have fun with it, and feel the page’s typography becoming much more malleable. We also put up a more “traditional” list of contenders on the wall…
…but it was in Fontificator where we learned the most.
I love internal UIs because they allow you to go very wild and very tactical. If you have one you’d be willing to share (maybe it, too, is on the other side of the statute of limitations?), or one you already wrote about or spotted someone else doing so, please let me know!
The software company Panic has a blog, and has had it since 2009. It has a clean, modern aesthetic that looks something like that:
However, something interesting happens as you start going back in time by clicking Older Posts at the bottom – the historical posts are rendered using their original styling:
This isn’t something that happens for free, as with any redesign every piece of content gets ported to a new framework and style by default. So, I gather this was an intentional thing that also required extra effort both to make it work like this to begin with, and to allow the old style to appear nicely within the different confines and technical realities of the new style (you can compare the above screenshot of Firewatch announcement as retrieved today with its original appearance in the late 2015).
I love this effort. I wonder if more places on the web could use that kind of thinking. As an example: what if your social posts or blog posts from long ago came adorned with the same avatar that you used when posting them, even if you updated it many times since?
From the Animation Obsessive newsletter, a fun and nicely illustrated recounting of the way Jordan Mechner animated his seminal games Karateka and Prince of Persia. It’s rotoscoping, as everyone knows by now, but on a hard difficulty mode:
Mechner’s setup for Karateka was wild. Over his Moviola screen, he taped thin paper, upon which he traced key frames from the Super 8 footage beneath. Then he took his pencil sketches to a VersaWriter — an early drawing tablet — and traced them on that. Frames of movement became pixels on his computer monitor. From there, he cleaned them up with an art program he’d coded.
Everybody who saw the game oohed and ahhed. It was like a great proof of concept, but it wasn’t that much fun to play, and I kind of had the sinking feeling as I realized that I’ve done almost everything I meant to do, but it just doesn’t have that excitement that I was hoping for.
Also there was a ticking clock, which is that the Apple II platform was dying.
[…] So this was the problem: two years into development, I’d used up all the memory to get as far as I’d gotten, but the game was missing that suspense and excitement and sense of conflict that had made Karateka so simple and so much fun. What was I gonna do?
It’s a great example of a creative technical solution, which also informed the game’s storyline – a perfect collaboration between design and engineering.
This is a really nice set of transitions when pinching in and out in Photos in iOS 26.
This is trickier than it seems, because it’s not just a linear zoom (like it would be in Maps or Sketch, for example). It’s a zoom and reflow – from 3 items to 1 item per column – which makes things a lot more complicated.
Here are a few nice details about this transition:
It reacts to your fingers rather than being a rigid transition with a fixed duration.
It always prioritizes the photo you’re pinching in and out, assuming that’s where you look.
It smoothly transitions the aspect ratio (from always square when the items are smaller, to native when items are bigger).
It crossfades the other photos. Cross-fade is the “cheap” answer for transitions, but here it feels appropriate, as it happens in the periphery – actually trying to move the other items linearly between their respective positions would feel unpleasant and distracting.
In contrast to the other transitions, these crossfades are not fully tied to fingers, meaning you cannot stop in the middle of a crossfade.
Nikita Prokopov on his blog published other examples of problematic transitions, and it seems most of them are struggle in the same way, as transitions that cannot simply be linear. The above transition in iOS shows it’s possible to do it well if you care.
And it’s not just about smoothness or nice feelings. Prokopov:
[…] Desynchronization can lead to a lot of confusion. For example, in Photos, when switching between Crop and Adjust mode, picture snaps into place immediately but the crop border is animated.
This creates a false feeling that something subtly changes when you switch between modes. And you know what? I don’t want my UI to give me false feelings. I want it to be a precise instrument, not an animated toy.
The above iOS transition feels very precise to me.
It’s an impressive list that garnered universally positive reactions, but I have one observation:
“Fast” and variants thereof appear on this list 59 times.
“Reliable” and similar words appear 22 times.
It’s true that everything could be faster and a whole many things should. Speed is paramount to great user experience. Speed is also more than just speed; there are nonlinear aspects when latency or delays cross invisible thresholds that can drastically change app usage for the better.
But in my experience, much more often the things that frustrate me about using Apple’s products are not issues of speed, but issues of reliability:
I don’t need faster network connectivity in Finder, but I struggle with computers not appearing, a pizza cursor that occasionally just dies spinning, or randomly being thrown to the root of the networking volume.
I don’t need AirDrop to be faster. I just want it to connect reliably every single time, give me consistent and understandable UI feedback, and stop forgetting I’m not just “everyone” when sending stuff to myself.
I don’t need Messages to be faster. I need them to just, you know, not haphazardly stop syncing across computers on occasion.
It’s not just me. 15 out of 17 bugs listed on the Bugs Apple Loves site are about reliability. None seem to be directly about speed, although more on that in a second. Or, here’s a recent list from Ilya Birman – different issues, but a similar shape.
I’m going to say it: Speed is an easier problem. Not easier in an engineering sense; I’ve seen an engineer try to carve ten milliseconds out a busy computer’s schedule, and in that moment, one must truly imagine Sisyphus happy. But it’s easier as a problem: it often comes with a lot of pre-built telemetry, plus a clear goal of “here’s Xms and X now needs to be smaller.”
Reliability is much harder, more difficult to debug, reproduce, agree on metrics for, even find ownership of – just generally fuzzy around the edges, and less obviously thrilling as a challenge. It needs more champions and structures.
I know a simple marketing slide is not meant to be an accurate representation of Apple’s efforts. I wasn’t at WWDC so perhaps the vibe in the room was different than what this slide represents. Yet, the slide exists and I’m allowed to judge it.
(And yeah, I know in some cases speed and reliability are correlated. After all, if you have a timeout, making something finish faster and do so before the timeout will turn it from unreliable to reliable. But hey, I wasn’t the one choosing the words on the slide.)
I just… I would be a lot more excited if the 3:1 ratio of fast-to-reliable on that slide went the other way.
In game development, there is this strange effect known as “tunneling.” It happens when you do collision detection. Imagine a simple situation where every time you move a cube, you also test whether it touches the wall – and if it does, you make it bounce off of it.
This works great, but if you move and detect the collision less frequently, something weird can happen:
Here, the movement was so coarse that there was no point at which the cube touched the wall, so the collision wasn’t detected, and the cube passed cleanly through… as if it made a tunnel.
The easy answer seems to be “well, run the collision detection more often then,” but… how often? And what if the entire game engine runs off of computer’s frame rate, which you are not in control of it at all? All in all, it’s not a trivial challenge, although various techniques exist to remedy it.
We talked before about another challenge with frame rate dependence. They’re not limited to games; for interfaces that are based on physics, they will rear their ugly head, too. But tunnelling happens in simpler UI situations as well. Here’s an example from Photoshop – I’m holding a button and if I drag slowly, each item will be toggled. But when I start moving fast…
Fortunately, the remedy here is much easier than in the complex world of videogame physics: just remember the last one touched, and toggle everything in between that and the current one.
Pointer input isn’t continuous. The browser reports the pointer’s position as a series of discrete samples, and when you move fast, those samples can land far apart. Flick your wrist and two consecutive pointer events might be a hundred pixels from each other, with three small shapes sitting untouched in the gap between them.
A naive eraser asks “what’s under the pointer right now?” on every pointer event. At slow speeds that works fine. At high speeds it tunnels straight through anything that happens to fall between two samples, which is exactly when people use the eraser most aggressively: big, fast, careless swipes to clear a region.
Here’s Ruiz’s example of the eraser in FigJam, with heavy tunneling:
And here’s one in his drawing tool:
You can click through to learn more and see the algorithms, but either way it’s worth remembering: if it applies to your interactions, think about the “flyover states” and make sure to make things deterministic, regardless of whether the mouse is a tortoise, or a hare.
And, I liked Ruiz’s sentiment at the end:
[…] The decision to test segments instead of points is the difference between an eraser you can trust at speed and one that mysteriously leaves survivors behind. Users will never notice it working, and that’s the point.
Though [the open source update framework] Sparkle serves us very well, it has one notable downside. Update announcements are most likely to appear at the least convenient time: right after you’ve launched the app.
With that in mind, we’re making changes to how update notifications appear throughout our apps. In the future, when the software’s timed automated check detects a newer version, it will no longer pop an obtrusive window like the one seen above.
Instead, a small “Update Available” indicator will be shown in the app’s interface. You can see it right here in Audio Hijack.
I first remember this approach from Chrome in the early 2010s. (If you know it from an earlier application, please reach out!)
The browser still uses it today, but the visual treatment was different early on; the update icon badge started with green, then yellow, eventually ending up red. While this resembles traffic lights the inspiration was, apparently, rotting fruit – you’d be more likely to want to clean up an old, stinky fruit than a fresh new one.
Here are, I believe, the first three visual treatments in Chrome, 2011-2013:
How effective was that treatment, I don’t know. (It definitely felt more thought through than the trash where the skeuomorphism undermined the function itself.) But it is all interesting to me in the larger context of the tensions underlying updates:
It’s in a company’s best interest for every user to be on the latest version, since that saves on support headaches.
A company needs to believe the newest version is the best ever – even if it’s not – similarly as our brains need to believe we are generally right most of the time, just so we can function.
That’s why I always appreciate the improvements that prioritize the user experience over the company’s.
