Marques Brownlee published a video commenting on YouTube’s just announced A:B testing feature. In short, for some creators, it will be possible to publish a slightly different version of the video on launch to part of their audience, if that version is deemed “close enough” to the original.

Brownlee digs into how confusing this feature might be in actual use for viewers, never sure whether they’re watching, commenting, or rewatching the same video as others see, or even as themselves earlier saw. Toward the end of the video, Brownlee shifts his focus toward other YouTubers, with a monologue that I wanted to quote here because it captures something important:

You know, this is being framed as a tool to help you improve your videos. But I think for 99.9% of creators – well, to be literal, it’s not actually helping you make better videos; it’s helping you find which version of your video has the best retention, which doesn’t necessarily mean it’s a better video – hot take.

And there are so many examples of creative decisions that I have made, and that so many others have made in the video creation process, that are definitely not the best retention-optimized decisions, but they’re fun, and creative, and interesting, and new, and entertaining, and characterful.

So there’s always going to be the top 0.1% of channels that are all about mass appeal, and that’s fine. But last time I checked, a lot of what makes YouTube fun is the stuff that is more niche down, that is not mass appeal by definition. It’s the cool small hobby you found, or it’s the fun, unique thing that you just discover that only a few people are talking about that – that stuff! That’s what makes it fun.

I think if a lot of those channels and videos suddenly became like superoptimized, retention-maxing, and everything, those would get a lot less fun to watch. And so imagine if all the feedback anyone ever listened to was just to have the best retention – as you optimize over and over and over again, you start to trim out all of the other creative stuff that happens to not be the most efficient way to tell a story.

So I guess what I’m trying to say is... the skill of being a video creator on YouTube is not always necessarily about the maxed out, most efficient, optimized way to do everything, but it’s to find creative ways to share something, or teach something, or explore or explain something, or review something. So I just think making and finishing and publishing a bunch of videos is a better use of a creator’s time than making one thing and then, you know, a whole bunch of different drafts of titles, and thumbnails, and different cuts of the video to try to optimize for, like, the last few percent. […]

That’s all. I think just stand by your creative choices. You made this video, this is the video you chose to make. And now you get to share it, and you put it out, and you learn from it. […] That’s the creative process. Have conviction in what you want to share with the world.

There is another half-pulled-on thread in his video; Brownlee doesn’t say it directly, but dances around the idea that the very fact this feature was launched is reflecting on YouTube’s own insecurity as a product. Jim Nielsen follows up on that on his blog:

I find it interesting how Brownlee shares his opinion that YouTube spends way too much time chasing their competitors and not enough time being YouTube.

Which, when you think about it, is exactly the kind of place a feature like this would stem from: an insecurity in your own creative choices. How YouTube approaches its own insecurities is now trickling down as a feature to its users.

“Let’s let people make lots of variations on things, try all of them, and see what performs best” is exactly the kind of thinking you get in a platform that doesn’t know what it wants to be. So it’s left spending its time 1) doing what others are doing, and 2) following the fickle whims of whatever it can measure.

Brownlee talks a lot about YouTube videos being “fun, and creative, and interesting, and new, and entertaining, and characterful.” Those might not be the most important adjectives to apply to all of software, of course, but there are some interesting throughlines here. I have also personally not seen a lot of thoughtful A:B testing in my career. I think those are an easy way to lose yourself, to have your app become a bundle of metrics that no one understands – and once you start opening the door to being driven by poorly-understood experiments, the desire to fall back on reusing patterns from other successful apps, without any reflection, only growing stronger.

That’s how you end up with stuff Nielsen and perhaps even Brownlee argue YouTube itself has become: a set of Brownian reactions, or a warmed-over software cosmic latte, if you will. From my perspective, this often also leads to systems of interactions or features that don’t come together well to the user, as you end up borrowing from other places without the necessary hard work of translating them to feel at home and consistent with your system and its quirks and conventions. From what I’ve seen in my career, A:B tests do not promote a lot of design execution rigor. The tests are meant to be light by definition, since a lot of them are expected to fail. Before the test, there is limited incentive to invest in quality, but neither there is one after the test – if A or B successfully raises a metric, you don’t want to mess with it then, since what if you change the very thing that made it work?

On top of all that, we can add what Brownlee started with in his video: overeager A:B tests feel like software’s version of gaslighting. How can you trust an app if it feels like it’s changing without rhyme and reason seemingly on a weekly basic, those changes are not announced or documented anywhere (small A:B tests never are), and other people don’t see the changes you see?