Bijan Sabet

Some thoughts about A/B testing

At lunch today, Andrew reminded me of a story that I had long forgotten.

When he was at USV, he attended a board mtg for a portfolio company that we have in common.

It was the early days of that company and their initial product hadn’t been out that long.

The topic of A/B testing came up and Andrew reminded me that I came out against A/B testing in that meeting.

The sad but true part of todays conversation is that I don’t recall that meeting or that particular conversation.

But I would say the same thing to any early stage company with their initial product.

When you have thousands or even tens of thousands of active users, the product comes from the founders vision and desire. It doesn’t come from a focus group or some survey. And it didn’t come from an a/b test of some landing page or the home page.

It comes from the heart.

a/b testing in the early days is never going to be as good as pursuing the founders vision for making the product everything that it’s supposed to be. Fix those annoying bugs. Make it faster. Make it easier. Make it right.

Now, that doesn’t mean you shouldn’t listen to your users. Their feedback is vital. We all know, as a perfect example, that may of Twitter’s best features came from their users (e.g @mentions and retweets).

But a/b testing is optimizing around the edges around a terribly small group in the early days. It’s not worth the energy in my opinion.

As a web service grows significantly, a/b testing goes from not helpful to critical. Analytics, a/b testing, linking new features to metric driven objectives become an essential tool in the arsenal.