You should NOT always be AB testing.

You should not be AB testing all the time. In a world of experimentation and agile, this might seem counterintuitive, but you should not be AB testing all the time.

If you’ve ever run an AB test, you know how satisfying it is to design a test and let it run for a bit while eagerly tracking the results. Sharing the final results that might have moved the metric is the cherry on the cake.

However, AB tests are expensive to run. AB tests take time to create and may block other development work till the test is running.
Therefore, running any AB test has to be a careful consideration.

Let’s look at when NOT to run an AB test:

1️⃣ When you don’t have a large enough sample size
• For AB testing results to have some credibility, the results of a test should be statistically significant.
• A large enough sample size of website traffic, number of inbound trials, or a large base of customers is essential to derive statistically significant results.
• A large sample size eliminates the chance of a sampling error and is a closer representation of the average user preference.

2️⃣ When you don’t have enough time to run the AB test.
• Similar to above, to get statistically significant results, you need to allow the test to run for a certain period to collect a large enough sample size and also allow some burn-in period for those samples to mature.
• Significant time is required to analyze the outcome of the test and share the results.
• Allocating such time to a test might not be the best use of resources, especially if you are an early-stage company.
• Some tests and their expected results don’t warrant the time they take to unearth.

3️⃣ When you don’t have a clearly defined hypothesis
• You should not run an AB test without a clearly defined hypothesis. • An AB test is run to validate a hypothesis or something you believe to be true.
• Without a clear hypothesis, you might be in a situation where you don’t know how to analyze the test results.

4️⃣ EVERYTHING doesn’t have to be an AB test.
• Especially when there is minimal risk in taking action right away. You don’t need to run an AB test just because you can. If there is minimal risk to making a specific change, do it.
• Things like microcopy changes or improvements with minimal risk of losing a customer can be actioned without an AB test.
• MOST IMPORTANTLY If you are an early-stage company, chances are you have very limited resources required to build an AB testing infrastructure. In such cases, you don’t have to run AB tests, but it’s best to work on the highest conviction ideas and changes and deploy them across the board.