A/B vs Multivariate testing

The first advantage of multivariate testing is that you can see the effect small changes on your site have. Of course, small changes can also be tested with an A/B test, but that’s suboptimal as you can only test one small change at a time. Generally, A/B tests are used for the big changes and multivariate testing is used for optimizing smaller elements.

The biggest drawback of multivariate testing is that you need an even bigger amount of traffic and especially conversions than you do for an A/B test. I always say you should have at least 100 conversions on each variation. So if your multivariate test has 3 different versions of 3 different elements, you’ll need at least 2700 conversions (3 x 3 x 3 = 27 combinations with each at least 100 conversions).

As I’ve already mentioned above, multivariate tests are used to test smaller elements on a page. You’re testing small variations of the same element, instead of overhauling the complete page as you would in an A/B test. So first and foremost, multivariate testing isn’t supposed to be used as a starting point.

So this obviously leads to a proper testing framework were multivariate tests need to be introduced following A/B testing framework testing is done, rather than a step 1, step 2 approach this must be part of the same analysis exercise. Knowing which tool to use is the carpenters decision which will affect the quality of the end product none the less.

Scroll to Top