So damn true, it's so difficult to get statistically significant A/B test results, and the popular tools actively lead you astray. It's rare enough for a startup to have the traffic to meaningfully get results on their homepage before the heat death of the universe, let alone random landing or in-app pages.
I'd recommend anyone reading this who does or wants to do A/B tests read: https://www.evanmiller.org/how-not-to-run-an-ab-test.html It was one of the trickiest lessons I had to learn.
Once you realize you need to set your sample size in advance, you actually have to do the math to figure out the traffic you'll need. That's when reality hits you.
Yup. The best thing for running an A/B test you can do, by far, is setting up the rules in advance. Our protocol was something like this:
This test is going to have 2 variations, looking for a 5% increase of the conversion rate that is currently X%, and to do so the test will run for Y iterations (based on the company standards for significance and power).
If the test shows a >= 5% increase, it wins and we use it. Yay! If not, we assume it is no different than the baseline, record the results and discard it. You are welcome to peek at the results all you want, but no tests are stopped early and no decisions are made until it reaches the set amount of iterations. This isn't the only good or valid A/B testing protocol, but it does force people to consider the costs of their tests in advance (in terms of time and iterations required), which I think had a positive effect on the type of tests people ran.
Just having the calculator discourages people from running tests with tons of variations looking for tiny increases (like the famous try different colors of your submit button), because just looking at the iterations required by the calculator it becomes obvious to everybody that those types of tests just cannot show any meaningful results for your average website.
