On the Convert report, we support two kinds of stats, Frequentist and Bayesian.
Frequentist
For Frequentist, based on a Z significance test, you have the option to configure the following:
The confidence value. For normal tests, 95% confidence is the standard, but for mission-critical experiments best to choose 99% confidence.
The power level is mainly used when you choose to use power calculations. For standard experiments, it should be 80%, but for mission-critical ones, you can choose 99% for maximum certainty that you do not miss small effects.
The test type one-tail or two-tails Z-test. We advise you for standardized tests to use a one-tail setting. One tail reaches significance sooner and is usually good enough. But for mission-critical experiments, two-tails is best, as it takes a tiny bit more time to reach significance.
For the multiple comparison correction methods, we offer none, Bonferroni or Sidak. For all intents and purposes, Sidak is the best for every use case that we know, especially for mission-critical experiments. It adequately controls the family-wise error rate while not influencing the power too much. However, we give you a choice to use it or not.
The Sensible Defaufts. For all the above settings, we provide you with a shortcut to set “preferred” values depending on if your want to run a “standard” test or a very important one, .i.e a “mission-critical” one.
Last but not least, we give you to option to leave your usual AB testing calculator alone by integrating optional post hoc power calculations.
For those, you can choose between the following:
• MDE - Use a Minimum Detectable Effect (MDE) of your choice. In that case, we calculate the number of samples needed to achieve the desired confidence and power levels for that MDE, also called the Minimum Effect of Interest. Knowing this, we can calculate the test progress in percent given the number of currently collected samples!
• Observed Effect - Another method that some may wish to use is to use the observed effect as MDE and calculate the test progress based on that.
• None - For those that do not want to do post hoc power calculations, we have the option to disable that and check only for statistical significance.
Bayesian
For bayesian, we give you the option to set a decision threshold, above which you would be satisfied with the chance to win probability. The default is 95%, but up to you to adjust it to the level of your liking. For maximum certainty, a value of 99% would be ideal.
Comments