Convert supports two distinct statistical approaches for your reports: Frequentist and Bayesian.
Important Update: As of October 13th, 2023, all new experiences utilize t-tests instead of z-tests. Experiences initiated prior to this date employed z-tests for statistical significance.
For our updated Frequentist stats, we utilize t-tests. You have the flexibility to configure the following parameters:
The confidence level. Typically, a 95% confidence level is used for most experiments. However, if your experiment is of critical importance, selecting a 99% confidence level would be prudent.
The test type, either a one-tailed or two-tailed t-test. We generally recommend a one-tailed test for standard experiments, as it tends to reach significance more rapidly. For high-stakes, mission-critical experiments, opt for a two-tailed test. It may take longer to reach significance but provides a more conservative result.
Concerning multiple comparison correction techniques, you can choose from Bonferroni, Sidak, or None. From a statistical robustness perspective, Sidak stands out as the optimal choice, especially for mission-critical experiments. It maintains the family-wise error rate without adversely affecting the test's power. Nevertheless, the choice is yours.
Additionally, a Sensible Defaults menu is available. It allows you to quickly set "preferred" parameter values based on the criticality of your test—be it "standard" or "mission-critical."
Within the Bayesian framework, you can set a decision threshold. This is the minimum "chance to win" probability you'd find acceptable for making decisions. The default is set at 95%, but this can be adjusted according to your risk tolerance. For those who seek maximal certainty, a 99% threshold is recommended.
Regarding priors, we employ uninformative priors, meaning that, a priori, each variant has an equal likelihood of either outperforming or underperforming the others. As data accumulate during the test, these priors are updated, resulting in posterior distributions that inform your decisions.