Understand Stopped Experiments
Convert Experiments helps you set up tests and tries to bring testing best practices inside the software. For example we consider an experiment successful (enough) at 97% and not before, and we suggest on every new test to let the test run for a minimum of 7 days. You can change these minimum days and add maximum days as well. However, there are some automations you should be aware of:
- An experiment is marked as "Completed" after at least one variation performs better with a significance greater than 97% (or what you have chosen for Confidence Level) OR all the variations were stopped because they performed worse AND the automation called "Stop Losing Variations" is checked. This automation can only be stopped by setting a minimum running time. For example, setting a minimum running time for 6 month will remove the automation for the whole duration of six months.
- An experiment can keep winning variation running - after test is marked as completed, the winning variation is kept active (this setting is off by default)
- An experiment can stop losing variations - when a variation is decided as being losing (performs worse with a significance of more than 97%) it is stopped and all traffic is redistributed across the other active ones. This setting is off by default.
You can do this by selecting the options menu in the experiments summary. If you have other default settings we should be considering or changing please let us know in the comments.
Restarting your Experiment
Before you restart an experiment, you may need to change your Automation Settings first (mentioned in the beginning of this article); otherwise the experiment may stop again.
In order to restart your experiment, go to your Experience Summary and in the first section, titled "What", click the "Completed" drop-down button and change it to Active. Then, be sure to click the "Activate Experience" button on the bottom right of the page.