Introduction
The success of your experiments is firstly influenced by whether they are set up properly or not. Therefore, a rigorous Quality Assurance (QA) process is needed before launching any new experiments.
There are several things that you can do to make sure your experiment will display and function properly that are described below.
Staging Environment
If you have separate environments, use a staging environment to set up the experiment and perform initial QA without exposing the experiment to visitors. Then, duplicate the experiment into your production environment project, and perform the same QA process there. When possible, ensure that your production and staging environments match so experiments function the same way in both.
If you don’t have a dedicated staging environment, don’t worry. You can still QA your experiment. Perform all the steps outlined below, but take special care when you use the test cookie to QA in a live environment.
Live Preview
Do a live preview of your variations, which can also be done when your experiment has Draft status.
Make sure your pages render as expected in the most popular browsers for your business. Also make sure that any button, link, or form still works properly. Certain options in the editor could break forms, links and buttons if they are not used properly; see http://support.convert.com/hc/en-us/articles/205160385-Why-Are-My-Pages-Showing-Content-from-the-Page-Where-I-Set-Up-the-Experiment-.
To find out how to preview a variation, read the following article: https://convert.zendesk.com/hc/en-us/articles/204506649-How-Do-I-Preview-Variations-
Live previews work with a high accuracy but there are still cases when they can render unexpected results due to how technology works. It is advisable that you always follow the steps described below.
Specific Audience
Place your experiment in "Active" status for a very specific audience that only you can meet the conditions for, and therefore it will run just for you.
To do that, you can create a new Audience rule similar to the following:
Medium matches exactly qa_test
You can replace "qa_test" with whatever you like; just replace it with your preferred text in the Audience rule.
Now to test this, open a fresh Incognito window in your browser. Do not open a second tab and make sure you have no other Incognito windows open. Make sure that you close the window when you are done with the current test and open a new window for each new test. *This is essential.
Before visiting the site, add the following to the end of your URL:
?utm_medium=qa_test (replace qa_test with the value you chose when you created the Audience).
Most of the time you would visit the URL that matches the Site Area conditions. Depending on how your experiment is structured, in some cases you may start out by visiting a different page and then navigate to the Site Area that triggers the experiment to run. In either case, make sure you add the query parameter to the very first URL you visit. This parameter will be remembered even if you navigate to different pages.*
For example, if the URL is http://www.mysite.com, you should visit the URL:
http://www.mysite.com?utm_medium=qa_test.
If you follow the above instructions, you will be shown any variation in the experiment, as this is selected randomly.
After you have done the above, and if you didn't already start with the URL that matches the Site Area conditions, you can then visit the URL that matches the Site Area. You should then be viewing either the Original or one of the Variations, depending on which is randomly selected by the experiment.
Important
* You will need to use the utm_medium=qa_test parameter only once to bucket yourself in the experiment. If the funnel you are testing includes several pages, or you are doing a multi-page test, you only have to use it in the first page you request when you open your incognito session, even if it that first page is not included in the experiment. Afterwards, when the test is triggered at the time you match the Site Area conditions, you will be shown the chosen variation, and the Experiment Report will be updated.
Force a specific variation
You can force a specific variation, by using certain query parameters.
Example: http://www.mysite.com/test-page.html?_conv_eforce=123.678 where 123 is the experiment ID and 678 is the variation ID;
You can find a more detailed explanation here.
Important
When forcing a variation, you must still match the Site Area and Audience conditions for this to work properly.
Tip
Forcing a variation is also handy when you have an experiment set to less than 100% traffic. It will force you to be bucketed in; whereas if you were to test normally, you have no control over whether you will be part of the percentage that randomly gets selected to be bucketed in.
Combining Parameters
You can also combine query parameters. For instance, if you want to do a qa test but also want to force a specific variation, you would append the query parameter to the end of the previous parameter with an ampersand (&) between them. Only the first parameter would have the question mark at the beginning, ie the below example shows both the force variation parameter and qa_test Audience condition paramter combined:
http://www.mysite.com/test-page.html?_conv_eforce=123.678&utm_medium=qa_test
Tip
Any change you make inside your account will become live in max five minutes. Therefore, you might need to wait up to five minutes until the experiment becomes live.
Tip
Many goals are restricted to the URL criteria you have set. So when testing in QA mode remember that goals that have a "Matches Exactly" criteria would need the chosen QA parameter with its value, ie "?utm_medium=qa_test"; otherwise, the goal will not be triggered.
Tip
Our Chrome debugging extension is also really useful when it comes to experiments not running correctly: http://support.convert.com/hc/en-us/articles/204506699-Chrome-Debugger-For-Convert-Experiments-beta-
If you are experiencing problems, for instance issues with click goals, please refer to this article for debugging.
Comments