QA Tools in Convert
Convert offers several tools to help your QA experiences while you are developing, and before launching them.
X-Frames Extension may be Required for Working in Visual Editor
If you see a blue "proxy message" bar in Visual Editor, please see this article.
Chrome Debugger Extension
This tool outputs the logs of the Convert script to the Chrome developer tools console.
The output includes experiments triggered, specific variations triggered and allows you to determine the sequence of events in regards to testing. This provides a lot of useful information when debugging a test.
This tool should be used throughout the whole QA process to determine if you are bucketed into the experiment and determine the variation that you are seeing.
You can find more information on how to install it in this article.
It is very common to be bucketed on the original variation and assume mistakenly that the experiment is not working. This is why we recommend that you enable the extension in incognito sessions and perform the QA there.
All variation and experiment references in the debugger output are made with the corresponding IDs. You can find out how to determine IDs in this article.
Preview Variations URL
These URLs are meant to be an aid while developing/drafting an experience using the Visual Editor. While working, the experiment QA tester should go back and forth between these URLs and the Visual Editor view. The Visual Editor view may not be completely accurate since the window in which the editing occurs is done in a frame. Also, the code that makes changes in the editor runs repeatedly for ease of use.
There are some considerations to take into account:
- The experiment conditions from the Site Area and the Audience are not taken into account when doing a preview with the Live Preview URL. The changes are made to the page URL on which you append the parameters. This is for making it easy to test different views.
- The Previews are served from our centralized app server, instead of the distributed CDN servers. Normal visitors to an experiment are served the experiment from the CDN servers. Because of this, there might be a difference in download speeds and other conditions when using the Preview URL but not during the normal visitor experience.
You can find these URLs automatically generated for you in the Visual Editor > Variation Menu > Live Preview, and in the report alongside the Original and each Variation (look for the eyeball icon as shown in the screenshot above).
Force Variation URLs
The purpose of Force Variation URLs is for you to test your experiment in the closest environment to what a visitor will experience. These URLs should be used to do the final testing of your experiment.
Variations served with this URL are served from the same CDN server we use to serve the experiments.
When using a Force Variation URL, the experiment conditions will also be evaluated, so make sure that they are matched.
Also, when using a Force Variation URL and testing goals, take into account that goal configurations often include URL conditions. If your goals do not trigger when using a Force Variation URL this might be why.
You will find more information on how to format them in this article.
Use of query parameters in "QA Test" Audiences
The purpose of a special audience is to allow you to Activate a test and verify it without allowing any visitors to be bucketed into it.
There is a defined process for this, as described in this article.
Also, be aware that adding a special audience might affect your ability to match the Site Area and goal URL conditions. You would have to take this into account. More information can be found in this article.
QA Stages in Convert
1st Stage - Visual Editor and Preview Variation URLs
- Preview Variations URLs
To validate (to a certain degree of accuracy) that the changes created by the Visual Editor look good in a normal browser window.
- Develop your test using the Visual Editor.
- Preview each variation change with the Preview URLs provided in either the Visual Editor or on the Reports page. Previews open in a new tab.
- Make changes in the Visual Editor, then preview; repeat until your variation changes look the way you want them.
2nd Stage - Preview with Force Variation URLs
- Force Variation URLs
- Incognito Window
- Chrome Debugger Extension (the extension needs to be enabled in Incognito)
- Query parameter Audience (eg, special qa_test audience defined by you)
Goal: To validate how a specific variation looks when served from Convert CDN servers and test your experiment Site Area and Audience conditions, but restrict it to your QA audience.
- Complete your experiment configuration, including the Site Area and Audience conditions.
- Set a QA audience in your experiment.
- Open a fresh Incognito browser window (only one instance should be open at a time as they share cookies between them -- this will affect your results if you have more than one incognito window or tab open or forget to close the incognito window after a test).
- Trigger each of the variations with “Force Variation URLs” (original is not necessary). Close and open another incognito session when testing each variation. Make sure you append the query parameter that you defined in your qa_test Audience to the end of your Force Variation URL (ie, something like ”&utm_medium=qa_test”).
- If you see that the forced variation is not showing what you expect, open the Chrome Debugger Extension, and go through the logs to see if the experiment conditions are met. You can search the log with the specific experiment ID to aid in your search.
- Fix any experiment issues if encountered. Repeat the testing process until the experiment and changes show correctly. If problems are found, please contact Convert Support.
3rd Stage - QA the whole setup while viewing the experiment report
- Force Variation URLs
- Incognito Window
- Chrome Debugger Extension (enabled on the incognito window)
- Query parameter Audience
- Experiment Report
Goal: Validate that the entire experiment works as expected.
- Open your experiment report.
- Open an incognito browser window and the page of the experiment including the QA audience parameter.
-- Use this if you don't want visitors to see the experiment yet -- additionally, if you have visitors increasing the numbers on your report, it's impossible to tell if the numbers that appear on the report are from your own visit and conversion, or your visitors' visits and conversions.
-- If your experiment was Activated in the past without a QA audience, previously bucketed visitors might trigger conversions even if you set a QA audience later. The reason for this is that once a visitor is bucketed into an experiment (and has not yet triggered all goals) when they return we do not check to see if they match the audience conditions).
- Note the variation you see. Check if your visit to that variation is shown on the report (visitor count will be incremented by one, usually within a couple of seconds). Trigger your goals and check that your conversions are counted on the report. Verify that the variation is functioning and visually showing what you expect. Also, remember that visitors and goals are unique - you cannot trigger them again unless you close and open a brand-new incognito window.
- Close and reopen the incognito window. Repeat process until all variations and goals are validated.
- Use Chrome debugger to aid you in identifying and fixing any issues encountered.
- If problems are found, please contact Convert Support.
4th and Final Stage - You're Ready!
Remove QA Audience and Reset Experiment
Once you have finished all of the above, remove your special QA Test Audience to allow your visitors to trigger the experiment. You can also use "Reset Experiment" on the Reports page under Advanced Settings. This will reset the start date of the experiment, as well as clear the previous visitor numbers and previous conversions to 0 on the report.
Once again, keep in mind that the Site Area and Goals might refer to specific URLs that might be matched with your special audience query parameter, but not without it, so check carefully.
Clone Experiment, Double-Check Configuration and Activate
Note that if you had your experiment Active at any point without a special audience to restrict visitors, and you use "Reset Experiment", it does not remove cookies from visitors browsers. If they return we will recognize them as previously bucketed into the experiment. In that case, you could find a conversion with no additional visitor on the report (if the visitor was bucketed prior to resetting the experiment and did not trigger all conversions). An option to avoid this is to Clone your experiment and Activate the cloned one instead - if doing this, double-check your experiment configuration on the Cloned experiment!