QA Guide

THIS ARTICLE WILL HELP YOU:

• QA Tools in Convert
• X-Frames Extension may be Required for Working in Visual Editor
• Chrome Debugger Extension
• Preview Variations URL
• Forced Variation URLs
• Use of query parameters in "QA Test" Audiences
• QA Stages in Convert
• 1st Stage - Visual Editor and Preview Variation URLs
• 2nd Stage - Preview with Force Variation URLs

• 3rd Stage - QA the whole setup while viewing the experiment report
• 4th and Final Stage - You're Ready!

QA Tools in Convert

Any company introducing experimentation into their digital marketing strategy should incorporate QA measures into their operational workflows, such as certifying integrations and previewing experiences before going live.

At Convert, we integrate the many pieces of a puzzle to create a personalized strategy in the simplest possible way. We’ve helped hundreds of businesses improve their ability to build, test, review, and launch experiences with our diverse tools and resources.

Let us show you what they are and how they can be a well-rounded solution to all your QA problems.

Chrome QA Widget

This is the primary tool you should utilize for quality assurance (QA). It enables you to view experiment changes and identify which experiment and variant you are assigned to when loading a web page. Additionally, it displays the goals and experiments that have been activated on the current page. For detailed guidance, you can refer to the documentation provided in this article.

Chrome Debugger Extension

This tool outputs the logs of the Convert script to the Chrome developer tools console.

The output includes experiments triggered, specific variations triggered, and allows you to determine the sequence of events in regards to testing. This provides a lot of useful information when debugging a test.

This tool should be used throughout the whole QA process to determine if you are bucketed into the experiment and determine the variation that you are seeing.

You can find more information on how to install it in this article.



It is very common to be bucketed on the original variation and assume mistakenly that the experiment is not working. This is why we recommend that you enable the extension in incognito sessions and perform the QA there.

All variation and experiment references in the debugger output are made with the corresponding IDs. You can find out how to determine IDs in this article.

Preview Variations URL

These URLs are meant to be an aid while developing/drafting an experience using the Visual Editor. While working, the experiment QA tester should go back and forth between these URLs and the Visual Editor view. The Visual Editor view may not be completely accurate since the window in which the editing occurs is done in a frame.  Also, the code that makes changes in the editor runs repeatedly for ease of use. 

There are some considerations to take into account:

The experiment conditions from the Site Area and the Audience are not taken into account when doing a preview with the Live Preview URL. The changes are made to the page URL on which you append the parameters. This is for making it easy to test different views.

You can find these URLs automatically generated for you in the Visual Editor > Variation Menu > Live Preview, and in the report alongside the Original and each Variation (look for the eyeball icon as shown in the screenshot above).

Forced Variation URLs

The purpose of Force Variation URLs is for you to test your experiment in the closest environment to what a visitor will experience. These URLs should be used to do the final testing of your experiment. These links will only work when the experiment is active.

Forced Variation URLs will only work on active experiments, and not on Draft ones.

Variations served with this URL are served from the same CDN server we use to serve the experiments.



When using a Force Variation URL, the experiment conditions will also be evaluated, so make sure that they are matched. If you are using a QA audience, you could complement the Force Variation URL with the QA audience parameters, like:

https://www.convert.com/?_conv_eforce=1001173467.100121503&utm_medium=qa

Also, when using a Force Variation URL and testing goals, take into account that goal configurations often include URL conditions.  If your goals do not trigger when using a Force Variation URL this might be why.

You will find more information on how to format them in this article.

Use of query parameters in "QA Test" Audiences

You can restrict the visitors that will see your experiment by adding a QA audience to the experiment. You will be able to be bucketed to the audience to see the experiment by adding the QA query parameters to the experiment URL like https://www.domain.com/mypage.html?utm_medium=qa. Make sure you use them on a fresh incognito window. Because, the Convert script will not read them after the first hit to the site.

Create a New Audience, with a condition like:
Medium matches exactly qa



The purpose of a special audience is to allow you to Activate a test and verify it without allowing any visitors to be bucketed into it.

First, attach this audience to your experiment.

Now to test this, open a fresh Incognito window in your browser. Do not open a second tab and make sure you have no other Incognito windows open. Make sure that you close the window when you are done with the current test and open a new window for each new test. *This is essential.

Before visiting the site, add the following to the end of your URL:
?utm_medium=qa (replace qa with the value you chose when you created the Audience).

Most of the time you would visit the URL that matches the Site Area conditions. Depending on how your experiment is structured, in some cases you may start out by visiting a different page and then navigate to the Site Area that triggers the experiment to run. In either case, make sure you add the query parameter to the very first URL you visit. This parameter will be remembered even if you navigate to different pages.*

For example, if the URL is http://www.mysite.com, you should visit the URL:
http://www.mysite.com?utm_medium=qa

If you follow the above instructions, you will be shown any variation in the experiment, as this is selected randomly.

After you have done the above, and if you didn't already start with the URL that matches the Site Area conditions, you can then visit the URL that matches the Site Area. You should then be viewing either the Original or one of the Variations, depending on which is randomly selected by the experiment.

Also, be aware that adding a special audience might affect your ability to match the Site Area and goal URL conditions.   You would have to take this into account.

QA Stages in Convert

These stages should be performed in each of the type devices that you plan to target such as desktop and mobile, to ensure experiment bucketing, display of changes, and goal tracking.

1st Stage - Visual Editor and Preview Variation URLs

Tools used:

1. Preview Variations URLs

Goal:

To validate (to a certain degree of accuracy) that the changes created by the Visual Editor look good in a normal browser window.

Process:

1. Develop your test using the Visual Editor.
2. Preview each variation change with the Preview URLs provided in either the Visual Editor or on the Reports page. Previews open in a new tab.
3. Make changes in the Visual Editor, then preview; repeat until your variation changes look the way you want them.

2nd Stage - Preview with Force Variation URLs

Tools Used:

1. Force Variation URLs
2. Incognito Window
3. Chrome Debugger Extension (the extension needs to be enabled in Incognito)
4. Query parameter Audience  (eg, special qa_test audience defined by you)

Goal: To validate how a specific variation looks when served from Convert CDN servers and test your experiment Site Area and Audience conditions, but restrict it to your QA audience.

Process:

1. Complete your experiment configuration, including the Site Area and Audience conditions.
   
   
2. Set a QA audience in your experiment.
   
   
3. Open a fresh Incognito browser window (only one instance should be open at a time as they share cookies between them -- this will affect your results if you have more than one incognito window or tab open or forget to close the incognito window after a test).
   
   
4. Trigger each of the variations with “Force Variation URLs” (original is not necessary). Close and open another incognito session when testing each variation.  Make sure you append the query parameter that you defined in your qa_test Audience to the end of your Force Variation URL (ie, something like ”&utm_medium=qa_test”).
   
   
5. If you see that the forced variation is not showing what you expect, open the Chrome Debugger Extension, and go through the logs to see if the experiment conditions are met. You can search the log with the specific experiment ID to aid in your search.
   
   
6. Fix any experiment issues if encountered. Repeat the testing process until the experiment and changes show correctly. If problems are found, please contact Convert Support.

3rd Stage - QA the whole setup while viewing the experiment report

Tools Used:

1. Force Variation URLs
2. Incognito Window
3. Chrome Debugger Extension (enabled on the incognito window)
4. Query parameter Audience
5. Experiment Report

Goal: Validate that the entire experiment works as expected.

Process:

1. Open your experiment report.
   
   
2. Open an incognito browser window and the page of the experiment including the QA audience parameter.
   
   -- Use this if you don't want visitors to see the experiment yet -- additionally, if you have visitors increasing the numbers on your report, it's impossible to tell if the numbers that appear on the report are from your own visit and conversion, or your visitors' visits and conversions.
   
   -- If your experiment was Activated in the past without a QA audience, previously bucketed visitors might trigger conversions even if you set a QA audience later.  The reason for this is that once a visitor is bucketed into an experiment (and has not yet triggered all goals) when they return we do not check to see if they match the audience conditions).
   
   
3. Note the variation you see. Check if your visit to that variation is shown on the report (visitor count will be incremented by one, usually within a couple of seconds). Trigger your goals and check that your conversions are counted on the report. Verify that the variation is functioning and visually showing what you expect.  Also, remember that visitors and goals are unique - you cannot trigger them again unless you close and open a brand-new incognito window.
   
   
4. Close and reopen the incognito window. Repeat the process until all variations and goals are validated.
5. Use Chrome debugger to aid you in identifying and fixing any issues encountered.
   
   
6. If problems are found, please contact Convert Support.

4th and Final Stage - You're Ready!

Remove QA Audience and Reset Experiment

Once you have finished all of the above, remove your special QA Test Audience to allow your visitors to trigger the experiment.  You can also use "Reset Experiment" on the Reports page under Advanced Settings.  This will reset the start date of the experiment, as well as clear the previous visitor numbers and previous conversions to 0 on the report.

Once again, keep in mind that the Site Area and Goals might refer to specific URLs that might be matched with your special audience query parameter, but not without it, so check carefully.

Clone Experiment, Double-Check Configuration, and Activate