Experiment Report

Understand Experience Report

Deriving meaningful insights from your experiences and applying them to improve your website conversion rate is your experiences' ultimate objective. The experience reports help you validate the hypothesis you test and derive a better understanding of your website's visitor behavior. 

To access your test report, select the experience for which you want to view the report, and go to the REPORT tab.

You will be able to see 4 different sections

  • Top Menu
  • Summary
  • Variations
  • Goals

mceclip1.png

Top Menu

On the top menu you can quickly see this information:

  • Start Date: the date when the experience started
  • Days Running: how many days the experience is running
  • Report Range: date filter enables you to filter data for a specified date range
  • Segment: filter the report by visitor segments
  • All Tested Users: number of users participating in the test
  • Total Conversions: total number of goal conversions
  • Goals: number of goals the experience has

mceclip2.png

You can also use the three-dot menu and apply some further actions:

mceclip3.png

Share Report

By clicking the first option, Share Report, a popup appears where you can download the experience data in various formats:

mceclip4.png

View Code

If you click the second option, View Code, you will be able to access your tracking code:

mceclip5.png

Stats Settings

From the third option, Stats Settings, you can configure these:

  • Confidence level (statistical sognificance) is described here
  • Traffic Distribution is described here
  • Transaction Outliers are described here
  • Automations are described here

mceclip6.png

The Reset Stats button can reset the experience data.

Summary

Summary part of the report summarizes the Site Area and Audience conditions that apply to this experience. Additionally, it provides some smart recommendations:

mceclip7.png

If you hover over the baseline screenshot, you will see some additional options:

mceclip9.png

The Preview Variation will open a popup and you will be able to get the Live Preview and Force Variation URLs.

Variations section

From the Variations section of the report you can quickly view the variations of your experience and some extra information:

mceclip10.png

If you click the three-dot menu you can enable/disable columns that you would like to be visible or even re-arrange their order:

mceclip11.png

If you click on the status you will be able to pause variations.

mceclip0.png

If you click the three-dot menu on one of the variations row you have some extra options:

mceclip12.png

Goals section

In the goals section, you can see all the goals of the experience and some interesting stats and graphs that we are going to analyze below. Before that, we need to explain few tags/terms that you might see and do not recognize.

  • Default Goal: If you see the term Default Goal next to one of your goals that means it is one of the 2 default goals we add to every experience
  • Primary Goal: This can only be one and is decided by you. It is the most important goal for your experience. This will be shown first on the goals list and experience status will be changed based on this.
  • SRM: Potential Sample Ratio Mismatch detected on your test and you need to contact support@convert.com or check the experience setup.
  • Baseline: By default is your experience baseline.

Columns

mceclip1.png

  • Variation Name: name of your variation. Next to it, there is a checkbox that you can enable/disable to show or hide the variation stats on the graph
  • Improvement: this column reports the percentage change of the variation compared to the Original. When computed confidence is greater than the confidence set in the stats settings the color changes
    • Red for -
    • Green for +
    • Gray for other 
  • Visitors: the amount of total bucketed visitors in the experience. These are unique visitors.
  • Conversions: this represents the total conversions for each goal/variation
  • Conversion Rate: this column shows the percentage of visitors that turned into conversions
  • Confidence Level (statistical significance): this column reports the significance, or how different the confidence interval for the conversion rate for the experiment variation is when compared to the original. If the confidence is not showing any number, this is because (by default) there is a minimum of 5 conversions needed for each variation in order to calculate it. It also must meet the minimum visitors set for each variation. If you have changed the minimum conversions then the minimum you chose would have to be met. The gray/green dots in that column indicate: 
    • 1 green dot for 75%-85% confidence
    • 2 green dots for 85%-95% confidence
    • 3 green dots for 95%-96% confidence
    • 4 green dots for 96%-97% confidence
    • 5 green dots for 97% and above
  • Confidence Interval: range of values where the true conversion rate falls.
  • Status: status of the variation

Graphs

There are three available groups of graphs that are explained below:

mceclip0.png

Conversions

mceclip2.png

  • Conversion Rate Over Time: Y axis shows conversion rate , X axis shows time, the lines are one for each variation (cumulative conversion rate) + one for conversion rate of the whole experiment (average of all variations) + a second axis Y at the right which represent the cumulative number of visitors inside the whole experimen
  • Conversions Over Time: Y axis shows conversions, X axis shows time, the lines are one for each variation (conversions) + one for conversions of the whole experiment (average conversions of all variations) + a second axis Y at the right which represent the cumulative number of conversions inside the whole experiment. 
  • Daily Conversion Rate Over Time: non cumulative conversion rate - same as graph Conversion Rate Over Time, only that we do not show a second Y axis with cumulative visitors and the non cumulative is used for the API
  • Improvement Over Time: improvement in conversion rate by day on the X axis and lines each variation (but not original). 
  • Daily Visitors over Time

Products

mceclip1.png

  • Products Per Visitor: cumulative average ordered products per visitor - Y axis shows average ordered products per visitor, X axis shows time, the lines are one for each variation (cumulative average ordered products per visitor) + one for average ordered products per visitor of the whole experiment (average of all variations) + a second axis Y at the right which represent the cumulative number of visitors inside the whole experiment
  • Daily Products Per Visitor: non cumulative average ordered products per visitor
  • Improvement over Time: improvement in products by day on the X axis and lines each variation (but not original)
  • Daily Visitors over Time

Revenue

mceclip0.png

  • Revenue over Time: cumulative average revenue per visitor - Y axis shows average revenue per visitor, X axis shows time, the lines are one for each variation (average revenue per visitor) + one for average revenue per visitor of the whole experiment (average of all variations) + a second axis Y at the right which represent the cumulative number of visitors inside the whole experiment
  • Daily Revenue Per Visitor: non cumulative average revenue per visitor
  • Improvement over Time: improvement in revenue by day on the X axis and lines each variation (but not original)
  • Visitors over Time

 

 

 

 

Need assistance? Start a conversation with our Support Team.
CONVERT MORE

Reliably Convert More Site Visitors, Even If Everything Else is Changing in Your Business.

START 15-DAY FREE TRIAL
✓ No Credit Card  ✓ Easy Set-Up  ✓ Access All Features

Comments