Understand Experience Report
Deriving meaningful insights from your experiences and applying them to improve your website conversion rate is your experiences' ultimate objective. The experience reports help you validate the hypothesis you test and derive a better understanding of your website's visitor behavior.
To access your test report, select the experience for which you want to view the report, and go to the REPORT tab.
You will be able to see 4 different sections
- Top Menu
On the top menu you can quickly see this information:
- Start Date: the date when the experience started
- Days Running: how many days the experience is running
- Report Range: date filter enables you to filter data for a specified date range
- Segment: filter the report by visitor segments
- All Tested Users: number of users participating in the test
- Total Conversions: total number of goal conversions
- Goals: number of goals the experience has
You can also use the three-dot menu and apply some further actions:
By clicking the first option, Share Report, a popup appears where you can download the experience data in various formats:
If you click the second option, View Code, you will be able to access your tracking code:
From the third option, Stats Settings, you can configure these:
- Confidence level (statistical sognificance) is described here
- Traffic Distribution is described here
- Transaction Outliers are described here
- Automations are described here
The Reset Stats button can reset the experience data.
If you hover over the baseline screenshot, you will see some additional options:
From the Variations section of the report you can quickly view the variations of your experience and some extra information:
If you click the three-dot menu you can enable/disable columns that you would like to be visible or even re-arrange their order:
If you click on the status you will be able to pause variations.
If you click the three-dot menu on one of the variations row you have some extra options:
In the goals section, you can see all the goals of the experience and some interesting stats and graphs that we are going to analyze below. Before that, we need to explain few tags/terms that you might see and do not recognize.
- Default Goal: If you see the term Default Goal next to one of your goals that means it is one of the 2 default goals we add to every experience
- Primary Goal: This can only be one and is decided by you. It is the most important goal for your experience. This will be shown first on the goals list and experience status will be changed based on this.
- SRM: Potential Sample Ratio Mismatch detected on your test and you need to contact firstname.lastname@example.org or check the experience setup.
- Baseline: By default is your experience baseline.
- Variation Name: name of your variation. Next to it, there is a checkbox that you can enable/disable to show or hide the variation stats on the graph
- Improvement: this column reports the percentage change of the variation compared to the Original. When computed confidence is greater than the confidence set in the stats settings the color changes
- Red for -
- Green for +
- Gray for other
- Visitors: the amount of total bucketed visitors in the experience. These are unique visitors.
- Conversions: this represents the total conversions for each goal/variation
- Conversion Rate: this column shows the percentage of visitors that turned into conversions
- Confidence Level (statistical significance): this column reports the significance, or how different the confidence interval for the conversion rate for the experiment variation is when compared to the original. If the confidence is not showing any number, this is because (by default) there is a minimum of 5 conversions needed for each variation in order to calculate it. It also must meet the minimum visitors set for each variation. If you have changed the minimum conversions then the minimum you chose would have to be met. The gray/green dots in that column indicate:
- 1 green dot for 75%-85% confidence
- 2 green dots for 85%-95% confidence
- 3 green dots for 95%-96% confidence
- 4 green dots for 96%-97% confidence
- 5 green dots for 97% and above
- Confidence Interval: range of values where the true conversion rate falls.
- Status: status of the variation
There are three available groups of graphs that are explained below:
- Conversion Rate Over Time: Y axis shows conversion rate , X axis shows time, the lines are one for each variation (cumulative conversion rate) + one for conversion rate of the whole experiment (average of all variations) + a second axis Y at the right which represent the cumulative number of visitors inside the whole experimen
- Conversions Over Time: Y axis shows conversions, X axis shows time, the lines are one for each variation (conversions) + one for conversions of the whole experiment (average conversions of all variations) + a second axis Y at the right which represent the cumulative number of conversions inside the whole experiment.
- Daily Conversion Rate Over Time: non cumulative conversion rate - same as graph Conversion Rate Over Time, only that we do not show a second Y axis with cumulative visitors and the non cumulative is used for the API
- Improvement Over Time: improvement in conversion rate by day on the X axis and lines each variation (but not original).
- Daily Visitors over Time
- Products Per Visitor: cumulative average ordered products per visitor - Y axis shows average ordered products per visitor, X axis shows time, the lines are one for each variation (cumulative average ordered products per visitor) + one for average ordered products per visitor of the whole experiment (average of all variations) + a second axis Y at the right which represent the cumulative number of visitors inside the whole experiment
- Daily Products Per Visitor: non cumulative average ordered products per visitor
- Improvement over Time: improvement in products by day on the X axis and lines each variation (but not original)
- Daily Visitors over Time
- Revenue over Time: cumulative average revenue per visitor - Y axis shows average revenue per visitor, X axis shows time, the lines are one for each variation (average revenue per visitor) + one for average revenue per visitor of the whole experiment (average of all variations) + a second axis Y at the right which represent the cumulative number of visitors inside the whole experiment
- Daily Revenue Per Visitor: non cumulative average revenue per visitor
- Improvement over Time: improvement in revenue by day on the X axis and lines each variation (but not original)
- Visitors over Time