Results look implausible
No conversions, an uneven split or numbers that seem too good to be true. Find the cause.
Sometimes the numbers in your results do not look right: a goal has no conversions, the visitors are split unevenly, or one variation wins by a huge margin after a day. This page helps you decide whether it is a measurement problem or just normal behavior.
Too early to tell
Small numbers fluctuate a lot. Early in an experiment, a big difference between variations is usually chance.
- Let the experiment collect enough visitors and conversions. The decision rules define when a result counts as a success.
- Use the Pre-test calculator to estimate how many visitors you need.
- Do not stop an experiment the moment it first looks significant, see Interpret results.
A goal has no conversions
If a goal stays at zero, the goal is probably never triggered. Check by goal type.
| Goal type | Typical cause |
|---|---|
| Pageview | The page of the goal does not match the URL, or the goal is not assigned to the page. The goal is only counted while its page is active. |
| Click | The CSS selector does not match the element that is clicked. Selectors that depend on generated class names stop working after a release. |
| Custom | The API name in the call ablyft.push() differs from the API name of the goal (it is case sensitive), or the call is never reached. |
| Revenue | Same as custom, and the call sends no value or a wrong value. |
Verify each goal live, without leaving ABlyft:
- Open your site in Preview mode with the quality assurance tool (see Preview & QA). Goals are not sent to the results in preview mode, but you can see whether they fire.
- Open Live Log in your project to see the events that really arrive from real visits.
See Verify measurement and Goal types.
Goals only count for visitors in an experiment
A goal is only sent for visitors who were assigned to a variation. A visitor who was not included, for example because of the traffic allocation, does not produce conversions.
Conversions look too low or too high
- Counting method. With Every, one visitor who clicks ten times counts ten times. Make sure the method fits the question you ask, see Counting method.
- Goal fires more often than you think. For example a click goal on an element that is also clicked by scripts, or a custom goal called in a loop.
- Revenue values. Check that your page sends the amount as a plain number, without currency sign or thousands separators, and consistently (same currency, with or without tax). Also check the value filter of the goal, see Value filters.
- Your own visits. Visits by your team count like any other visits. Exclude yourself with the opt-out (once per browser) or use the preview mode, which does not count goals.
Goals on another domain
Visitors are assigned to a variation in their browser. A goal can only be counted if the page on which it happens knows the assignment.
- On a subdomain of the same website this works out of the box.
- On a different domain (for example a separate checkout on
pay.example.org), enter that domain in Settings → Project Settings under External tracking domains. ABlyft then passes the assignment on through a URL parameter when visitors move over. The snippet must be installed on the other domain as well.
Without this, conversions on the other domain are lost, and results look too low. See Snippet settings.
The split between variations is uneven
If you expect 50/50 but see, for example, 60/40 with thousands of visitors, you may have a sample ratio mismatch (SRM). Small deviations are normal chance. A strong deviation points to a technical problem, and the results cannot be trusted.
ABlyft can warn you, see Check the traffic split (SRM).
Typical causes:
- A variation loads slower or breaks. If the code of a variation has an error, redirects slowly, or crashes in some browsers, those visitors leave before they are counted.
- Changes during the run. Changing traffic percentages or the audiences while the experiment runs shifts the split.
Check variations in other browsers and on mobile devices, and see Changes are missing.
Overlapping experiments
A visitor can be in several experiments at the same time on the same page. If the experiments change the same area or influence the same goal, they distort each other. Put such experiments into an exclusion group, so that a visitor only takes part in one of them. See Exclusion groups.
Other effects on the numbers
- Bots are filtered by their browser identification only, so some automated visits may remain.
- Seasonal effects such as campaigns, holidays or a sale in the middle of the run. Let the experiment run for full weeks.
- Changes on your website during the experiment, such as a new release, can change behavior for all variations.
- Results update with a delay. Events are processed regularly; very recent visits can take a short moment to appear in the results. The Live Log shows events earlier.
Still unsure?
Contact ABlyft support and include the experiment, the goal in question and, if possible, a screenshot of the Live Log.
Next steps
- Interpret results
- Verify measurement
- Debugging for developers