Interpret results
When a result is reliable, what winner, loser and inconclusive mean, and what to do next.
The results page tells you for every variation and goal whether it is a winner, a loser or inconclusive. This page explains how to read these states and what to check before you act on them.
What the states mean
| State | Color | Meaning |
|---|---|---|
| winner | Green, trending up icon | The variation is better than the baseline in the winning direction of the goal, and all decision rules are met. |
| loser | Red, trending down icon | The variation is confidently worse than the baseline. |
| inconclusive | Gray, clock icon | ABlyft cannot say yet. Either the result is not certain enough, or a decision rule is not met. |
"Better" depends on the winning direction of the goal. With Decrease, a variation that lowers the metric is the winner. See Goal types.
Inconclusive does not mean "no effect". It means the data does not allow a statement yet, or that the effect is too small to detect with the traffic you have.
When is a result reliable?
Check these points before you decide:
- The state is winner or loser for your primary goal, not only a high percentage.
- The decision rules are met. The summary shows minimum runtime and minimum conversions, see Decision rules.
- Enough data. See How long to wait.
- The interval does not cross zero. In the Confidence Interval or Credible Interval bar, the whole range should be on one side of zero.
- No traffic problem. No SRM badge next to the variations (see below).
- The goal measures what you think. Check Verify measurement.
Do not stop at the first green result
Results fluctuate, especially at the beginning. The decision rules protect you from stopping on a random peak. Let an experiment run for full weeks so that all weekdays are included.
How long to wait
For inconclusive variations, the Statistical State column shows an estimate such as "≅ 12 days (≅ 3,400 participants) left". It is a rough guide that depends on the current data and changes over time. "The duration cannot be determined yet" means there is not enough data for an estimate. To plan the runtime before you start, use the Pre-test calculator.
Check the traffic split (SRM)
If SRM check is enabled under Project → Settings → Statistics & Analytics, ABlyft compares the actual number of participants per variation with the traffic allocation you planned. If they differ more than chance explains, a badge SRM appears next to the variation, with the tooltip "SRM detected".
SRM = Sample Ratio Mismatch. The setting marks p < 0.005 in orange and p < 0.001 in red. A mismatch points to a technical problem, for example a variation that breaks on some browsers or a redirect that loses visitors. Do not use the results of an experiment with a red SRM warning until you found the cause. See Implausible results.
Compare more than one goal
Look at the primary goal first: it is the goal you chose as your success metric. Secondary goals help you understand why a variation works or what it costs, for example a higher click rate but lower revenue per participant. If you look at many goals, a single green result by chance becomes more likely, so give the primary goal the most weight.
Use the filters to look deeper, for example by Device types. A variation can win on desktop and lose on mobile. Filtered results show fewer participants, so they are less certain.
What to do next
| Result | Typical next step |
|---|---|
| Winner on the primary goal | Implement the variation on your website for everyone, then stop the experiment. |
| Loser | Keep the original and use the learning for your next idea. |
| Inconclusive at the end of the planned runtime | The change has no effect you can detect. Decide on the original or on the simpler version, or test a bolder change. |
To start, pause or stop an experiment, see Start, pause & stop.
Get help interpreting
The Interpret button on the results page opens a chat with the ABlyft AI assistant, which can explain the results of the experiment. See AI features.