Multi-Armed Bandit
Let ABlyft shift traffic automatically to the variations that perform best on your primary goal.
By default, you set the traffic share of each variation yourself and the split stays fixed until you change it. With the Multi-Armed Bandit, ABlyft adjusts the split automatically: variations that perform better on the experiment's primary goal get more traffic, and every variation still gets enough traffic to keep learning about it.
Use it when you want to win as many conversions or as much revenue as possible while the experiment is running. If your main aim is a clean comparison with equal groups, keep the manual split.
- It optimizes the existing primary goal of the experiment. You do not need to set up anything else.
- It works with a Bayesian method (Thompson Sampling). A variation whose result is still uncertain keeps getting enough traffic to be evaluated.
- Visitors keep the variation they were assigned to. A new split only affects visitors who are bucketed for the first time.
- It does not change how the results are calculated. Your statistical methods (Frequentist or Bayesian) stay as configured.
Distribution modes
| Mode | Description |
|---|---|
| Manual (default) | You define the traffic share of each variation, see Traffic allocation. |
| Multi-Armed Bandit | ABlyft calculates the traffic share of each variation automatically. |
Every experiment uses Manual until you enable the Multi-Armed Bandit.
Enable the Multi-Armed Bandit
- Open the experiment.
- Click the cog icon (Edit settings) at the top.
- In the section Traffic Split & Distribution, click Enable Multi-Armed Bandit.
- Confirm the window Enable Multi-Armed Bandit? ("Traffic allocation will be automatically optimized based on your Primary Metric. Variations with 0% traffic will not take part. Existing visitors will remain assigned to their current variations. Only new visitors will be affected by allocation changes.").
Requirements:
- At least two variations with more than 0 % traffic. Otherwise the button is disabled with the tooltip "At least two variations with traffic are required."
- A primary goal, which is the first goal of the experiment. Without it, ABlyft keeps the split unchanged.
What happens when you enable it:
- All variations with more than 0 % traffic take part.
- Variations with 0 % traffic are left out and stay at 0 %. To include them later, switch back to Manual, give them traffic and enable the bandit again.
- The traffic is immediately split evenly across the participating variations, for example 34 / 33 / 33 % for three variations. The remainder goes to the baseline.
- The learning phase starts.
You can enable the bandit in any status, also while the experiment is running. The split is only optimized while the experiment is Running.
Learning
Before it optimizes anything, ABlyft collects first data with the even split. The learning phase ends when both requirements are met:
| Requirement | Value |
|---|---|
| Runtime since the bandit was enabled (or since learning restarted) | At least 24 hours |
| Visitors per participating variation in that time | At least 100 |
Visitors are counted in full days, so all visitors from the day learning started are included. These thresholds are fixed and independent of the decision rules of your project or experiment. You do not need to do anything: optimization starts automatically once both requirements are met.
Optimizing
After the learning phase, ABlyft recalculates the split every 15 minutes while the experiment is running.
- Variations that are more likely to be the best on the primary goal get more traffic. A high result that is still uncertain does not automatically get most of the traffic.
- Every participating variation keeps at least 5 % of the traffic, so the bandit keeps exploring.
- The shares are whole numbers and always add up to exactly 100 %.
- The split is only changed if a share moves by at least 2 percentage points. This avoids constant small changes.
- Every change is published automatically.
- If no calculation is possible, for example because the experiment has no primary goal, the last split stays as it is. The next run tries again.
The primary goal can be any of these:
| Primary goal | What the bandit compares |
|---|---|
| Goal with counting method One | Conversion rate per visitor |
| Goal with counting method Every | Conversions per visitor (can be above 1) |
| Revenue goal | Average value per visitor, with the goal's value filter applied |
The winning direction of the goal is respected: with Decrease, the variation with the lowest value is preferred.
Status
While the bandit is enabled, ABlyft shows its status in two places:
- In the experiment settings, section Traffic Split & Distribution: a badge Multi-Armed Bandit — Learning with the note "The system is collecting initial observations before optimizing traffic allocation.", or Multi-Armed Bandit — Optimizing with the time of the last change, for example "Last optimized: 12 minutes ago".
- In the Status card of the experiment overview: a badge Multi-Armed Bandit · Learning or Multi-Armed Bandit · Optimizing. Hover over it to see the time of the last optimization.
| Status | Meaning |
|---|---|
| Learning | ABlyft collects first data. The traffic is split evenly. |
| Optimizing | ABlyft adjusts the split every 15 minutes. |
The Traffic fields of the variations are read-only while the bandit is enabled. The slider Overall traffic allocation (%) stays editable: it still decides how many eligible visitors take part in the experiment.
Changes while the bandit is enabled
Some changes make the previous data incomparable. ABlyft then resets the traffic to an even split across the participating variations and restarts the learning phase. Optimization resumes automatically when the learning requirements are met again.
| Change | Effect |
|---|---|
| You add a variation | The new variation takes part. Learning restarts. |
| You delete a variation | The variation is removed from the bandit. Learning restarts. |
| You change the primary goal (move another goal to the first position) | Learning restarts. |
If you reset the results of the experiment, the bandit only uses data collected after the reset. In all cases, visitors keep the variation they were assigned to.
Disable the Multi-Armed Bandit
- Open the experiment settings (cog icon, Edit settings).
- In Traffic Split & Distribution, click Disable Multi-Armed Bandit.
- Confirm the window Disable Multi-Armed Bandit? ("Automatic traffic optimization will stop. The current traffic allocation will be preserved and can be adjusted manually.").
The last split is kept; it is not reset to even shares. The Traffic fields are editable again. If you enable the bandit again later, a new learning phase starts.
Good to know
Do visitors switch variations when the split changes?
No. A visitor's variation is assigned once and stored in the browser. Only visitors who are new to the experiment are affected by a new split.
Does the bandit change the statistics?
No. The statistical method and the decision rules stay as configured. Keep in mind that variations with a smaller share collect data more slowly, so they can take longer to reach a clear result.
Why is there no SRM warning?
The SRM check compares the participants with the planned split. Once the bandit has changed the split, that comparison is no longer meaningful. ABlyft therefore does not show the SRM check for experiments whose traffic was optimized by the bandit, also after you switch back to Manual.
Can I change the traffic of a variation through the API?
Not while the bandit is enabled. Updating a variation's traffic_allocation through the
REST API fails with a 422 validation error, and the
MCP server returns an error. Disable the bandit first.
What happens when I clone the experiment?
The clone always starts with the distribution mode Manual.
Why has the split not changed for a while?
- The experiment is still in the learning phase.
- The experiment is not Running.
- No share would change by at least 2 percentage points.
- The experiment has no primary goal.
Can I change the thresholds?
No. The 24 hours, 100 visitors per variation and the minimum of 5 % per variation are fixed.