Pre-test calculator
Estimate how long a test needs to run and how large an effect it can detect.
Before you start an experiment, the Pre-Test Calculator helps you answer: with my traffic, how long must the test run, and how small a change can it still detect? The answer is the Minimum Detectable Effect (MDE): the smallest relative change in conversion rate that your test can reliably detect. A lower MDE needs more participants and a longer runtime.
Open it in your project under Utilities → Pre-Test Calculator. The calculator is a planning tool. It does not read data from your experiments and changes nothing in your project.
Enter your data
Use the numbers of the page and the metric you want to improve, ideally your primary goal.
Test page data
| Field | Description |
|---|---|
| Weekly traffic | Total unique visitors or sessions per week. |
| Weekly conversions | Number of conversions (for example purchases or sign-ups) per week. |
| Number of variations | From 2 to 10, including the control (original). Default: 2. |
Calculator settings
| Field | Description |
|---|---|
| Baseline conversion rate | Calculated automatically as weekly conversions ÷ weekly traffic, in percent. You can overwrite it. |
| Confidence level | From 80% to 99%. It starts with the confidence level of your project. |
The results appear as soon as you enter weekly traffic and a baseline conversion rate. They update while you type.
Read the results
The table shows what happens if you run the test for 1, 2, 3, 4, 6, 8, 10 or 12 weeks:
| Column | Meaning |
|---|---|
| Number of weeks running test | The runtime of the test. |
| Min. Detectable Effect (MDE) | The smallest relative change that the test can detect after this time. |
| Participants / Variation | Weekly traffic × weeks, divided by the number of variations, rounded up. |
The rows are shaded from green (short tests) over yellow to orange (long tests).
Example: with a baseline of 5% and an MDE of 10%, the test can detect a change from 5.0% to 5.5% or more. A smaller change stays hidden in the noise. If your idea is likely to cause a smaller change, you need more weeks, fewer variations or more traffic.
How it is calculated
The calculation uses a statistical power of 80% and this formula for the absolute difference δ:
δ = (Z(confidence) + Z(power)) × √(2 × p × (1 − p) ÷ n)
where p is the baseline conversion rate, n the participants per variation, and Z(confidence) the two-sided z-value of your confidence level. The MDE is δ divided by p, shown in percent.
The calculator expects a baseline conversion rate between 0% and 100% and positive traffic. Otherwise it shows "Enter your test data above to see results."
Tips for planning
- Test for full weeks so that every weekday is included.
- If the MDE is higher than the effect you expect, the test will probably end inconclusive. Test a bolder change, a page with more traffic or fewer variations.
- Compare the runtime with the Minimum runtime and Minimum conversions of your project, see Statistical methods.