Everything you need to know about variance reduction at low traffic

The Delta Method is the tool that makes online experiments trustworthy when the randomisation unit differs from the analysis unit. It replaces a variance estimate that is quietly wrong with one that survives contact with real traffic. Natively supported by every serious experimentation platform, it is the foundation everything else is built on.

In this guide we will go through every part of it: the estimator, why the naive variance fails, how CUPED buys sensitivity, and how to choose a test when the data is skewed.

Why the naive variance is wrong

Before reaching for a fix, it is worth understanding why the usual formula fails. It is not a matter of taste: it is a design problem that produces false positives in almost every product that measures per-page rather than per-user.

A formula inherited from textbooks

Let $X_1, \dots, X_n$ be the control observations and $Y_1, \dots, Y_m$ the treatment ones. The average treatment effect is estimated by $\hat{\delta} = \bar{Y} - \bar{X}$, and under independence its variance is

End of story — except that observations are rarely independent. Here are the pitfalls that follow:

# Page views are correlated within a user: a heavy user drags the mean
import numpy as np
 
views = np.array([1, 1, 2, 48, 51, 2])          # six sessions, two users
naive = views.var(ddof=1) / len(views)          # wrong: assumes independence
print(naive)                                     # 407.9 — far too small
 
# Clustering by user first is what the Delta Method formalises
clusters = [views[:3], views[3:]]
means = np.array([c.mean() for c in clusters])
print(means.var(ddof=1) / len(clusters))         # 552.1 — the honest number

The cost of getting it wrong

Underestimating the variance narrows the confidence interval, which inflates the false positive rate. A test that reports 5% is really running at 15% or worse.

Below roughly 10³ users the interval covers zero: the effect is real and undetectable. Variance reduction moves that row off zero without collecting more data.

Reducing the variance

Three techniques matter in practice, and they trade off differently.

CUPED in one line

CUPED replaces the metric $Y$ with $Y - \theta X$, where $X$ is any covariate measured before the experiment. The optimal coefficient is the regression slope of $Y$ on $X$, and the variance becomes $\operatorname{Var}(Y)(1 - \rho^2)$.

def cuped(y, x):
    theta = np.cov(x, y)[0, 1] / np.var(x)
    return y - theta * (x - x.mean())

When the covariate correlates strongly with the metric, $\rho \to 1$ and the variance collapses. In practice, last week's value of the same metric is the best covariate you will find¹.

What it does not fix

CUPED cannot help when there is no pre-experiment data — new users, new surfaces, a cold start. That is the case where stratification remains the only lever, and where the distributed-assignment bias has to be measured rather than assumed.

Choosing the test

There is no miracle test. The choice depends on skew, on metric type, and on whether you can afford the computation.

• t-test — simple, cheap in a distributed system, but loses control of the false positive rate under heavy skew.
• Mann–Whitney — robust to skew, expensive at scale.
• Bootstrap — flexible, the greediest of the four.
• Delta Method — the right default for ratio metrics.

Any language, coloured the same way

The pipeline is not tied to one grammar. Rust, SQL and a shell session sit side by side without ceremony.

fn cuped(y: &[f64], x: &[f64]) -> Vec<f64> {
    let theta = covariance(x, y) / variance(x);
    let mean = x.iter().sum::<f64>() / x.len() as f64;
    y.iter().zip(x).map(|(yi, xi)| yi - theta * (xi - mean)).collect()
}

-- The pre-period covariate, one row per user
SELECT user_id, SUM(clicks) AS clicks_before
FROM events
WHERE ts BETWEEN :start - INTERVAL '7 days' AND :start
GROUP BY user_id;

python -m experiments.cuped --metric ctr --covariate clicks_before --alpha 0.05