reading-experiments

Empirical Bayes shrinkage

The winner's-curse correction, and the moment the prior store pays for itself.

Why raw winners are inflated

A result is reported because it crossed a threshold. Conditioning on crossing selects the realisations noise pushed up, so the expected reported effect exceeds the true one, and the excess grows as power falls. An underpowered test that "wins big" is the most inflated object in the building: with 20% power, a just-significant result overstates the true effect by roughly a factor of two or more. The sign is usually right; the magnitude is not, which is worse for planning than a clean false positive.

The model

Normal-normal. The prior is what this metric has actually done, estimated from the store by method of moments:

mu   = mean of past effects
tau² = max( var(past effects) − mean(past se²), 0 )

Posterior mean = w·effect + (1−w)·mu, with w = tau²/(tau² + se²).

python -m gallop.shrink eb --effect 0.0031 --se 0.0012 \
    --store priors.jsonl --metric activation_rate --unit pp

Reading the output:

• w near 1: the test was precise relative to how much true effects vary; the data mostly stands.
• w near 0: the test was noisy relative to the metric's history; the estimate collapses toward the prior mean. This is not the method being timid, it is the observation carrying little information.
• tau² = 0: past effects are indistinguishable from noise around their mean; the model is saying this metric has never produced a distinguishable effect, and the new result shrinks all the way to mu. That verdict is itself worth surfacing in the readout.

The shrunk number is the planning number: what to forecast from, what to write into the prior store, what to tell the roadmap. The raw number and interval are still reported beside it, labelled.

Preconditions, checked by the tooling

• At least 3 past effects for the metric (gallop.shrink refuses fewer; below ~8 the prior itself is noisy, so say so).
• One unit. Effects in pp do not pool with effects in relative; from_store refuses mixed units rather than averaging them.
• Comparable readouts. The store should hold effects from the same metric under comparable designs; a store polluted with one 10x outlier from a pricing change will inflate tau² and under-shrink. supersedes exists for corrections; use it.

When there is no store

First tests at a company have no prior. Options, in honesty order: borrow the published base rate (most experiments do nothing; the honest prior mean is roughly zero, so at minimum discount the raw estimate mentally and say the readout is unshrunk); or start the store with this readout and accept that shrinkage begins from test four. What is not available is treating the raw number as a forecast: that is the exact error the correction exists for.

The trust gate

Two mechanical checks that run before any effect is read. Both exist because their failure modes produce readouts indistinguishable from healthy ones.

Sample ratio mismatch

The check: chi-square goodness of fit of assignment counts against the intended split.

python -m gallop.trust srm --counts counts.csv        # columns: arm, assigned

Alpha is 0.001, not 0.05, deliberately. The check runs on every experiment ever analysed; at 0.05 it would fire on one healthy test in twenty, be ignored within a quarter, and protect nothing. At 0.001 a firing means the test is broken, with high confidence.

Why there is no correction. An SRM means units left (or never entered) one arm non-randomly. The remaining populations differ in whatever made those units leave, which is usually correlated with the outcome. Reweighting cannot recover a comparison that randomisation no longer underwrites.

The cause taxonomy (Fabijan et al.), worked in order, stopping at the first that fits:

1. Assignment – bucketing bug, hash collision, a variant filter upstream.
2. Execution – one variant errors or times out; its events never arrive.
3. Logging – telemetry differs by variant; a slower page fires fewer beacons.
4. Experiment definition – the ramp changed mid-flight and the expected ratio was not updated. (The one benign cause: fix the expectation, not the test.)
5. Triggering – the trigger condition is itself affected by treatment.
6. Interference – units move between variants or share state.
7. Filtering – a bot or outlier filter removes units at different rates by variant.

Check daily during the run, not once at the end: an SRM caught on day two costs two days; caught at readout it costs the whole test.

Exposure versus eligibility

The check: exposure counts per arm against assignment counts.

python -m gallop.trust exposure --counts counts.csv   # columns: arm, assigned, exposed

Two distinct failures:

Differential exposure (the exposed counts fail an SRM at the assignment split): the arms trigger unequally, which is a bug in the trigger or the surface, and the comparison is invalid outright. Treat exactly like an SRM.

Uniform dilution (both arms under-exposed at the same rate r): the intention-to-treat estimate is attenuated by roughly r. Two honest readings:

• ITT as measured, labelled "diluted by exposure at r".
• The exposed-only comparison, if exposure is logged symmetrically in both arms (control fires the event on seeing the control surface). If only the treatment arm logs exposure, the exposed subsets are not comparable and scaling ITT by 1/r is the safer statement.

The deeper fix belongs to design: log exposure, not eligibility, and alert on event volume so a dark platform or country is visible within a day. A test that "ran fine" with a 60% exposure rate was a test on a 40% smaller effect than anyone thought they were testing.

Variance reduction at readout

CUPED

Deng, Xu, Kohavi & Walker (2013). Adjust the metric by a pre-experiment covariate that treatment cannot have touched:

Y_adj = Y - theta (X - mean(X)),   theta = Cov(Y,X) / Var(X)

Var(Y_adj) = Var(Y)(1 - rho²): the entire benefit is the squared pre/post correlation, so measure rho before promising anyone a number. At rho 0.7 the required sample halves; at rho 0.2 the gain is cosmetic.

python -m gallop.variance cuped --data units.csv --y metric --x pre_metric --arm arm

The validity rule, in one sentence: the covariate must be fully determined before the first unit was exposed, and defined identically for every unit, including units with no history (impute a constant, usually the mean; never drop them).

Failure modes, in order of danger:

1. Contaminated covariate. X computed over a window that overlaps the experiment, or otherwise touched by treatment. The only failure that biases the estimate rather than weakening it: CUPED then subtracts part of the true effect. If the covariate window's end is not provably before first exposure, do not use it.
2. Per-arm theta. Theta and mean(X) must be estimated pooled across arms. Estimated per arm, each theta fits its own arm's noise and most of the variance reduction is thrown away (gallop.variance pools; a hand-rolled version often does not).
3. Missing pre-period. New users have no history. Coverage c of the covariate cuts the reduction by roughly c²; with 40% coverage, expect ~16% of the nominal gain. Report the effective rho, not the rho among covered users.

The natural covariate is the same metric over an equal-length pre-period. Anything pre-treatment and correlated works; the same metric is usually the most correlated thing available.

The alternatives

Post-stratification on a discrete covariate (platform, country, tenure): weight each stratum by its pooled share. Wins over CUPED only when the covariate is genuinely categorical or must be fixed before launch.

Triggered analysis: restrict to units that could have been affected at all. Not a variance trick on the same population; it changes the estimand to the effect on the triggered subset. Legitimate and often what the decision wants, but say which population the number is about.

What is not on the list: collecting more traffic. At readout the sample is what it is; the levers above are the only ones left, which is why the covariate is planned at design time.