Causation

skills/designing-experiments

designing-experiments

Designs an A/B test around the four choices that cannot be repaired after launch: the randomisation unit and whether units interfere, the minimum detectable effect sized from prior experiments rather than from a wish, exposure logging rather than eligibility logging, and a decision rule written before traffic starts. Use when planning, powering, or pre-registering an experiment, when deciding whether a question is testable at the available traffic, or when a feature is about to ship without a flag.

shell
$ git clone https://github.com/0trm/gallop
$ cp -r gallop/skills/designing-experiments .claude/skills/
Read the source on GitHub↗

Ships with

Four choices, none of which a bigger sample can fix afterwards. The output is an analysis plan attached to the ticket before the flag exists: unit, MDE, duration, guardrails, logging spec, and a decision rule. No plan, no flag.

Precondition: the primary metric comes from the registry with trusted status (python -m gallop.priors registry --registry metrics.jsonl --status trusted). If it is not there, this question belongs to defining-metrics first. And the hypothesis has a mechanism: if we change X, metric Y moves because Z. No mechanism, no test.

Choice 1 · The randomisation unit, and interference

Pick the unit the treatment actually touches, then check whether treating one unit changes another unit's experience. The full decision guides are reference/randomisation-unit.md and reference/interference.md; the short form:

  • Default to the user (stable ID, consistent across sessions and devices). Randomising sessions or pageviews when the metric is per-user breaks the analysis before it starts.
  • If units share supply, a feed, a market, an inventory, or each other's attention, the control group gets treated through the back door and the effect reads larger than it is. That is switchback or cluster randomisation, decided now or never. No sample size fixes interference.
  • One live test per surface, in its own layer. Splitting the same scarce audience across concurrent tests buys the same answers with wider intervals plus an interaction risk nobody will notice.

Choice 2 · The MDE, from the prior store

The minimum detectable effect is a fact about traffic and history, not an ambition. Compute it before anyone commits to the test:

python skills/designing-experiments/scripts/size_test.py \
  --store priors.jsonl --metric activation_rate \
  --baseline-rate 0.12 --units-per-day 8000

The script reads what this metric has actually moved by across past tests, computes the MDE at the available traffic (the arithmetic is 2.8 × sqrt(2p(1-p)/n) at the usual thresholds), and returns a verdict: fundable, or not at this traffic. Without a store, use python -m gallop.power mde and say out loud that the target is unanchored.

Honest priors are humbling: most experiments move nothing, so the prior mean is roughly zero and typical true effects are a fraction of what gets proposed. If the MDE at your traffic is a 16% relative lift, a copy tweak is not a hypothesis, it is a hope. The design responses, in order of value:

  1. Test a bigger swing. Propose changes large enough to plausibly clear the MDE. Saying which those are, before design work starts, is the job.
  2. Spend effort on variance, not on more traffic. A pre-period covariate for CUPED (plan the covariate now; it must be fully determined before first exposure), a primary metric with a higher base rate, denominators you control, targeting the segment where the effect should be largest.
  3. Move the primary metric closer to the change. The nearer metric is powered; keep the deeper one as a directional guardrail and say in the plan that it is underpowered.
  4. Decide without a test, and write that down. The most valuable sentence in the quarter: this question is not answerable at our traffic in under four months, so make the change bigger, pick a nearer metric, or decide it untested and record that we did.

Duration falls out of the same arithmetic (size_test.py prints it). A test needing seven weeks will contain a holiday or a season; randomisation handles it, interpretation does not. An effect measured across Christmas is an effect measured across Christmas; the plan should say so.

Choice 3 · Log exposure, not eligibility

The most common bug in the chain and the one that survives longest. If the log records who was bucketed rather than who actually saw the variant, the estimate is diluted toward zero and nothing downstream flags it, because the test looks exactly like a test that ran.

  • Define the exposure event now: the moment a unit actually experiences the treatment surface, fired identically in both arms (control fires it on seeing the control surface).
  • Give the data engineers a specification, not a request: event name, parameters, when it fires, and the query the readout will run. "We will add tracking later" means the first weeks of data do not exist and a six-week test now needs eight.
  • Ask for alerting on event volume. The two failures that ruin tests from below are a sample ratio mismatch and an event that stops firing on one platform; both are visible in volume within a day if anyone is looking.

Choice 4 · The decision rule, before launch

Written on the ticket before traffic starts, in the form: if the interval excludes X, we ship; otherwise we do not. Plus:

  • Guardrails attached up front: crashes, latency, refunds, unsubscribes, and the gaming guardrail from the metric's registry entry. A guardrail breach means rollback, and a rollback is the system working.
  • Peeking policy decided now. If anyone will look before the horizon (they will; the platform shows everyone a curve), the test is sequential from the start: python -m gallop.sequential bound for always-valid monitoring or obf for a fixed look schedule. Deciding this after looking is not available.
  • One pre-registered segment, chosen from the mechanism, not eleven chosen from the readout.
  • The full pre-registration template is reference/pre-registration.md.

The design review, thirty seconds

Before the flag flips, the plan states: unit and interference call; MDE, its source, and duration; exposure event and who owns its alerting; decision rule, guardrails, peeking policy, segment. If any line is blank, the test is not designed yet. If all are filled, file the plan on the ticket; reading-experiments will hold the readout to exactly this plan, and writing-readouts will file what it taught.

reference interference.md

Interference

The assumption every standard A/B analysis makes without saying so: one unit's assignment does not change another unit's outcome (SUTVA). When it fails, the control group is treated through the back door, the comparison is between two treated groups of different intensity, and the estimate is biased in a direction you can sometimes predict and often cannot. No sample size fixes it. It is decided at design time or not at all.

Where it lives

Shared supply. Marketplaces, delivery, inventory, ad budgets. Pay one courier more and they take the jobs a control courier would have taken; the treatment effect reads as the sum of a real effect and a transfer. The canonical false win.

Feeds and social surfaces. Treated users post more, everyone's feed changes, control users respond to treated content. Effects leak both ways.

Ranking and recommendation. One model variant's clicks train the shared model, or the variants compete for the same slate positions.

Word of mouth and shared devices. Features visible between users (collaboration, gifting, referrals) recruit the control arm into treatment.

Capacity. Anything that saturates: support queues, fraud review, promo budgets. Treatment consumes shared capacity; control experiences the shortage.

The design responses

  1. Switchback. Flip the entire system between arms on a randomised schedule (hours or days). Handles supply-side interference directly. Costs: temporal correlation (analyse at slice level), carry-over (leave burn-in gaps), and day-of-week confounds (balance the schedule).
  2. Cluster randomisation. Assign whole markets, cities, or social clusters, chosen so interference is contained within a cluster. Costs: effective n = number of clusters; needs many clusters to power.
  3. Budget-split / slate-split designs. For ads and ranking: split the constraint itself (separate budgets, interleaved slates) so arms stop competing for one pool.
  4. Accept and bound. If interference is plausibly small (a mostly single-player surface with a weak social edge), run user-level anyway, say so in the plan, and treat the estimate as an upper bound in the direction the leak inflates it. Written down before launch, this is a legitimate call; discovered afterwards, it is an excuse.

The design-time question

Ask it in one sentence at the review: if we treat half the users, does the other half's experience change through any shared resource? Name the resource or name "none". If a resource is named, the design is switchback or cluster, and the duration arithmetic must be redone at the new unit, which usually changes the answer to "is this testable at our traffic".

reference pre-registration.md

Pre-registration

The analysis plan, written before the flag flips, attached to the ticket that launches the test. Its purpose is to make the readout a comparison against commitments rather than a negotiation with hindsight. Everything in it is short; all of it is binding.

The template

EXPERIMENT        <name, ticket link, flag name>
MECHANISM         if we change <X>, <metric> moves because <Z>
UNIT              randomised on <unit>; analysis aggregates to <unit>
INTERFERENCE      shared resource: <named, or "none">; design: <user | cluster | switchback>
ARMS & SPLIT      <arms and intended ratio>
PRIMARY METRIC    <registry name, trusted status confirmed on YYYY-MM-DD>
MDE               <value> (<source: prior store n=<k> | unanchored>)
POWER / ALPHA     0.80 / 0.05 unless stated; sequential: <none | always-valid | OBF k looks>
DURATION          <days>, from <units/day>; calendar events in window: <named>
EXPOSURE EVENT    <event name>; fires when <moment>; both arms; volume alert owned by <who>
GUARDRAILS        <crashes, latency, refunds, unsubscribes, + gaming guardrail from registry>
SEGMENT           one, pre-registered: <segment, chosen from the mechanism>
DECISION RULE     if the interval excludes <X>, ship; else <kill | iterate>
                  guardrail breach ⇒ rollback, readout still written
READOUT DATE      <the Tuesday it will be read, in the room>

The rules the template encodes

The decision rule is a number, not a mood. "If the interval excludes a 0.2pp lift we ship" survives contact with an ambiguous result; "significant and meaningful" does not. Write the rule so that both possible readouts lead to a named action.

The peeking policy is part of the design. If the platform will show anyone a curve before the horizon, the test is sequential from day one (always-valid bounds if looks are unscheduled, group-sequential if they are fixed), and the plan says which. The social half of the rule matters as much: nothing counts as a result until the scheduled readout, whatever the curve does on day three.

One segment. Chosen because the mechanism predicts a difference there, not because the platform offers eleven cuts. Anything found in an unregistered segment is a hypothesis for the next test, never a result of this one.

Guardrails are read before the primary. The order is part of the plan: trust gate, guardrails, then the primary metric. A win on the primary with a guardrail breach is a rollback, not a negotiation.

Amendments are dated. Reality intervenes: traffic shifts, a bug forces a restart. Amend the plan in writing, with the date and the reason, before the readout. An amended plan is honest; a retrofitted one is fiction.

Why this is worth a page of ceremony

Five of the six steps of an experiment can hand back a wrong number without the readout looking any different. Every line above pins one of those steps to a commitment made while nobody knew the answer. Written after the numbers are visible, the same lines are just the story of what happened; the entire value is in the timestamp.

reference randomisation-unit.md

Choosing the randomisation unit

The unit is the thing the coin flip assigns. It fixes what the analysis may treat as independent, so it is the first choice and the least repairable.

The rule

Randomise on the unit that (a) experiences the treatment consistently and (b) the primary metric is defined on. When those two disagree, the analysis must aggregate to the randomisation unit, and the design must accept the power cost of doing so.

The options

User (stable ID). The default. Survives sessions and devices if the ID does; check the identity join before trusting it. Anonymous or logged-out traffic randomised on cookies leaks across devices and clears; if the surface is pre-login, accept the dilution and note it, or move the test post-login.

Session or pageview. Only when the treatment is genuinely stateless (a ranking tweak, a latency change) and the metric is per-session. Never when the user can notice the interface changing between visits: inconsistency is itself a treatment, and not the one being tested.

Cluster: city, market, store, team. When the treatment cannot be held to individuals (pricing a market, a courier bonus, anything word-of-mouth) or when interference forces it (see interference.md). The effective sample size is closer to the number of clusters than the number of users; a two-country product has two clusters and therefore, for most designs, none to spare.

Time slice (switchback). The whole system flips between treatment and control on a schedule. The standard answer for marketplaces and anything with shared supply. Randomise the schedule, analyse at the time-slice level, leave burn-in gaps between flips so carry-over from one period does not contaminate the next.

The mismatch that produces false positives

Randomise on users, analyse on sessions, and every user contributes several correlated rows that the test treats as independent. The standard errors shrink, the false positive rate multiplies, and nothing looks wrong in the readout. Under a true null this can reject several times the nominal rate.

The fix is mechanical: aggregate to the randomisation unit before testing (one row per user), or use a variance estimate that respects the clustering (delta method or a bootstrap over units). What is not available is reading the session-level t-test and hoping.

Ratio metrics hide the same trap: sessions-with-conversion / sessions is a per-session metric under user randomisation. Aggregate to per-user rates, or delta-method it. The naive interval can be roughly half the honest width.

Practical checks

  • Write the unit into the analysis plan as a sentence: "randomised on user_id from the assignment service; analysis aggregates to user."
  • Confirm assignment is sticky: the same unit gets the same arm on every visit, across the whole test. Re-bucketing mid-test is an SRM in the making and an experience bug besides.
  • Confirm the unit exists before exposure. Assigning at signup a treatment that changes the signup page is assigning downstream of treatment, and the arms are no longer comparable populations.