Causation

skills/choosing-causal-designs

choosing-causal-designs

Selects a causal design when assignment already happened and randomisation was never possible: interrupted time series for one unit and one switch date, difference-in-differences for treated and untreated units seen before and after, synthetic control for one treated unit and many donors, matching on observables, instrumental variables, and the exit that says there is no valid comparison group. Use when measuring the impact of something already rolled out, a launch, a migration, a pricing change, or a campaign that reached everyone at once.

shell

$ git clone https://github.com/0trm/gallop
$ cp -r gallop/skills/choosing-causal-designs .claude/skills/

Read the source on GitHub↗

Assignment already happened; the design's job is to reconstruct the comparison that randomisation would have provided. Pick by how assignment happened, not by which method is fashionable, and hand back an effect size plus the assumptions it rests on, stated in the readout, because here the assumptions are doing the work the coin flip would have done.

First, one check worth thirty seconds: is randomisation really unavailable? If the change has not fully shipped, a holdout or staggered rollout is still possible and strictly better. This skill is for the past tense.

The selection table

Work down; take the first row whose conditions hold.

Assignment looked like	Design	Needs
One unit (or everyone), one sharp product switch on a known date, no untreated twin	ITS – reference/its.md	A long, stable pre-period; no simultaneous shock; a discrete switch, not a diffuse push
Some units got it, some did not, both observed before and after	DiD – reference/did.md	Parallel pre-trends
One treated unit, many untreated candidates to blend	Synthetic control – reference/synthetic-control.md	A donor pool the treatment did not touch
Individuals selected themselves in, on things you can observe	Matching – reference/matching-iv.md	Selection on observables (a strong claim; say it)
Something nudged uptake without touching the outcome directly	IV – reference/matching-iv.md	A real instrument (rare; most candidates fail)
None of the above	No comparison group – reference/no-comparison-group.md	Honesty

The last row is a first-class exit, not a failure. Saying "there is no defensible number here" out loud beats shipping a number that dissolves under the first follow-up question, and the reference file says what to offer instead.

One row needs guarding against a tempting misread: a campaign or marketing push that ran everywhere at once is not an ITS candidate, even though "everyone, one date" appears to match. ITS needs a discrete switch in the product itself; a media flight is a diffuse, ramping shock whose effect shape is unknown, it usually arrives with seasonality and other pushes, and the pre-trend projection cannot separate any of that. That case takes the last row: bounds at most, a referral to whoever owns mix modeling or panel data, and a holdout designed into the next flight.

The second row has its own misread: two units, one treated, is still DiD. An Android switch with iOS untouched has a comparison group, and the twin is the counterfactual, not a falsification check bolted onto an ITS. With two clusters the standard errors are weak and the readout says so in words; that is a reason to state the inference honestly, never a reason to drop to a design that throws away the only comparison there is.

What every design here must state

The counterfactual, in one sentence. What would have happened without the change, and which data stands in for it. If the sentence cannot be written, the design is not chosen yet.
The identifying assumption, in the requester's language. "Madrid would have moved like Barcelona" (DiD), "the trend would have continued" (ITS), "adopters and matched non-adopters differ only on what we matched" (matching). This is the sentence the readout hangs on.
The falsification checks run and passed. Pre-trends for DiD, placebo dates for ITS, placebo units for synthetic control, balance tables for matching. A design whose checks were not run is an assertion, not an estimate.
What would change the answer. The known threat the design cannot exclude: the simultaneous marketing push, the seasonality no control captures, the unobservable that matching cannot see.

Discipline shared across all five

Pick the design before computing the estimate, from the assignment story alone. Running three designs and reporting the one with the nicest number is the observational version of peeking.
Effects here deserve wider error bars than their standard errors say. Standard errors price sampling noise, not assumption risk. Say which of the two dominates.
Modeling is the machinery, not the design. DiD is a regression; synthetic control is a weighted counterfactual model. The design decides what comparison is valid; the model just computes it. Adding controls to a regression is not, by itself, a design.
File the result through writing-readouts with design set honestly in the prior-store record (its, did, synthetic_control, matching, iv). Observational effects mixing into the experimental prior is exactly the kind of thing the field records to prevent.
The calibration habit: when a causal-design estimate later gets tested properly (a holdout on the next iteration), compare. A team that never checks its observational calls against experimental ones never learns which of its designs to trust.

reference did.md

Difference-in-differences

Some units got the change, some did not, and both are observed before and after. The counterfactual for the treated units is their own pre-period level plus the untreated units' change over time. The effect is (treated after − treated before) − (untreated after − untreated before).

When it fits

Assignment varied across units for reasons unrelated to the outcome's trajectory: a rollout by market, a policy that hit one country, some CRM waves getting the change earlier.
The identifying assumption: parallel trends – absent treatment, the treated units would have moved like the untreated ones. Untestable directly; the pre-period is the evidence.

How to run it honestly

Plot both groups' pre-trends first, before any regression. If they were not moving together before, the design is dead on arrival, and no fixed effect resurrects it. This plot goes in the readout.
The event-study version by default: estimate a per-period effect relative to the switch, not one pooled number. Pre-period coefficients near zero are the parallel-trends check made visible; the post-period path shows whether the effect grows, fades, or is a one-time level shift.
Cluster standard errors at the assignment unit (market, country, wave). With few clusters (under ~20, and certainly at 2) the asymptotic errors are fiction; the design still stands, the inference is what weakens. Use wild-cluster bootstrap or say plainly that inference is weak. Two markets is not enough units for credible DiD inference; treat that case as ITS with a control series.
Staggered adoption needs care: with units treated at different times, the classic two-way fixed-effects estimate mixes already-treated units into the control group and can even flip sign when effects vary over time. Use a modern staggered estimator (Callaway–Sant'Anna style, comparing each cohort to not-yet-treated units) rather than naive TWFE.
Falsification: a placebo outcome the treatment could not touch, and a placebo date in the pre-period.

What breaks it

Selection into treatment on the trajectory. If the markets chosen to get the feature first were chosen because they were trending up, the design measures the selection. Ask how the rollout order was actually decided; the answer is usually in a planning doc.
Spillovers: untreated units affected through shared supply or word-of-mouth make the control dirty and bias the effect toward zero (or beyond).
Composition shifts: if the change alters who shows up in each unit (new-user influx in treated markets), the units are not the same units after.

Stating the result

"Redemptions rose 2.3pp more in the provinces that got the feature than in those that did not (event study: flat pre-trends for 8 weeks, effect stable from week 2), assuming the provinces would have moved in parallel; rollout order was alphabetical, which supports it." Effect, check, assumption, and why the assumption is plausible in this instance.

reference its.md

Interrupted time series

One unit (often the whole product), one switch date, no untreated twin. The counterfactual is the pre-period's own trajectory, projected forward; the effect is the level (and possibly slope) break at the switch.

When it fits

The change hit everyone at once on a known date: a migration, a policy change, a redesign shipped to 100%.
The pre-period is long enough to establish trend and seasonality: as a rule of thumb, several full seasonal cycles of the metric, and never less than 3x the post-period you intend to read.
Nothing else big happened at the same date. This is the assumption that usually fails: launches ship with marketing pushes, and ITS attributes the sum to the switch.

How to run it honestly

Model the pre-period only: trend + seasonality (weekly at minimum; annual if the history supports it). Segmented regression is the transparent default; a forecasting model works too, but transparency is worth more than fit here.
Project across the switch and read the post-period gap: an immediate level change, a slope change, or both. Decide which of the two the mechanism predicts before looking.
Uncertainty from the projection, not the residuals alone. Autocorrelated series make naive standard errors far too small; use Newey-West errors or a forecast interval that widens with horizon.
Falsification, non-negotiable:
- Placebo dates: run the same analysis at several pre-period dates where nothing happened; the effect there should be ~zero, and its spread is an honest floor on the real one's uncertainty.
- A control series if any exists, even an imperfect one (another market, a metric the change could not touch): if it jumps at the same date, the jump is the environment, not the change.
Read a fixed post-window, declared before looking, matched to the mechanism's timescale. Extending the window until significance is peeking with extra steps.

What breaks it

Simultaneous shocks (the marketing push, a price change, a competitor event, a pandemic). ITS cannot separate them; say which ones are in the window.
Slow ramps. A rollout over six weeks has no interruption; either model the ramp explicitly against exposure share or use the staggered rollout as a DiD instead. A staggered rollout is better data than a clean switch; check for one before settling on ITS.
Metric drift: a definition change near the switch date is indistinguishable from an effect. Check the registry's version history first.

Stating the result

"Deliveries ran 4.1% above the pre-trend projection for the eight weeks after the migration (placebo dates: −0.5% to +0.9%), assuming the pre-period trend would have continued and noting the June price change sits inside the window." Effect, falsification range, assumption, known threat, one sentence each.

reference matching-iv.md

Matching, and instrumental variables

The two designs for self-selection, in descending order of how often they apply. Both answer "did adopting X cause Y" when individuals chose X themselves; both rest on assumptions stronger than the other designs in this skill, and the readout must say so.

Matching (and its regression cousins)

The claim being made: adopters and non-adopters differ only on things you observed and matched. Selection on observables. Say the claim out loud before running anything, because it is a strong one: the reason someone adopted is usually the hardest thing to observe (motivation, intent, need).

When it is defensible:

The selection mechanism is substantially captured by the observables: adoption driven by tenure, platform, plan, geography, prior usage, and you have all of those.
Rich pre-treatment behaviour exists. Pre-period activity is the best proxy for the unobservables, for the same reason CUPED works: past behaviour absorbs stable individual differences.

How to run it honestly:

Match or weight on pre-treatment covariates only (propensity scores, nearest-neighbour, or plain stratification; the choice matters less than the covariate set).
Report balance, before and after matching, on every covariate. An unbalanced match is not a design.
Trim to common support: units with no counterpart on the other side get dropped, and the readout says which population remains.
Sensitivity analysis: how strong would an unobserved confounder need to be to erase the effect? If a confounder as strong as the best observed covariate would kill it, the result is fragile, and that sentence belongs in the readout.
Compare against the naive gap. If matching barely moves the raw difference, either selection is weak (good, argue it) or the observables miss the selection entirely (likely, admit it).

The honest framing: matching produces "the difference not explained by what we could measure", which is an upper bound on the causal effect whenever the unmeasured selection points the usual direction (the motivated adopt). Say "upper bound" when that is what it is.

Instrumental variables

The shape: something (the instrument) shifted uptake of the treatment without touching the outcome through any other path. Effect = the outcome shift attributable to the instrument, scaled by the uptake shift.

Real instruments in product work are rare and mostly man-made:

A randomised encouragement: the prompt experiment from designing-experiments (randomise the nudge, instrument adoption with the nudge). The one instrument you can always manufacture, and the reason the trap case routes there.
A staggered rollout's timing, where order was operationally arbitrary.
An outage or eligibility quirk that blocked uptake for some users for reasons unrelated to them.

The two conditions, both mandatory:

Relevance – the instrument actually moves uptake, testably: report the first stage. A weak first stage (F < 10 as the classic screen) makes everything downstream noise amplified.
Exclusion – the instrument touches the outcome only through uptake. Untestable, and where candidates die: the prompt that annoys users touched the outcome twice.

Interpretation: IV estimates the effect on compliers, the units whose uptake the instrument changed, not on everyone. For a prompt instrument that is exactly the population the decision is about, which is why encouragement designs are the clean case.

Choosing between them

Ask what drives selection. Observable and measured → matching, with the sensitivity analysis. Unobservable → IV if a real instrument exists, otherwise this question has no comparison group yet; see no-comparison-group.md, and remember the encouragement design converts "no instrument" into "instrument next sprint".

reference no-comparison-group.md

No comparison group

The exit that says so. It fires when every row of the selection table has failed: everyone got the change at once, no untreated twin exists anywhere, the pre-period is too short or too shocked to project, selection is on unobservables, and there is no instrument. The canonical case: the campaign that ran across every market simultaneously with nothing held back, asked about after the fact.

Why shipping a number anyway is the worst option

Whatever number gets produced under these conditions is a before/after difference wearing a method's name. It will be repeated without the caveats, become the campaign's official value, justify the next three campaigns, and be indistinguishable from seasonality, mix shift, and whatever else moved that month. The refusal costs one uncomfortable conversation; the number costs a planning cycle. This exit exists because "I would rather say that out loud than ship the number anyway" is a complete professional answer.

What to say

Three sentences, in order:

The fact. "The change reached everyone at once with nothing held back, so no data we hold can separate its effect from everything else that moved."
The referral. Name where the question can be partially answered, if anywhere: the media team's mix model, panel data, the retailer's numbers. Routing the question to whoever holds the right data is a better answer than a number you cannot defend.
The offer. "Bring me the next one while it is still a draft and we hold something back." Concretely: a geo holdout, a staggered rollout, a randomised encouragement, a 5% control. The refusal converts into design leverage on the next iteration, which is the entire point of making it politely and in writing.

What can still be said honestly

Without a comparison group there is still descriptive work with integrity, as long as it is labelled description:

The metric's trajectory annotated with the event date, against its own seasonal history, with no effect claimed.
Bounding: "if the entire month-over-month change were the campaign, the ceiling is X; the floor is indistinguishable from zero." Bounds are honest where point estimates are not.
A forward monitoring plan: what would have to stay elevated, for how long, to be even consistent with an effect.

File the refusal

Write the routing and the reason into the knowledge repo via writing-readouts (decision: no-measurement, with the offer recorded). Eight months later, when the question returns, the entry is the difference between re-litigating the refusal and pointing at it, and the count of refusals per quarter is itself the argument for the holdout policy.

reference synthetic-control.md

Synthetic control

One treated unit, many untreated candidates. Build the counterfactual as a weighted blend of donor units chosen so the blend tracks the treated unit's pre-period closely; the effect is the post-period gap between the unit and its synthetic twin.

When it fits

Exactly one (or very few) treated units: one country got the launch, one city got the policy.
A donor pool of genuinely untreated units, plausibly driven by the same forces: other countries, other cities, other categories.
A long pre-period over which a good fit can be established, and enough donors that the blend is not one unit wearing a hat (rule of thumb: 10+ donors, 2+ years of pre-period for weekly data, less for high-frequency metrics).

Two units total is not a synthetic control; that is a comparison with one donor, and it should be called DiD-with-one-control and given its weak inference honestly.

How to run it honestly

Freeze the donor pool first, excluding any unit the treatment could have touched (spillovers make donors dirty and the effect biased toward zero, or away from it if donors absorbed displaced demand).
Fit weights on the pre-period only, matching the outcome path (and optionally a few covariates). Report the pre-period fit; a synthetic twin that cannot track the past has no claim on the future. Weights are non-negative and sum to one, which keeps the twin an interpolation rather than an extrapolation, and makes it inspectable: name the donors and their weights in the readout.
Inference by placebo: run the same procedure on every donor as if it were treated. The treated unit's gap is credible only if it is extreme in that distribution (the permutation p-value is the honest one; with 20 donors, the best achievable is 1/21).
Backdate placebo: fit the twin on the first half of the pre-period, check it tracks the second half. A twin that fails backdating is overfit to noise.

What breaks it

A shocked donor: one donor gets its own event in the post-period and drags the twin. Inspect the twin's components over time.
Interpolation bias: the treated unit sits outside the donor pool's range on a dimension that matters; the blend then matches the path without matching the mechanism.
Short pre-periods: with little history, many weight vectors fit equally well and the choice among them is arbitrary; report the sensitivity across them.

Stating the result

"Sell-out in the launch country ran 6% above its synthetic twin (0.4 Germany, 0.35 Netherlands, 0.25 Sweden) for the twelve post-launch weeks; in placebo runs on the 14 donors, the largest gap was 2.8%, so the effect is outside the noise distribution. Assumes no launch spillover into donor markets." Twin, gap, placebo distribution, assumption.