intake

Run this on
every question

One lookup, three questions and one gate, asked before you open a query editor. Each exit names what you hand back.

algorithm

Theory Layer knowledge repo · what was already tried on this surface, wins and losses alike prior store · what this metric has actually moved by, so the MDE is a number read before the question is scoped, written when the decision is made settled, stale, or new? and what is the prior? a question arrives do I trust the metric and the definition of success? is this description, or is it about a change? is the decision made once, or continuously? who assigned the treatment? Measurement Framework fix the definition, the source of truth, the registry hands back: a number the rest of the map can stand on Exploratory Analytics deep dives · funnels · segmentation · opp sizing hands back: a hypothesis Statistical Modeling forecast · optimizer · uplift model hands back: a ranking, an allocation, a prediction Experimentation A/B · switchback · power · sequential · HTE hands back: an effect size Causal Inference match the method to how assignment happened hands back: an effect size, plus its assumptions Decision ship · kill · reallocate yes a change once one ship-or-kill call no description what · where · how many · who continuously per user · per day · at volume you can, or could have it already happened effect size effect size the model itself still needs an experiment to prove it moves anything a hypothesis is a change question, so it re-enters here what ships changes the data, so the next question enters a different world what we learned

it already happened

keyed to causal inference

Pick the method by how assignment happened

ShapeMethodUse it when
ITSone unit, one switch date, no untreated twin
DiDtreated and untreated units, both seen before and after
Synthetic
control		one treated unit, many untreated ones to blend
Matchinggroups differ on things you can actually observe
IVsomething shifted uptake without touching the outcome

If none of these fit, you do not have a comparison group. Say that out loud rather than shipping the number anyway.

the diamond

keyed to experimentation

Before it ships, it is a choice

Once it has launched, can you randomise? is a fact about the past. Asked early, it is a design decision you still own:

  • hold out a slice of users
  • stagger the rollout by market or by cohort
  • randomise the prompt, not the feature