field guide

The method map

Product data science on one page. A question enters at the left and leaves as a decision. Measurement is a foundation, not a phase, because what ships changes the data; theory is a ceiling, not a report, because what you learn has to outlive the test that produced it.

1 · the map

causal study design split by who assigns treatment Exploratory Analytics deep dives · funnels segmentation · retention root cause · opp sizing what is happening? can you randomise? Experimentation you control assignment A/B · switchback · power sequential tests · HTE Causal Inference assignment already happened synthetic control · DiD matching · IV · ITS Statistical Modeling regression · forecasting simulation · optimization ML · uplift models Decision ship · kill · reallocate hypothesis yes no effect size effect size supplies estimators automates the decision Theory Layer knowledge repo · one searchable entry per question · losses written up like wins prior store · per-metric effect distribution · what a realistic MDE actually is written by every decision, read before the next question what we learned what we already believe what ships changes the data priors: sizing and shrinkage Measurement Framework north-star & core metrics · event instrumentation source-of-truth reporting · the definition of success consumed by every box above 1 2 3 4 5 6 7 8

2 · the legend

The eight objects, and what each one produces

The map has six labelled objects, one decision point, two kinds of arrow, and two bands. Read left to right and you are following the life of a single question. Read top to bottom and you are moving down a stack: a memory runs across the top, the main path sits in the middle, a support layer sits under it, and a foundation sits under that.

  1. 1

    Exploratory analytics

    Deep dives, funnels, segmentation, retention, root cause, opportunity sizing. The box states its own question: what is happening?

    producesA hypothesis worth testing.

  2. 2

    Can you randomise?

    The only diamond on the map, so the only true fork. Everything upstream is one path; everything downstream splits on this answer.

    producesA routing decision: yes or no.

  3. 3

    Experimentation

    You control assignment. A/B, switchback, power, sequential tests, HTE.

    producesAn effect size.

  4. 4

    Causal inference

    Assignment already happened. Synthetic control, DiD, matching, IV, ITS.

    producesAn effect size.

  5. 5

    Statistical modeling

    Forecasts, rankings, allocations, and the estimators the causal designs run on: below the main path, not on it.

    producesEstimators, and automated decisions.

  6. 6

    Decision

    Ship, kill, reallocate. An ellipse because it is a terminal state, not another activity.

    producesA change to the product, and new data.

  7. 7

    Theory layer

    Two halves. The knowledge repo is prose: one entry per question, holding the design, the number, the decision and what you would do differently. The prior store is a table: for each metric, the distribution of effects the last hundred tests produced. Not a stage; a memory, and the only object on the map that gets more valuable the longer a team runs it.

    producesA prior, a realistic MDE, and a question you do not have to ask again.

  8. 8

    Measurement framework

    The band underneath everything: north-star and core metrics, event instrumentation, the source of truth, the definition of success.

    producesA number the rest of the map can stand on.

3 · inside the box

What is inside the experimentation box

The map draws experimentation as one rectangle, which is honest at two tests a quarter and misleading at two hundred. The rectangle contains six steps, and five of them have a way of handing back a wrong number without the readout looking any different. An experimentation platform is the thing that makes those five steps configuration instead of craft, and the reason mature teams describe themselves as running a learning system rather than running tests.

Theory Layer prior store · what this metric has actually done, over the last hundred tests knowledge repo · what was already tried, and what it taught what the platform automates Idea Problem · Mechanism Metric · Prior mechanism, not a guess Design MDE · duration power · guardrails sized from the priors Assignment flags · randomiser holdouts · layers one test per surface Logging exposure events did this user see B? the most common bug Analysis CUPED · SRM · CATE sequential p-values shrunk toward the prior Decision ship · kill guardrail rollback a rollback is a result prior · MDE shrinkage what it taught
The chain runs left to right. The theory layer feeds it twice and is written by it once, which is the whole difference between a team that runs tests and a team that accumulates. The dashed container marks what a mature platform does for you: sizing, assignment, logging and analysis are configuration rather than judgment, and the two steps outside it are the two nobody can automate.

Where each step breaks

Each of the six steps has its own way of handing back a wrong number, and the corrections are the skills, not the map. designing-experiments covers the four choices that cannot be repaired after launch, with the MDE taken from the prior store. reading-experiments covers whether the result is a result: SRM, exposure, the sequential bound, CUPED, shrinkage.

4 · day to day

Using the map day to day

Search the theory layer before you scope anything. Either the question is settled and you hand back the entry, or the prior store gives you the effect size this metric has historically moved by, which is what turns the MDE from a wish into a number. The lookup costs a minute and the test costs a sprint. Then route the question, and underneath all of it ask whether you trust the metric at all: if you do not, the measurement framework is the actual work.

Worked routings

The question as it arrivesRoutes toWhy, and what you get back
"Mobile checkout conversion dropped four points last month. What happened?"Exploratory analyticsPure description, no intervention proposed. Funnel and segmentation work until the drop localises to a step, a platform, a cohort. Output is a hypothesis, which then enters the diamond.
"Does the new one-page checkout increase completion?"ExperimentationYou control who sees which checkout, so randomise. Power the test before you launch it, not after. Output is an effect size with a confidence interval, and the decision is ship or kill.
"We rolled the new pricing page out to all users in March. Did it help?"Causal inferenceAssignment already happened and it was not random; everyone got it at once. That is an interrupted time series, or DiD against a comparable unaffected market. Output is still an effect size, with assumptions you must state.
"Users who enable notifications retain better. Should we push everyone to enable them?"ExperimentationThe trap case. The observed gap is self-selection, not effect: engaged users enable notifications. Randomise the prompt, not the outcome. If you truly cannot, matching or IV in the causal inference box, with the bias acknowledged.
"Which of our twelve million users should get the win-back discount this week?"Statistical modelingA decision made repeatedly at volume, so it belongs to the automation arrow. An uplift model ranks who is moved by the discount, not who is likely to return anyway. Validate the model itself with an experiment.
"Does the new courier bonus increase deliveries completed?"Experimentation, redesignedRandomising couriers is the obvious move and it is wrong: pay one courier more and they take the jobs a control courier would have taken, so the control group is treated through the back door and the effect looks larger than it is. Randomise time or city rather than people. Switchback or cluster, decided before launch, because no sample size fixes interference.
"Didn't we try something like this two years ago?"Theory layerA lookup, not an analysis. The entry says what was tested, what the effect was, and what has changed since. If it predates the redesign it is a prior rather than an answer: it sharpens the next test, sets a realistic expectation, and stops the team arguing from memory.
"Our activation number differs in two dashboards and nobody trusts either."Measurement frameworkNot a stage on the path, a foundation crack. Fix the definition of success and the source of truth first; every analysis built on top of it inherits the error invisibly.