sizing-opportunities

Decomposition

Two identities that turn "the metric moved" into "this is what moved it". Both are exact, so the shares they produce sum to the change and nothing is left to interpretation.

Mix versus rate

A rate over a population is a weighted average of segment rates:

R = Σ s_i · r_i        s_i = share of units in segment i, r_i = its rate

Between two periods it changes because rates changed or shares changed:

ΔR = Σ (r_i' − r_i) · (s_i + s_i')/2               the rate effect
   + Σ (s_i' − s_i) · ((r_i + r_i')/2 − R̄)         the mix effect

with R̄ the average of the two overall rates. gallop.explore.mix_rate computes both from segment counts, and the two sum exactly to the change. The centring on R̄ costs nothing in the sum (share changes sum to zero) and makes each segment's term readable: a growing segment that converts below the whole drags the total down, a shrinking one that converts above it drags too. Each segment contributes a term to each effect, and those terms are the ranking that matters: a segment's contribution to the total, not the size of its own swing.

Reading the output.

• Rate effect dominates. Segments got better or worse. The surface, the flow, the product is where to look next, and the funnel step decomposition says where.
• Mix effect dominates. The population changed. Acquisition, a campaign, a channel shift, a market launch, a bot wave. No segment did it; the traffic did. The hypothesis is about who arrives, and the owner is upstream of the surface.
• Simpson's case. Every segment improved and the total fell. This is a pure mix move and the function flags it by name, because it is the case most often reported backwards.

One cut at a time. A decomposition by channel and another by platform each explain the whole change on their own terms; they are two views, not two halves. A shift in one dimension shows up in every other dimension's view as a rate effect, spread across the cells: when paid social's share triples, every platform cell now carries more paid-social traffic and every platform cell's rate falls by about the same amount. The tell is uniformity. A rate effect that is the same size in every cell of a view is a mix shift in some other dimension, and the view that shows it as a mix effect is the one that found the cause.

Funnel steps

Overall conversion is a product of step rates:

C = n_K / n_1 = Π r_k        r_k = n_k / n_{k−1}

so its change in logs is a sum:

log(C'/C) = Σ log(r_k'/r_k)

gallop.explore.funnel_steps reports each step's term and its share of the total. The step with the largest share is where the change happened, and it is the step to cut by segment; the other steps are noise for this question. When two steps carry opposite signs (one improved, one got worse) both are reported, because a redesign that helps one step and hurts the next is a common shape.

Log shares are exact but can exceed one in magnitude when steps offset each other; read the sign and the size together.

What decomposition cannot do

It says where, not why. A rate effect in the payment step on Android is a location, and the mechanism (a broken wallet integration, a new fee, a competitor's launch) is the hypothesis the next step writes. And it inherits the floor: decomposing a tracking change produces a precise location for a bug.

The floor first

Most sudden moves in a product metric are changes in what is counted, not in what people did. This is the checklist that runs before any segment is cut, and the tells that say which kind of move you are looking at.

What to check, in order

1. Raw event volumes by day. The numerator event and the denominator event separately, not the rate. A rate stays flat while both halve, and a rate falls when only the denominator doubles. Plot both.
2. Every step of the funnel together. If landing views, starts and completions all fell by the same share on the same day and the step rates are flat, nothing in the flow changed. Either fewer people were counted or fewer people arrived. Both are upstream of the surface.
3. What shipped in the window. Tracking releases, tag-manager publishes, SDK and app-version rollouts, consent or cookie banner changes, redirects and URL changes on tracked pages, bot and internal traffic filters, sampling thresholds, a renamed event, a new surface nobody instrumented. A step change on a release day with no product change is tracking until proven otherwise.
4. A source of truth the tracking does not touch. Orders in the database, accounts created, invoices, support tickets. If the truth is flat while the measured number fell, the number fell and nothing else did. This one check settles most cases.
5. The calendar. Partial days at either end of the window, a timezone boundary that moved, late-arriving events, a backfill that has not run, a holiday in one market.
6. The definition. Did anyone change the metric's query, filter or window? defining-metrics keeps the registry; check the entry's notes and owner.

The tells

• Consent banners. A new or stricter banner reduces the share of visitors who are measured, not the share who convert. Every client-side number falls together; server-side counts do not move.
• App releases. A drop confined to one platform and one app version, starting on the rollout day and growing with adoption, is an SDK or event bug in that version.
• Numerator without denominator. Completions fall, starts do not: a broken completion event, or a real flow problem. The source-of-truth count decides which.
• Round-number cliffs. A metric that fell to exactly zero for a segment, or a step that fell to a suspiciously round share, is a filter or a join, not behaviour.
• Sampling. Reports built on sampled data drift as traffic crosses the sampling threshold; the unsampled export disagrees.

What to hand over

When the floor is live, the hand-back is a measurement finding, not a behavioural hypothesis: which event, which day, which release, the source-of-truth comparison, and the size of the counting error. It goes to defining-metrics, which fixes the definition or the instrumentation and re-enters the question once the number can be trusted. Analysing the drop "in the meantime" produces a story about a bug.

The segment scan

What a segment result is, what it is not, and the discipline that keeps a deep dive from finding whatever it went looking for.

The forking paths

Six dimensions with six or seven levels each is forty cuts. At the usual threshold, two of forty look significant when nothing is happening, and the analyst who stops at the first one has found noise. The defence is mechanical and it is in gallop.explore:

• Every cut is counted. The scan reports how many segments it examined. That number goes in the hand-back next to the result, so the reader can judge one striking segment against the forty it was picked from.
• False-discovery adjustment. Benjamini-Hochberg across all cuts, with the adjusted p beside the raw one. A raw p of 0.02 that is the smallest of forty adjusts to about 0.8; it is not evidence of anything.
• Ranked by contribution, not by swing. A segment of four thousand units that lost a point moved a total of two hundred thousand by almost nothing. Contribution to the total change is the sort order; the segment's own change is a detail.

What a segment result is

A hypothesis with a location: "the payment step on Android in Spain fell from 3.8% to 2.9%; if a wallet integration broke in the last release, fixing it recovers about a tenth of the total drop". It goes into the hand-back as a hypothesis, sized by its contribution, and it is tested before anyone calls it the cause. An unregistered segment found while exploring is a hypothesis for the next test, the same rule reading-experiments applies to unregistered segments in a readout.

What it is not

• Not the cause. A segment where the metric fell is where the fall shows, not why it happened.
• Not an effect size. "Users who did X convert 4x" is a gap between two populations that chose themselves. The gap is not what making everyone do X would produce; see sizing.
• Not a finding, on its own. Without the count of cuts and the adjustment, a striking segment is an anecdote with a p-value.

Which segments

• Behavioural over demographic. What users did (arrived from search, used the wallet, opened the app twice) suggests a mechanism. Who they are (age band, country, plan) suggests a stereotype and rarely a lever.
• Cohort over period. Retention read by calendar period mixes users of every age; read it by signup cohort or the mix shift masquerades as a trend.
• Survivors. A segment defined by an outcome (users still active in month three) has already selected on the thing being measured. Segment on what was true at the start.
• Averages hide distributions. A mean session length rising because a few bots spent hours is not engagement. Look at the median and the tails before segmenting the mean.

When to stop cutting

When the change has localised to a step and a segment that together carry most of it, and a mechanism can be written in one sentence. Cutting past that point finds smaller segments with larger swings and less meaning. Time-box the scan and hand back what is localised; the next segment is the next question, not this one.

Sizing

The number a roadmap weighs, and the three ways it goes wrong: the whole gap is assumed to close, the estimate is anchored on nothing, and the size is quoted without the traffic that would be needed to see it.

Three numbers, always all three

gallop.explore.size_opportunity returns them together:

1. The ceiling. Gap times the units affected: every affected unit moves the full distance. It is the upper bound and it is presented as one. "If mobile checkout matched desktop, that is 1,900 orders a month" is a ceiling, and the sentence should say so.
2. The anchored estimate. From the prior store, for this metric: the mean effect past changes produced (the planning number, usually a small fraction of the ceiling, sometimes zero) and the largest effect ever recorded (the most a test could plausibly find). A change that would need to beat every effect this metric has ever produced is priced as a long shot, whatever the ceiling says. Without a store the estimate is unanchored, and the hand-back says so in those words.
3. The MDE at the surface's traffic. gallop.power at the longest run the surface can afford. Sizing a change nobody could measure is sizing a belief.

The verdicts

• Largest recorded effect at or above the MDE. A test can find what this lever produces. Route to designing-experiments with the anchored mean as the expectation and the ceiling as the upper bound.
• Ceiling below the MDE. Even total success is invisible at this traffic. Decide without a test and write that down, or propose a change big enough to be seen; a copy tweak on a surface with a sixteen percent MDE is a hope, not a hypothesis.
• Anchored mean near zero, ceiling large. The usual case. The opportunity is real and the lever is unproven; the test is worth funding if the ceiling justifies the traffic, and the expectation stays at the anchored mean until the readout says otherwise.

The gap is not the prize

"Users who add a payment method in their first session convert at 41% against 9%." The 32-point gap multiplied by every signup is not the value of forcing the step. The two groups differed before the behaviour: the ones who added a card had already decided. The ceiling for a mandatory step is not the gap, and the realistic size comes from what onboarding changes have moved before, from the prior store, or from nothing. The hypothesis is the step, randomised, with signups as a guardrail, because a mandatory step also loses people. This is the trap case from routing-questions, seen from the sizing side, and it is the single most common way a roadmap number is wrong by an order of magnitude.

Assumptions visible

An opportunity estimate is a chain of assumptions, and the hand-back lists them as line items with their values: the affected share, the capture share, the conversion from the on-platform metric to the business number, the horizon. Then, when the readout lands under the estimate, the conversation is about which assumption was wrong rather than about whether the estimate was right. The line everyone skips is the last conversion: on-platform behaviour into an off-platform business claim (sessions into sales, signups into revenue, redemptions into retained customers). It is usually the least certain number in the chain and it belongs on the page with its source.

Cost on the same page

A sized opportunity is half a decision. The other half is what the change costs: engineering time, the surface it occupies, the tests it displaces, the guardrail it risks. The verdict a PM needs is the two side by side, and it is the PDS's to write, because nobody else in the room has both numbers.

<What moved, in one line: metric, direction, size, window>

<date · surface(s) · requested by · the decision this unblocks>

Against which baseline. <Same period last year, trend, seasonal pattern. The ordinary wobble (week-to-week sd) beside it, and whether the move is outside it.>

The floor, checked. <Raw volumes by day for numerator and denominator; what shipped in the window; the source-of-truth count; the calendar. One line each, with the result. If any is live, the record stops here and goes to defining-metrics.>

Where it localised.

• Mix or rate: <rate effect / mix effect, in metric units, and the share of each>
• Funnel step: <the step carrying most of the change, and its share>
• Segment: <the segment ranked first by contribution, with its raw and adjusted p, out of N cuts examined>

The mechanism.

<If we change X, metric Y moves by about Z, because W. One sentence. If it cannot be written, there is no hypothesis yet.>

The size.

• Ceiling: <gap × units affected, per period>
• Anchored estimate: <prior-store mean and largest recorded effect for this metric, or "unanchored: no store">
• MDE at this surface's traffic: <value, at the longest affordable run>
• Assumptions, as line items: <affected share · capture share · on-platform to business conversion, with source · horizon>

Verdict. <the floor first / not a question / a hypothesis worth testing / too small to measure>

Route. <designing-experiments (randomise what) / choosing-causal- designs (assignment already happened how) / defining-metrics / backlog>

Filing rules: one record per question, attached to the question issue. It re-enters routing-questions at step 3 as a change question. It does not contain an effect size, it does not recur, and the cuts examined are never left out.