The floor

skills/defining-metrics

defining-metrics

Turns a metric name into a computation, a source of truth, a registry entry, and a written statement of how it will be gamed, then decides whether it is trustworthy enough to promote. Use when defining a north-star or guardrail metric, when two dashboards disagree on the same number, when arbitrating between conflicting metric definitions, or when a readout depends on a metric nobody has validated.

shell

$ git clone https://github.com/0trm/gallop
$ cp -r gallop/skills/defining-metrics .claude/skills/

Read the source on GitHub↗

The floor. Not a stage: everything above it inherits its errors, and a wrong definition does not raise an error, it just returns the wrong number. This skill produces four artifacts for one metric: a computation, a source of truth, a registry entry, and a statement of how the metric will be gamed. A metric missing any of the four is not ready to carry a readout.

When two numbers disagree

The most common entry point: the same metric shows different values in two places and nobody trusts either. Do not average them, and do not pick the one closer to expectations. Trace each to its query and diff the definitions. The difference is almost always one of five things, checked in this order:

Window – 7-day vs 28-day, calendar vs rolling, and the timezone the day boundary uses.
Filter – bots, internal traffic, test accounts, one market, one platform; applied in one query and not the other.
Unit and dedup – users vs sessions vs events, and whether repeats within the window count once or every time.
Join – a join that drops units with no activity, turning a rate's denominator into "active units" without anyone deciding that.
Freshness – one source lags the other; the numbers were never computed over the same days.

Name the discrepancy mechanically ("A excludes bounced sessions, B does not"), then decide which definition serves the decision the metric exists for, and register that one. The other query gets updated or deleted, not left as a second opinion.

Writing the definition

A definition is precise enough when someone could reimplement it from the text alone and match the number. It states:

The computation. Numerator, denominator, window, dedup rule, filters. "Activation rate: users who completed at least one core action within 7 days of signup / users who signed up, excluding internal and bot accounts, UTC days."
The unit of analysis. What one observation is: user, session, order, day. This is also what experiments randomise on, so a mismatch here becomes a broken analysis later.
The source of truth. One table or model, named. When a dashboard and the source disagree, the source is right by definition and the dashboard is a bug.
The direction. Which way is better. Sounds trivial; guardrails and automated checks need it explicit.

The proxy bridge

Most product metrics stand in for an outcome the product cannot observe (a purchase in someone else's store, long-run retention, revenue attributed weeks later). If this metric is a proxy:

Name the bridge in every readout. One sentence: this result is a change in X, here is what we currently believe X is worth in Y, and here is the assumption doing the work.
Validate the proxy on a schedule, against something real: cohorts, panel data, whatever exists. Once a year, off-roadmap, non-negotiable.
Prefer metrics whose outcome you own. Given two candidate framings of similar value, take the one that ends inside your own instrumentation.

How it will be gamed

Before the metric is used to judge anything, write down how a well-meaning team hits the number without creating the value it stands for. Every metric has at least one; a metric whose gaming nobody can describe is a metric nobody has thought about. Common patterns and worked examples are in reference/gaming.md. The output is one or two sentences in the registry entry's gaming field, plus the guardrail metric that would catch it.

The promotion gate

A metric is trusted – promotable to primary or guardrail duty in experiments – only when the checklist in reference/promotion.md passes: definition, source of truth, instrumentation validated end to end, gaming statement, owner, and stability checked over enough history to know its variance. Until then it is provisional: usable for exploration, barred from readouts. Retired definitions become deprecated, kept in the registry so old readouts remain interpretable.

The provisional exit

Barring provisional metrics from readouts is the rule. A rule with no exit gets ignored rather than followed, and an ignored rule leaves no record of what it failed to stop. This exit is narrow, and it is logged.

Work may proceed on a provisional metric when all three hold:

A person signs off. Named, not a team: whoever would be the registry owner once the metric is promoted.
The defect is bounded and cannot change the answer. Name the defect, bound it, and show the conclusion holds at both ends of the bound. Two sources 8% apart on the level do not stop "which segment is largest"; they do stop "did it decline". An unreconciled discrepancy is not a bounded defect: until the diff is traced to a filter or a broken join, nobody knows which end is which, and nothing can be shown to hold at both. If the question turns on the quantity the defect touches, the exit is closed and the floor work is the work.
Nothing is filed. The result carries an expiry, not a belief: no prior-store record, no knowledge-repo entry. It answers one decision and dies with it.

The exit opens toward description only, where a hypothesis is the hand-back anyway. An experiment's primary or guardrail metric comes back from the trusted filter or the experiment does not run; designing-experiments has no equivalent hatch, and this one does not reach it.

The exit's backlog

Every use of the exit writes its reason onto the metric's registry line, in provisional_reason:

blocked – the promotion work cannot be done now. The instrumentation is missing, the source table does not exist, no owner exists to arbitrate the definition. The fix belongs to someone who is not in this conversation, and promotion_blocker names the unmet checklist item and who owns closing it.
deferred – the promotion work could be done and has not been ranked high enough to do. Nobody is blocked; the metric has not been worth the week.

The split is what makes the registry a backlog instead of a leak. The blocked lines are a dependency list to take to the team that owns the fix. A deferred line that keeps carrying questions is the argument for ranking its promotion, and one nobody has used is correctly ignored.

The registry entry

The registry is a JSONL file beside the prior store, one metric per line, validated against templates/metric-registry.schema.json. Field-by-field guidance and a worked example are in reference/registry-schema.md. Read it back with:

python -m gallop.priors registry --registry metrics.jsonl --status trusted

Experiments must take their primary metric from the registry, not from a text box. That single rule is what makes the floor hold: it turns every definition argument into a one-time cost instead of a per-readout one.

Maintenance

The definition of success is maintained continuously, like calibrating a scale you weigh things on every day, not set once in January:

Every ship changes the data. A launched feature changes user mix and event volume; after a significant ship, check that the metric still means what it meant. This is the dashed loop on the method map closing.
Version changes. When a definition changes, bump it explicitly, note the change date on any chart spanning it, and treat pre/post numbers as different series. Metric drift in the catalog is how last year's numbers stop matching without anyone deciding anything.
Alert on the floor, not the ceiling. The two failures that ruin tests from below are events that stop firing on one platform and exposure logs that drift; alert on event volume, not just on metric values.

Hand-back

To the question that routed here: a number the rest of the map can stand on. State what changed (the definition, the source, or both), what the corrected current value is, and which past readouts, if any, are now suspect. Then re-enter routing-questions with the original question, which can now be answered on a floor that holds.

reference gaming.md

How metrics get gamed

Goodhart's law, made operational: when a measure becomes a target, it stops measuring. The registry requires a gaming statement because the failure is predictable before it happens, and predicting it names the guardrail. None of these require bad faith; each is a reasonable team optimising exactly what it was told to optimise.

The patterns

Widen the numerator. The cheapest game. "Activated" grows to include lighter and lighter actions; the rate climbs while the behaviour it stood for does not. Guardrail: pin the qualifying action list in the definition and version any change.

Shrink the denominator. Filter out the users least likely to convert (bots today, then low-intent channels, then a whole platform) and the rate rises with no change in anyone's behaviour. Guardrail: report the denominator's absolute size next to every rate.

Move volume across the boundary. A 7-day activation window teaches teams to cram nudges into day 6 and steal from day 8. The metric improves; the user's month does not. Guardrail: a longer-window twin of the same metric, read directionally.

Cannibalise the neighbour. Clicks on the promoted module go up because clicks on the module below it went down. Any surface-level metric can be fed by its neighbours. Guardrail: a same-page total, so reshuffling nets to zero unless something real happened.

Harvest intent instead of creating it. Attribution games: intercept users who would have converted anyway (branded search, checkout interstitials) and book them as caused. Guardrail: incrementality checks by experiment, not by attribution model.

Degrade the unmeasured. Push the measured number by spending something the registry does not watch: latency, support load, refund rate, trust. Guardrail: the standing guardrail set (crashes, latency, refunds, unsubscribes) attached to every experiment by default.

Ship the prompt, not the product. Any metric countable as "users who did X once" is inflatable with a modal. The lift is real, the value is not, and retention of the prompted cohort shows it. Guardrail: cohort retention of prompted vs organic users on the same action.

Writing the statement

One or two sentences in the registry gaming field, concrete enough to recognise when it starts happening:

activation_rate – gamed by widening the qualifying action list or by day-6 nudges that move activation inside the window without changing week-4 retention. Guardrail: week-4 retention of activated users, and the action list is versioned.

If no plausible gaming exists, write why. That is rare and worth recording; it usually means the metric is very close to actual value exchange, which is also the argument for its promotion.

reference promotion.md

The promotion checklist

A metric moves from provisional to trusted only when every item below passes. Trusted is the status that licenses primary or guardrail duty in an experiment; everything else in the registry is context. Run the checklist as written, record the date it passed, and re-run it after any definition change.

1 · Definition closed

Someone who has never seen the codebase can reimplement the metric from the registry text and match the number within rounding. Test this literally when the stakes justify it: hand the definition to a second person, have them compute yesterday's value from the source of truth, compare.

2 · One source of truth

Exactly one table or model is named. Every surface that shows the metric (dashboard, report, experiment readout) reads from it or from a documented derivative. If two surfaces currently disagree, promotion is blocked until one of them is fixed or killed; a metric with two values is not a metric.

3 · Instrumentation validated end to end

The events the metric is built on have been traced from the client (or the service) to the warehouse at least once, by someone, on purpose:

Fire the event in staging, watch it land in the source table with the expected fields.
Check volume by platform and by market against expectation. The failure that hides best is an event that stopped firing for one platform or one country; the total looks plausible while a segment is dark.
Confirm the identity join. A metric per user is only as good as the mapping from event to user, across devices and sessions.

4 · Gaming statement written

The gaming field is filled in with a plausible mechanism, and the guardrail that would catch it exists or is explicitly declined with a reason. See gaming.md.

5 · Owner named

One person answers for the definition. Not a team: a person. The owner is who arbitrates the next disagreement and who signs off on version changes.

6 · Stability known

Enough history exists to know the metric's normal variance: week-over-week movement under no intervention, seasonality shape, and the size of day-level noise. Without this, no experiment on the metric can be sized honestly and no anomaly on it can be triaged. Concretely: pull at least eight weeks of daily values, compute the coefficient of variation, and record the typical week-over-week swing in the registry notes.

7 · Sensitivity plausible

The metric can move at the traffic and timescale a test actually has. A metric that is 95% determined by behaviour outside the product's control, or that responds to change with a six-week lag, can be a guardrail but should not be a primary. If the answer is "this cannot detectably move in a two-week test", promote a nearer proxy as primary instead and keep this one directional, and say so in notes.

While provisional

A metric that fails any item above stays provisional, and the item it failed is what goes in the registry's promotion_blocker, by number: "3, scroll events never traced from client to warehouse; owned by the web platform team". Paired with provisional_reason, blocked or deferred, that turns the provisional lines into ranked floor work instead of a list of metrics nobody got round to. The conditions under which a question may run on one of them are in the skill body, under the provisional exit.

Demotion

Deprecate, never delete. A deprecated entry keeps its definition and dates so old readouts stay interpretable. Deprecation triggers: the definition was superseded, the instrumentation broke irrecoverably, or the gaming statement came true and the guardrail did not hold.

reference registry-schema.md

The metric registry, field by field

One JSONL file, one metric per line, diffable in a pull request. The contract is src/gallop/templates/metric-registry.schema.json, published at the repo root as templates/metric-registry.schema.json; gallop.priors validates every line on read and fails loudly with the line number on a malformed entry.

Fields

Field	Required	What goes in it
`name`	yes	Snake-case, the key the prior store uses. Never reuse a name for a changed definition; version it (`activation_rate_v2`)
`definition`	yes	The computation: numerator, denominator, window, dedup, filters, timezone. Precise enough to reimplement
`source`	yes	The one table or model that is the source of truth
`unit_of_analysis`	yes	What one observation is: `user`, `session`, `order`, `day`. Experiments randomise on this
`direction`	yes	`increase_good` or `decrease_good`
`role`	no	`primary`, `guardrail`, or `diagnostic`: how experiments may use it
`gaming`	yes	How the metric gets hit without the value being created, and the guardrail that catches it
`status`	yes	`trusted`, `provisional`, or `deprecated`. Only trusted metrics carry a readout
`provisional_reason`	if provisional	`blocked` (the promotion work cannot be done now) or `deferred` (it could be and has not been ranked)
`promotion_blocker`	if provisional	The unmet promotion checklist item, by number, and for `blocked`, who owns closing it
`owner`	no	The person who answers for the definition
`notes`	no	Stability facts (typical weekly swing, seasonality), proxy bridge, version history

The last two are optional in the schema because gallop.priors.validate has no conditional rules; the discipline is that a provisional line without them is incomplete, the same way an intake record with an empty field is not ready to rank. They are what the provisional filter reads as a backlog.

A worked entry

{"name": "activation_rate",
 "definition": "users with >=1 core action (save, share, or publish) within 7 days of signup / signups; excludes internal accounts and known bots; UTC day boundaries; user counted once",
 "source": "warehouse.marts.user_activation",
 "unit_of_analysis": "user",
 "direction": "increase_good",
 "role": "primary",
 "gaming": "widen the qualifying action list, or day-6 nudges that move activation inside the window without moving week-4 retention; guardrail: week-4 retention of activated users",
 "status": "trusted",
 "owner": "0trm",
 "notes": "weekly swing ~0.4pp under no intervention; signup mix shifts seasonally in September; v1 since 2026-05"}

And one provisional entry, which is a line of backlog as much as a metric:

{"name": "engaged_reader_rate",
 "definition": "sessions with >=30s on an article and >=50% scroll depth / article sessions; 28-day window; excludes known bots; UTC day boundaries",
 "source": "warehouse.staging.article_events",
 "unit_of_analysis": "session",
 "direction": "increase_good",
 "role": "diagnostic",
 "gaming": "lazy-loaded or autoplaying elements fire scroll events with no reader behind them; guardrail: return visits within 7 days",
 "status": "provisional",
 "provisional_reason": "blocked",
 "promotion_blocker": "3, scroll events never traced from client to warehouse; owned by the web platform team",
 "owner": "0trm",
 "notes": "carried two sizing questions in Q3; source is a staging table, not a mart"}

Reading it

python -m gallop.priors registry --registry metrics.jsonl
python -m gallop.priors registry --registry metrics.jsonl --status trusted

The second form is what designing-experiments runs: an experiment's primary metric must come back from that filter. If it does not, the experiment is blocked on defining-metrics, not on more traffic.

Conventions

The registry lives in version control next to the prior store. Changes go through review; the reviewer is the owner.
A definition change is a new version, not an edit: deprecate the old entry (set status), add the new one under a versioned name, and note the changeover date on any chart or readout that spans it.
The registry is append-friendly but not append-only: status and notes may be edited in place. History lives in git, which is one of the two reasons the format is JSONL. The other is that a registry nobody can review in a pull request is a registry that goes stale.