Prediction

skills/automating-decisions

automating-decisions

Decides whether a repeated, at-volume decision belongs to a model, which output it needs (a forecast, a ranking, an allocation) and whether targeting needs propensity or uplift, validates the model out of time against the base rate and the naive baseline, and hands the impact claim to an experiment. Use when someone asks for a churn, propensity, LTV, scoring, forecasting, uplift, recommendation or allocation model, when a model's offline accuracy is offered as evidence that something worked, or when deciding who gets an offer, a discount or an intervention.

shell
$ git clone https://github.com/0trm/gallop
$ cp -r gallop/skills/automating-decisions .claude/skills/
Read the source on GitHub↗

Ships with

The prediction position on the map: a decision made repeatedly, per user or per day, by a system rather than by a person reading a readout. A forecast that sets inventory, a ranking that picks who gets the offer, an allocation that splits a budget. The bucket asks what will happen and who gets what, hands back a forecast, a ranking or an allocation with its uncertainty attached, and fails by breaking the moment you intervene, because the model learned a world in which nobody had acted on it yet.

Two rules arrive with every question routed here and leave with it. The model needs volume, history and a trusted measurement floor. And the model claims nothing about impact until an experiment says so: a churn model that predicts beautifully says nothing about whether the campaign works.

The mechanical part is scripted. Given scores on an out-of-time window:

python skills/automating-decisions/scripts/validate_model.py \
  --data scored.csv --y churned --score p_churn --date week --cutoff 2026-07-01 \
  --k 0.1 [--arm arm --control control] [--features tenure,logins_30d]

Or individually via python3 -m gallop.validate {lift,calibrate,screen,qini,mase}. The package scores a fitted model; it does not fit one. Fit with whatever library is at hand.

0 · Is this a model's decision?

Three things have to be true. Routing checked only the first.

  • Decided continuously. A ship-or-kill call made once needs an effect size, not a forecast. Send it back to the causal branch.
  • The floor holds. The outcome the model predicts is trusted in the metric registry. A model fits noise and reports confidence; it amplifies an instrumentation problem rather than revealing it. If the outcome is provisional or disputed, the work belongs to defining-metrics first.
  • Volume and history. For a classifier, a few hundred positive outcomes in the out-of-time window alone, and about a thousand in total; for a forecast, several full seasons. Below that the validation cannot separate the model from the base rate at the operating point, so nothing about the model can be believed, and the verdict is not fundable. Say so and stop. Rolling folds do not create outcomes that do not exist, and "build it thin and validate as data accrues" ships an unvalidated model for a year. The useful object at that size is a rule: a threshold on one or two fields, written down with the metric it will be judged on and the volume at which a model would become checkable. A regional squad rarely has the volume to justify anything heavier, and saying so beats shipping a model nobody can validate.

1 · What will happen, or who gets what?

Three outputs, and the request names one.

  • A forecast. A metric's value in a future window. Hands back a point and an interval, conditional on nothing changing.
  • A ranking. Units ordered by an expected outcome, acted on from the top. Hands back the ordered list and the operating point: how far down the list the decision goes.
  • An allocation. A fixed resource split across options. Hands back the policy and its expected value, with the forecast or ranking it rests on.

Watch for the causal question in a model's clothing. "Build a churn model so we can see whether the retention emails work" asks two things. The model predicts who churns; the question is whether the emails change that. Prediction rides on correlation and does not care why. The emails were aimed at the people most likely to leave, so the model learns the campaign's targeting and reports it back as risk. Refuse to read impact from a model. Split the request: the model for targeting stays here, the impact goes to designing-experiments. The same refusal covers "the forecast says we beat plan, so the launch worked": a forecast error is not an effect size.

2 · Propensity or uplift

A propensity model ranks by the chance of the outcome. An uplift model ranks by how much the treatment changes that chance. For a decision about who gets an intervention, the second is the question and the first is the usual answer, which targets the wrong people: the sure things who would have stayed anyway and the lost causes who leave regardless sit at the top of a propensity list, and the persuadables sit in the middle. The four kinds of unit and what each model does with them: reference/propensity-vs-uplift.md.

Uplift needs randomised rows: units where the treatment was assigned at random and the outcome logged. Observational rows cannot supply it; a model fit on them learns whoever chose the targeting. If no randomised data exists, the experiment comes first: route to designing-experiments, randomise the offer across the eligible population with exposure logged, and train the uplift model on that experiment's rows. Propensity is still the right model when the question is who will do something rather than whom to treat: fraud, capacity, support routing.

3 · Validate out of time, against the base rate

The order is fixed and the checks come before the number. Mechanics and the leakage taxonomy: reference/validation.md.

  • Split by time. Fit on rows before a cutoff, score rows after it (gallop.validate.time_split). Random folds on time-ordered data leak the future into the training set and report an accuracy the model will never see in production. Leave a gap the length of the label horizon.
  • Screen for leakage. A feature that separates the outcome near-perfectly on its own is usually the outcome, or something logged after it (leakage_screen). Every flagged feature is explained before any number below counts.
  • Beat the base rate at the operating point. The decision acts on the top k. Report precision at k against the base rate, as a lift (baseline_lift). A lift of 1.0 is the base rate wearing a model. AUC on its own is not a validation, because no decision acts on the whole ranking.
  • Calibration, if the scores are used as probabilities. Thresholds and expected values need calibrated scores; a ranking does not. Reliability table, Brier score against the base-rate forecast, expected calibration error (calibration).
  • Uplift: Qini on the randomised rows (qini), against random targeting. Precision at k means nothing for an uplift model.

The number reported reads "at the top 10% the precision is 31% against a base rate of 8%, a lift of 3.9, out of time on July", never "AUC 0.91".

4 · Forecasts and interventions

A forecast is conditional on nothing changing. It has to beat the seasonal naive (mase): at a MASE of 1 or above, last season's value did as well and is the forecast. Baselines and intervals: reference/forecast-baselines.md.

A launch inside the horizon is the failure mode in its purest form. The model has never seen one and cannot see through it. Hand back two objects: the forecast under no change, and the launch's effect routed to the causal branch as its own question, with a holdout if the plan is still open. A forecast that "includes the launch" is a guess with an interval drawn round it.

5 · The holdout that measures impact

Offline validation says the ranking beats the base rate. It does not say that acting on the ranking moves the metric. That claim belongs to an experiment: hand to designing-experiments with the model as the treatment and the current rule as the control, never no rule at all, randomised at the unit the decision acts on, exposure logged, the metric the model exists to move as the primary. reading-experiments reads it. That readout enters the prior store with design set to experiment. The model's lift and AUC never do; an offline metric is not a readout.

When the decision runs continuously, keep a permanent holdout: a small slice never scored, so the model's contribution can be read at any time and drift shows up as the gap closing.

6 · Drift and expiry

What ships changes the data. A model in production is scored on the population it changed, and its calibration decays from the day it goes live. Compare production scores with realised outcomes at the label's horizon, on a schedule. The knowledge entry names the expiry event as a retrain trigger: calibration drift past a stated bound, the holdout gap closing, or a product change to a feature's source. A model without a stated expiry is a stale belief automated at volume.

The verdict

One of four, stated with the evidence beside it:

  • Not a model's decision → route back: a once-only call needs an effect size; an untrusted outcome needs defining-metrics first.
  • Not fundable at this volume → a rule, written down, with the metric it is judged on and the volume at which a model becomes checkable. Not a thinner model with a warning attached: that is the failure this verdict exists to prevent, and the rule's effect is read by a holdout like any other policy.
  • Validated, impact unmeasured → ship behind a holdout; the impact question goes to designing-experiments.
  • Impact measured → writing-readouts files the readout, the belief and the prior-store record, from the experiment's numbers.

Worked requests

The request as it arrives Verdict Why
"Build a churn model so we can see if the retention emails work." split The model targets; the emails' effect is an experiment
"Score users by churn risk and send the riskiest 20% the discount." uplift, not propensity Sure things and lost causes top a propensity list; randomise the offer first
"Which users get the win-back discount this week?" ranking Continuous, at volume; validate out of time, ship behind a holdout
"Forecast Q4 signups. New pricing lands in November." two objects A forecast under no change, and the pricing effect as a causal question
"Our churn model has 0.94 AUC on cross-validation. Ship it?" not validated Random folds on time-ordered data; rerun out of time, report lift at k
"We have 300 conversions a month. Build a propensity model." not fundable Too few outcomes to separate a model from the base rate; hand back a rule
"Build a demand forecast for the regional warehouse." forecast Beat the seasonal naive or ship the naive; intervals widen with horizon
"The forecast beat plan by 8%, so the campaign worked." refuse A forecast error is not an effect size; route to the causal branch
reference forecast-baselines.md

Forecast baselines

A forecast is a claim that the future looks like the past in a stated way. The baseline states the simplest such claim, and the model has to beat it.

The seasonal naive

Next period equals the same period one season ago. Weekly data with a yearly cycle forecasts this week from the same week last year; daily data with a weekly cycle forecasts Monday from last Monday. It costs nothing, it is explainable to anyone, and on product metrics it is hard to beat by much.

gallop.validate.mase scales the model's error on the test window by the seasonal naive's error on the training window (Hyndman and Koehler's MASE):

  • MASE below 1: the model beats the naive by that factor.
  • MASE at or above 1: the naive did as well. The naive is the forecast, and the model is retired or reworked. This is a verdict, not a tuning note.

The function also reports the naive's error on the test window itself, which is the baseline the model is actually competing with on those dates.

Intervals

A point forecast is a decision input only with its interval, and the interval has to widen with the horizon. A model whose 90% band is the same width at week 12 as at week 1 is not reporting uncertainty; it is reporting the residual variance of its fit. Check the band on the out-of-time window: roughly nine in ten actuals inside a 90% band, and no fewer. A band that holds 60% of actuals is a 60% band whatever the label says.

Interventions

The forecast is conditional on nothing changing. Every product change inside the horizon breaks that condition, and the model, fit on a past without that change, cannot see through it. Three cases:

  • The change is in the past and the model was fit across it. The training data mixes two regimes. Either fit on the post-change window only, if it is long enough, or add the change as a level shift and say so.
  • The change is inside the horizon. Hand back two objects: the forecast under no change, and the change's effect as a causal question routed to designing-experiments or choosing-causal-designs. If the plan is still open, ask for a holdout so the effect can be read properly. Adding the change to the forecast by hand is a guess with an interval drawn round it.
  • The forecast is the argument for the change. A forecast that beat plan is not evidence that something worked. It says the past was a good guide to the future; it cannot say what caused the difference. Route the impact claim to the causal branch.

Forecasts as inputs to allocation

An allocation policy (inventory, budget, staffing) is only as good as the forecast under it, and its cost is asymmetric: under-forecasting a stock out and over-forecasting a write down are different losses. State the loss on each side before choosing the point in the interval to act on; the median is rarely it. The allocation's own impact, once it runs, is again an experiment or a holdout, never the forecast's accuracy.

reference propensity-vs-uplift.md

Propensity versus uplift

Two models that look alike on a dashboard and target different people.

The four kinds of unit

Cross whether a unit responds without the treatment with whether it responds with it:

Responds if treated Does not respond if treated
Responds if untreated Sure thing Sleeping dog
Does not respond if untreated Persuadable Lost cause

The treatment only earns its cost on persuadables. It is wasted on sure things, wasted on lost causes, and harmful on sleeping dogs, the users a retention email reminds to cancel.

A propensity model estimates the chance of the outcome. Its top decile is full of sure things (about to convert anyway) or, for a churn model, lost causes (leaving whatever you send). Persuadables sit in the middle of the list, where a top-k policy never reaches them.

An uplift model estimates the difference the treatment makes: P(outcome | treated) minus P(outcome | not treated), per unit. Its top decile is the persuadables, and its bottom decile, when negative, is the sleeping dogs, the people the campaign should skip.

What uplift needs

Randomised rows. Every unit in the training data was assigned the treatment or the control at random, and the outcome was logged for both arms. That is the only data in which the arm is independent of everything else about the unit, so the model can learn the difference rather than the targeting.

Observational rows cannot supply it. If marketing sent the offer to whoever looked risky, the treated and untreated units differ in exactly the ways that predict the outcome, and a model fit on them learns marketing's rule. It will confidently reproduce last quarter's targeting.

If no randomised data exists, the experiment comes first. Route to designing-experiments: randomise the offer across the eligible population, log exposure, run long enough for the outcome to mature, and train the uplift model on those rows. The experiment doubles as the impact readout of the untargeted campaign, which is the baseline the model later has to beat.

Fitting it

Any of the standard constructions works and none is in the package:

  • Two models, one per arm, uplift as the difference of their predictions. Simple; the difference of two noisy estimates is noisier still.
  • One model with the arm as a feature and interactions, uplift as the prediction with the arm flipped.
  • The class-variable transformation, which turns uplift into a single classification target when the arms are balanced.

The choice matters less than the validation.

Reading the Qini curve

gallop.validate.qini ranks the randomised rows by score, targets the top share, and counts the outcomes gained over what the control rate predicts for that many units. The curve runs from zero to the whole population's incremental outcomes; the diagonal is random targeting. Area above the diagonal is the model's value, and a curve that rises, peaks and falls says the bottom of the list contains sleeping dogs: stop targeting where the curve peaks, not at a round percentage.

Precision at k, AUC and calibration are propensity metrics. Reported for an uplift model they measure the wrong thing.

When propensity is still the right model

When the question is who will do something rather than whom to treat: fraud scoring, demand at a location, support routing, capacity. Nobody intervenes on the unit scored, so the correlation the model learned is the one that holds. The line to draw: if the score decides who receives a treatment meant to change the outcome, the model has to be uplift.

reference validation.md

Validation

What has to be true before an offline number means anything, in the order the skill checks it.

Split by time

Production scores units whose outcome has not happened yet, using a model fit on units whose outcome has. Validation has to look the same: fit on rows before a cutoff, score rows on or after it.

gallop.validate.time_split takes the row dates and returns the masks. Two details an improvised split gets wrong:

  • The gap. A unit whose label window straddles the cutoff carries information from after it. Leave a gap the length of the label horizon (gap in days) between the last training row and the first test row: a 30-day churn label needs a 30-day gap.
  • Several cutoffs. One cutoff gives one estimate. Three or four rolling cutoffs give a sense of how stable the lift is across seasons, which is the thing that decides whether the model survives the year.

Random k-fold cross-validation on time-ordered data is the most common reason a model that scored 0.9 offline scores 0.6 live. It is not a validation and the skill treats a number from it as unvalidated.

The leakage taxonomy

A feature is leaking when it carries the outcome, or information from after it, into the training set. gallop.validate.leakage_screen ranks features by how well each separates the outcome on its own and flags anything above a threshold; a single feature with an AUC of 0.97 has almost never earned it. The kinds, in the order to check:

  • The outcome under another name. A cancellation reason, a refund flag, a "last active" date computed after the churn window closed.
  • Recorded after the outcome. Fields written by the process the outcome triggers: a win-back email sent because the user churned, a support ticket opened at cancellation.
  • Aggregates that include the test window. A 90-day average computed over the whole table before the split, so training rows contain test outcomes.
  • Duplicates across the split. The same user on both sides, with a near-identical feature row.
  • A proxy of the label. Not leakage strictly, but a feature the decision can act on only after the outcome, such as "payment failed".

A flagged feature is removed or explained; explained means a sentence saying why it is available at scoring time and not downstream of the outcome.

The operating point

A decision acts on the top k of a ranking. Everything about the model that matters happens there, and AUC, which averages over every possible k, says little about it. gallop.validate.baseline_lift reports, at the k the decision uses:

  • Precision at k: the outcome rate among the targeted units.
  • The base rate: the outcome rate among everyone.
  • Lift: precision over base rate. A lift of 1 is a coin toss dressed as a model; a lift of 3 at the top decile on a 8% base rate means the targeted units convert at 24%.
  • Recall at k: the share of all outcomes the targeted set captures, which bounds how much any policy acting on that set can move the metric.

If k is not yet chosen, report lift at two or three candidate points and let the cost of the treatment choose. Lift always falls as k grows.

Calibration

A ranking needs no calibration. A threshold, an expected value or a budget allocation does: "treat everyone above 30%" is only a decision if 30% means 30%. gallop.validate.calibration returns:

  • A reliability table: mean score against observed rate, per bin. The bins should sit on the diagonal.
  • The Brier score, mean squared error of the probabilities, against the Brier score of always forecasting the base rate. The skill score 1 - brier / brier_base is the share of the base-rate error the model removes; at or below zero the model is worse than the base rate.
  • The Murphy decomposition: reliability (calibration error, lower is better), resolution (how far the bins' observed rates sit from the base rate, higher is better) and uncertainty (the base rate's own variance, fixed). Brier = reliability − resolution + uncertainty.
  • Expected calibration error, the weighted mean gap between score and observed rate across bins.

Calibration drifts before ranking does. It is the first production check to schedule.

Volume

For a classifier, the out-of-time window needs enough positives that the lift at k has an interval narrower than the lift itself. A working rule: a few hundred positive outcomes in the test window; at the top decile that gives a few dozen targeted positives, the least that separates a lift of 2 from a lift of 1. Below that, the model cannot be told from the base rate and the verdict is not fundable. The hand-back at that volume is a rule on one or two fields, with the metric it is judged on and the volume at which a model becomes checkable.