PYTHON · ML · CLOUD · B2B SaaS, Dublin
ChurnRadarSubscription Churn Early-Warning Scoring
Churn at a Dublin B2B SaaS company was discovered at renewal or not at all: 1.9% of its 8,400 accounts left every month, and customer success had capacity to work about 60 a week with no defensible way to choose which 60. I built a daily churn score, calibrated it so the probability means what it says, and shipped it as a ranked worklist.
- 23 days median warning ahead of a churn event
- 0.81 ROC AUC on the held-out quarter, calibrated
- −27% logo churn in the cohort worked from the score
01 — Context
The problem.
Churn was treated as a renewal-desk problem. An account manager saw a contract 30 days out, made a call, and discovered the champion had left four months earlier. Meanwhile the product emitted a rich event stream that nobody had turned into a leading indicator.
The team had also tried a rules-based health score, which failed in a specific and instructive way: it fired on almost every enterprise account because it keyed off absolute usage rather than change relative to that account’s own norm.
02 — Method
The approach.
The build, in the order it happened.
-
Features relative to the account’s own baseline
Every behavioural feature is expressed as a deviation from that account’s trailing 90-day norm — seat activation, weekly active users, depth of feature use, support-ticket sentiment and admin logins. A large account going quiet is the signal, not a small account being small.
-
A defensible label and no leakage
Churn is defined as non-renewal or a downgrade past 40% of contract value, dated at notice rather than expiry. All features are computed as of the score date, and anything touched by the renewal process itself is excluded — it predicts beautifully and tells you nothing.
-
Calibrated, then thresholded on cost
XGBoost with isotonic calibration, so a 0.7 score genuinely means roughly seven in ten. The intervention threshold was then chosen against the team’s real weekly capacity and the cost of a wasted call versus a lost account.
-
Shipped as a worklist, with reasons
A Cloud Run job scores every account nightly and writes a ranked list with the top three contributing drivers per account. Customer success get a queue and an explanation, not a black-box number.
03 — Outcome
What changed.
- Median warning ahead of a churn event is 23 days, against effectively zero under the renewal-desk process.
- ROC AUC of 0.81 on a fully held-out quarter, with calibration error under 4 percentage points across all deciles.
- Logo churn in the worked cohort fell 27% against a matched control over two quarters.
- Precision at the team’s working capacity of 60 accounts a week is 0.46 — roughly one in two calls reaches an account that would otherwise have left.
04 — The numbers
Monthly logo churn in the worked cohort.
Tracked as % of accounts, from Jan through Oct — low 1.38, high 1.9.
05 — Screens
What it looks like in use.
Calibration and precision-at-capacity. The threshold was set from this page and the team’s weekly headcount, not from a default of 0.5.
Per-account drivers. A score without a reason gets ignored on the second week.
06 — Stack & role
Built with.
- Python (Pandas, scikit-learn, XGBoost)
- BigQuery
- Cloud Run
- dbt
- Looker Studio
- Role
- Consulting data scientist — feature design, modelling, deployment
- Duration
- 9 weeks
- Client
- B2B SaaS, Dublin
- Period
- 2025
07 — Questions
The questions I get asked about this one.
It measured absolute usage, so every large account looked healthy and every small one looked at risk. Normalising each feature against the account’s own trailing baseline fixed that — though it does depend on a product event stream. On billing data alone, three weeks of warning is not available.
The worked cohort was compared against a matched control held back from the worklist for two quarters — matched on contract value, tenure and segment. Without a control, any improvement is just a good quarter.