experimental-design
A/B test with a twist
It is the 2010s, and a website change is about to ship to millions — but you suspect it helps new users and quietly harms returning ones, and a single averaged number would hide that cancellation entirely. You cannot simply double your traffic to test every subgroup; attention and revenue are the budget. Design an online experiment that detects heterogeneous effects — who is helped, who is hurt — without doubling cost, and that respects the messy reality of users cycling through. State how randomization and modeling combine to make the subgroup claims trustworthy. Get it wrong and the company rolls out a change that lifts a headline metric while silently driving away its most loyal customers, invisible until they are already gone.
Who this problem belongs to
The two figures whose methods fit it best, out of 40 in contention.
This problem is nearly a description of Murphy's actual research program. Her micro-randomized trials, developed in the 2000s-2010s for mobile-health interventions, randomize each user repeatedly over time precisely because people cycle through states — engaged, disengaged, returning — and treatment effects differ across those states. She built the estimands and weighted estimators for time-varying, person-specific effects under sequential randomization, and her dynamic treatment regime work formalizes who should get which version when. The problem's twin demands — detect heterogeneous effects without doubling traffic, and respect users moving through the system — are exactly what micro-randomization buys: each user contributes many randomizations, so subgroup contrasts come cheap. Her methods were designed for the 2010s platform setting this problem lives in. Era gap: essentially none.
Rubin's potential-outcomes framework, articulated from the 1970s onward, is the language in which the problem's central claim — helps new users, harms returning ones — becomes a precise statement about unit-level causal effects rather than a vibe about averages. His work supplies the machinery the subgroup analysis needs: pre-specified strata versus post-hoc fishing, principal stratification for subgroups defined by post-treatment behavior (returning is partly an outcome), and Bayesian imputation of missing potential outcomes to estimate conditional effects without doubling sample size. His insistence that the assignment mechanism be modeled explicitly is what makes subgroup claims trustworthy under messy user churn. He predates online platforms, but 2010s tech A/B practice imported his framework wholesale; the era gap runs in his favor. Marginally weaker only on the streaming, adaptive engineering side.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
40 figures are scored on this problem. Draw it in a battle to see where you land.