Every Campaign Is an Experiment: Learning from Small, Noisy A/B Tests and Bandits

Marketing runs on experiments, but they're hard to read. Each campaign runs for days, every customer sees only one variant, effects are small, and the audience drifts between runs. A/B tests and contextual bandits can pick a winner within one campaign. The harder part is knowing what you actually learned, and what carries over to the next campaign. In this hands-on workshop, you work through that with marketing campaign data covering both A/B tests and contextual bandits: Reading A/B tests: small-sample statistics, bootstrap confidence intervals, and correction for multiple comparisons across segments and variants. Reading bandit logs: a bandit chose who saw what, so naive averages are biased. You use off-policy evaluation on logged selection probabilities to estimate how each variant would have performed, including a policy you never ran. Learning across campaigns: you find which customer attributes and content features, such as tone or offer type, explain why a variant won. Content features let a lesson carry over to a campaign whose variants are entirely new. You then pool evidence across experiments and turn each finding into a hypothesis the next campaign can test. You also learn to spot the classic failures: lucky early wins hardening into "facts", analyses repeated until something turns significant, and a proxy metric improving at the expense of the real one. You leave with notebooks for reading A/B test and bandit results correctly, and a practical sense of what small, noisy experiments can and cannot tell you. These are also the guardrails any automated optimisation loop needs. Prerequisites: Python, pandas, basic statistics.