How to Run a Holdout Test Your Finance Team Will Trust
Attribution data tells you what happened. Holdout tests tell you what your ads actually caused. Here's how to run one that earns CFO confidence.
Why Holdouts Matter
Every attribution model, including data-driven, makes assumptions about how credit should be distributed. They tell you what touched the customer, but not whether the ads actually changed their behaviour.
A holdout test is the only way to answer the incrementality question: if we hadn't run these ads, would the customer have purchased anyway?
The Finance Perspective
CFOs don't trust attribution because it's based on correlation. Holdout tests provide causal evidence that advertising changed outcomes, not just that it touched customers who later converted.
Designing the Test
The most practical approach for ecommerce is geographic holdouts. Select regions that are statistically similar in terms of customer behaviour, then pause advertising in some while continuing in others.
Key design principles:
- Holdout regions should represent 10-20% of your market, enough to be statistically meaningful
- Match test and control regions by historical sales patterns, not just demographics
- Run for 4-8 weeks minimum to smooth out weekly variation
- Avoid major promotional periods or seasonal shifts
Running a Holdout
Start by documenting baseline performance in both test and control regions for the 4 weeks before the test. This gives you a comparison point and shows whether the regions behave similarly.
During the test, monitor daily but resist the temptation to end early. Statistical significance requires sufficient data, and early termination biases results.
Protecting the Test
Don't change pricing, promotions, or inventory levels differently between regions. Any variation other than advertising presence invalidates the comparison.
Interpreting Results
Calculate the lift by comparing sales in advertised regions versus holdout regions, adjusted for any pre-test baseline differences.
Key metrics to extract:
- Incremental revenue: Additional sales caused by advertising
- Incremental ROAS: Incremental revenue divided by ad spend
- Baseline rate: Sales that would have happened without advertising
If your incremental ROAS is significantly lower than your attributed ROAS, that gap represents customers who would have purchased anyway but are being credited to ads.
Presenting to Finance
Lead with the business question: "How much revenue does our advertising actually create?"
Show the test design, emphasising the control methodology. Finance teams understand controlled experiments because they use similar logic in financial modelling.
The Key Number
Present the incremental ROAS alongside the attributed ROAS. This gives finance the confidence interval they need: the true value is somewhere between the incremental (floor) and attributed (ceiling) figures.
Next Steps
Running your first holdout test provides evidence that transforms budget conversations from trust-based to evidence-based. It's the foundation for confident scaling decisions.