DMA tests — experiments at the Designated Market Area level — are one of the most powerful tools for measuring marketing incrementality. They are also easy to screw up. Here are the lessons I paid for.
Why DMA tests
User-level attribution is narrow. It misses a lot: word of mouth, cross-device conversions, brand effects that show up weeks later.
DMA tests catch more of the full picture. You treat whole markets, not cookies, so you pick up:
- Brand lift that takes weeks to show
- Cross-device conversions
- Social and word-of-mouth spillover
- The full funnel from awareness to purchase
This is why I keep going back to them, even though they are a pain.
Common ways to mess this up
Not enough markets
Power in geo tests comes from the number of geo units, not total users. With 20 DMAs you need a huge effect to see anything.
Rule of thumb: at least 50 markets. Ideally 100. I know this limits the channels you can test. Underpowered tests are worse than no test — you get a non-answer you then defend.
Spillover between markets
People travel. Digital ads ignore borders. If your “holdout” market is leaking treatment, your estimate shrinks toward zero.
What I do: buffer zones, exclude border DMAs, or model the spillover explicitly. I have learned to be paranoid.
Ignoring seasonality
A November test tells you little about February. Marketing effects move with the calendar.
Run long enough to cover a cycle. Or use methods that handle seasonality directly.
Why I prefer synthetic control
Simple treatment-vs-control comparisons work sometimes. But markets are heterogeneous and trends differ by region, so the comparison is usually noisy.
Synthetic control builds a weighted mix of untreated markets that best matches the treated market before the campaign. Then you measure the gap after.
It handles:
- Different baselines across markets
- Region-specific trends
- Noisy outcomes
I get much cleaner reads from synthetic control than from naive DiD for geo tests.
What I would tell a new analyst
- More markets beats more users per market.
- Design for spillover from day one. Not as an afterthought.
- Pre-register the analysis. Stops you from p-hacking yourself later.
- Try synthetic control before DiD.
- Run a power calculation. Every time.
If you skip the power calc, you are setting yourself up for an inconclusive result you will then have to explain.