Here’s a quick primer on when you can use a test vs. control (or experimental design) and when you should go for a fully specified regression model.
Test when you:
- Need just one answer: You get only a measurement of the impact for which you test.
- Need the answer quickly: Testing offers immediate and fairly constant feedback.
- Are testing something crazy or expensive: You can test in small markets to limit risk.
- Have suitable controls available: With a volatile or unstable businesses, testing isn’t really an option.
- Believe the tested factor acts independently from other factors: If there are synergies or other interactions, the test is at risk.
Regress when:
well, none of the above are true, and…
- You have data, time, and funding: To get high levels accuracy requires a lot of each, but it can pay for itself many times over.
- Imprecise won’t do it: If you’re making big important decisions, precision matters.
Many of our clients have successfully used both an annual marketing mix modeling program alongside of regular testing capabilities for smaller in-market or heavy-up tests. Testing capability can usually reside in an internal team’s capabilities, while advanced modeling is usually best done externally.