Here’s a quick primer on when you can use a test vs. control (or experimental design) and when you should go for a fully specified regression model.

Test when you:

• Need just one answer: You get only a measurement of the impact for which you test.
• Need the answer quickly: Testing offers immediate and fairly constant feedback.
• Are testing something crazy or expensive: You can test in small markets to limit risk.
• Have suitable controls available: With a volatile or unstable businesses, testing isn’t really an option.
• Believe the tested factor acts independently from other factors: If there are synergies or other interactions, the test is at risk.

Regress when:

well, none of the above are true, and…

• You have data, time, and funding: To get high levels accuracy requires a lot of each, but it can pay for itself many times over.
• Imprecise won’t do it: If you’re making big important decisions, precision matters.

Many of our clients have successfully used both an annual marketing mix modeling program alongside of regular testing capabilities for smaller in-market or heavy-up tests.  Testing capability can usually reside in an internal team’s capabilities, while advanced modeling is usually best done externally.