When to Test and when to Regress

Here’s a quick primer on when you can use a test vs. control (or experimental design) and when you should go for a fully specified regression model.

Test when you:

  • Need just one answer: You get only a measurement of the impact for which you test.
  • Need the answer quickly: Testing offers immediate and fairly constant feedback.
  • Are testing something crazy or expensive: You can test in small markets to limit risk.
  • Have suitable controls available: With a volatile or unstable businesses, testing isn’t really an option.
  • Believe the tested factor acts independently from other factors: If there are synergies or other interactions, the test is at risk.

Regress when:

well, none of the above are true, and…

  • You have data, time, and funding: To get high levels accuracy requires a lot of each, but it can pay for itself many times over.
  • Imprecise won’t do it: If you’re making big important decisions, precision matters.

Many of our clients have successfully used both an annual marketing mix modeling program alongside of regular testing capabilities for smaller in-market or heavy-up tests. ┬áTesting capability can usually reside in an internal team’s capabilities, while advanced modeling is usually best done externally.