How to run a marketing experiment with a holdout group
A customer came back after a reminder. Would they have come back anyway? A holdout group helps you decide whether sending the next reminder is worth it.
Start with the decision you would change
Suppose a team wants to bring back customers who skipped their usual weekend purchase. Write down the decision before picking the copy. Will you send a reminder every weekend if it brings back enough people to pay for itself? Choose one measure, such as the share of eligible customers who buy within seven days. Set the smallest worthwhile lift and the most opt-outs you can accept.
Give some eligible customers the usual experience
Choose the audience before sending anything. Randomly assign each eligible customer to a group and keep that assignment for the test. One group gets the reminder. The holdout gets the usual experience without this extra message. Both remain eligible for the same prices and offers.
A test of two messages can tell you which version performs better. Adding the holdout lets you answer a second question: did sending either message help? If the audience is small, test one message against the holdout first. Use your usual purchase rate and the smallest worthwhile change to estimate how many customers you need. Set the end date before looking for a winner. Firebase’s guide to experiment results
Calculate the lift using everyone assigned
Suppose 4,000 eligible customers are split evenly. Count each customer once if they buy during the seven-day window.
| Group | Customers who bought | Purchase rate |
|---|---|---|
| Reminder | 220 of 2,000 | 11% |
| No reminder | 200 of 2,000 | 10% |
The difference is one percentage point. Relative to the 10% holdout rate, that is 10% lift. Count everyone assigned, including people who never opened the message. Comparing only openers gives the reminder credit for people who were already more likely to buy.
For this example, an approximate 95% interval for the difference runs from a 0.9-point loss to a 2.9-point gain. At the planned end, weigh that uncertainty alongside the cost of sending and the opt-out rate. If the result is consistent with both a loss and a worthwhile gain, keep the existing policy while deciding whether a larger test is worth running.
What a Lyft driver test could tell the team
In a February–March test during prior in-house work at Lyft, eligible drivers had earned at least $100 one weekend and skipped the next. A third got a reminder of their prior earnings, a third saw how close they had come to a higher amount, and 34% received no message. Neither message included a bonus. The question was whether wording alone could bring drivers back.
Driver hours were 0.72% higher for the near-miss message than for the holdout (p = 0.080). For the earnings reminder, the difference was 0.56% (p = 0.176). Both estimates pointed upward, but neither comparison met a 5% significance threshold. The figures also don’t establish that one message beat the other. A larger test could inform a rollout before the team committed to automating the messages. Read the Lyft driver retention experiment