Skip to content
Published 2026-11-18•3 min read•By Levered

How to run a marketing experiment with a holdout group

A customer came back after a reminder. Would they have come back anyway? A holdout group helps you decide whether sending the next reminder is worth it.

Marketing experimentsLifecycle marketing

Start with the decision you would change

Suppose a team wants to bring back customers who skipped their usual weekend purchase. Write down the decision before picking the copy. Will you send a reminder every weekend if it brings back enough people to pay for itself? Choose one measure, such as the share of eligible customers who buy within seven days. Set the smallest worthwhile lift and the most opt-outs you can accept.

Give some eligible customers the usual experience

Choose the audience before sending anything. Randomly assign each eligible customer to a group and keep that assignment for the test. One group gets the reminder. The holdout gets the usual experience without this extra message. Both remain eligible for the same prices and offers.

A test of two messages can tell you which version performs better. Adding the holdout lets you answer a second question: did sending either message help? If the audience is small, test one message against the holdout first. Use your usual purchase rate and the smallest worthwhile change to estimate how many customers you need. Set the end date before looking for a winner. Firebase’s guide to experiment results

Calculate the lift using everyone assigned

Suppose 4,000 eligible customers are split evenly. Count each customer once if they buy during the seven-day window.

GroupCustomers who boughtPurchase rate
Reminder220 of 2,00011%
No reminder200 of 2,00010%

The difference is one percentage point. Relative to the 10% holdout rate, that is 10% lift. Count everyone assigned, including people who never opened the message. Comparing only openers gives the reminder credit for people who were already more likely to buy.

For this example, an approximate 95% interval for the difference runs from a 0.9-point loss to a 2.9-point gain. At the planned end, weigh that uncertainty alongside the cost of sending and the opt-out rate. If the result is consistent with both a loss and a worthwhile gain, keep the existing policy while deciding whether a larger test is worth running.

What a Lyft driver test could tell the team

In a February–March test during prior in-house work at Lyft, eligible drivers had earned at least $100 one weekend and skipped the next. A third got a reminder of their prior earnings, a third saw how close they had come to a higher amount, and 34% received no message. Neither message included a bonus. The question was whether wording alone could bring drivers back.

Driver hours were 0.72% higher for the near-miss message than for the holdout (p = 0.080). For the earnings reminder, the difference was 0.56% (p = 0.176). Both estimates pointed upward, but neither comparison met a 5% significance threshold. The figures also don’t establish that one message beat the other. A larger test could inform a rollout before the team committed to automating the messages. Read the Lyft driver retention experiment

Talk through your marketing priorities

Start with a website check to see what needs attention and where Levered could help.