How to Tell Whether an Intervention Worked, Without Fooling Yourself

Recovered revenue is the easiest number in e-commerce to inflate. A fair test needs a comparison, a cost line, and a window decided in advance.

Published 2026-10-12 · 6 min read

How to Tell Whether an Intervention Worked, Without Fooling Yourself

The Report That Always Looks Good

Every intervention tool ships with a results dashboard, and the dashboard is almost always positive. The pop-up was shown, some people bought, and the revenue appears under "recovered."

The weakness is in the word. Recovered assumes the sale would have been lost. Some of those buyers would have completed the purchase without any message, and some would have paid full price. The dashboard counts them as wins.

This is not dishonesty by the tool. It is a limit of the method: without a comparison, there is no way to separate the sales an intervention caused from the sales it merely accompanied.

Habit 1: Name The Decision Being Tested

"We added personalization" is not testable. A testable statement names one decision that changed: for visitors showing repeated interaction with freight information, show the delivery estimate earlier instead of sending a product discount after they leave.

One decision, one audience, one change. If several things change at once, a good result cannot be attributed and a bad one cannot be diagnosed.

Habit 2: Keep A Group That Does Not Receive It

The only reliable way to know what would have happened is to let it happen for some visitors. Hold back a share of eligible traffic, chosen at random, that continues to receive the previous treatment.

A before-and-after chart is not a substitute. Between last month and this one, the season changed, a campaign ran, and a competitor adjusted prices. Any of those can produce a lift or hide one.

The size of the group depends on traffic. What matters more is that the split is random and decided before looking at results.

Habit 3: Measure What It Cost

Conversion is half of the result. The other half is what was paid for it. For an e-commerce intervention, the cost side includes:

Discount spend per order. A lift bought with deeper discounts may be a loss.

Margin per order. The number that tells you whether the extra orders were worth having.

Returns and cancellations. Orders pushed by urgency come back more often.

Support contacts. An intervention that creates confusion moves the cost to another team.

Repeat purchase. Whether these customers return, and whether they now wait for an offer.

These are guardrails. An intervention that improves the primary result while a guardrail deteriorates has moved a cost somewhere else.

Habit 4: Decide The Window Before You Start

A result measured after three days and a result measured after six weeks can point in opposite directions. Returns take time to arrive, and repeat purchase takes longer.

Write down in advance the primary result, the guardrails, and the date on which the test will be read. Deciding when to stop after seeing the numbers is the most common way to turn noise into a finding.

What To Record For Each Intervention

A test is only as good as its log. For each decision, keep:

who was eligible,

which signals were available,

which action was chosen, including no action,

the confidence level,

the outcome within the window.

Recording the decisions not to act matters as much as the rest. They are part of what is being tested.

Reading Our Own Numbers The Same Way

We apply this standard to our own claims. The figures we publish, a 40% reduction in churn and a 2.3x increase in conversion across more than 3.4 million profiles, are results reported by a partner pilot. They are not a published controlled study, and they are not a guarantee for another store.

That distinction is the point of this article. A lift is a starting place for a question. It becomes evidence when there is a comparison, a cost line, and a window that was set before anyone saw the result.

Read our measurement framework.