Decide what a win means before the test goes live

Decide what a win means before the test goes live

The single most common way an experiment lies to you is that nobody set the bar before launch. The test runs, a number comes back, and the team gathers to interpret it. By then it is too late: whatever the number is, someone will build a case that it is good, because no one wants to have wasted the week. The only defence is a threshold written down before the test is live.

So before launch, commit three numbers. The metric that decides it, the value that means ship it, and the value that means kill it. "Demo requests per hundred visitors, ship at three or above, kill below two" leaves no room to negotiate the result afterwards. The grey band between, where you neither clearly won nor clearly lost, is honest information too: it usually means the effect is too small to matter, which is itself a decision to move on.

Pre-committing the threshold also forces you to size the test properly. If you need a two-point lift to call it a win, you can work out how much traffic you need to see that lift through the noise. Teams that skip this run tests on a trickle of traffic, watch the number wobble, and read the wobble as a result. Decide the bar and the sample size together, and you stop fooling yourself with random variation.

The threshold is not just for the winners. A clear kill number is what gives you permission to stop, and stopping is where most of the discipline lives. Without a kill line, losing tests linger, get "one more week", and quietly consume the throughput you needed for the next idea. The kill number is a gift to your future self: it ends the argument before the emotion shows up.

Write the threshold into the experiment ticket, not into someone's memory. The moment it lives only in a head, it drifts, because the head has a stake in the outcome. A number on the ticket from before launch is a number you cannot quietly move, and that immovability is the entire reason it works.

INTERVIEW EWOUD: Describe a time a pre-set win threshold stopped you from talking yourself into a result, or a time you skipped it and paid for it. What was the threshold, and what would have happened without it?