Common failures
Common failures
Calling a tweak an experiment. Changing a colour or a word without a written hypothesis, threshold, or single variable produces activity that looks like testing and teaches nothing. It happens because tweaks are easy and feel productive. Avoid it by refusing to launch anything without a one-sentence hypothesis, a metric, and a pre-set bar, so every test can actually be wrong.
Changing several things at once. When the headline, image, and call to action all move together and the number lifts, you have learned that something worked, which you cannot repeat or explain. It comes from impatience to ship improvements. Hold to one clean variable per test, even when bundling feels faster, because an unrepeatable win is barely a win.
Stopping the moment it looks good. Early data is mostly noise, and a variant that leads on day two leads by luck as often as by merit. Teams stop early because the good number is exciting. Fix it by committing the run length and sample size before launch and leaving the test alone until it reaches them, with the daily number getting no vote.
No threshold set before launch. Deciding afterwards whether a result is good guarantees every result becomes a win, because someone will always find the story. It happens because writing the bar first feels pedantic. Commit the ship number and the kill number in writing before the test is live, in the ticket, where you cannot quietly move them.
Testing where the funnel is already fine. A clever experiment on a stage that is not the constraint is wasted effort, no matter how well it runs. It comes from working on what is fun rather than what is binding. Score every idea by the stage it relieves and point the testing at the bottleneck, even when the bottleneck is the boring stage.
Losing the lessons. Without a durable log, the same failed idea gets proposed, rebuilt, and re-lost every few months at full cost. It happens because writing things down is the easiest task to skip when busy. Make the log entry part of closing every experiment, win or lose, so capturing the lesson is the work rather than an afterthought.