An experiment is a bet you wrote down before you placed it
An experiment is a bet you wrote down before you placed it
Most teams call almost any change an experiment. They tweak a headline, swap a button colour, send a different subject line, and label the activity testing. It is not. An experiment is a specific thing: a written prediction about what will happen, tied to a number, decided in advance, so the result can prove you right or wrong. Without that written bet, you are not experimenting, you are redecorating and hoping.
The discipline starts with a hypothesis you can actually be wrong about. "Improve the landing page" is not a hypothesis, it is a wish. "Moving the pricing above the fold will lift demo requests because buyers are bouncing before they understand the offer" is a hypothesis, because it names the change, the predicted effect, the metric, and the reason. If the result comes back flat, that sentence told you something. The wish never could.
Writing the bet down before you run it is what protects you from your own storytelling. Human beings are extraordinary at explaining any result after the fact. Conversion went up, the new copy worked. Conversion went down, the market was soft that week. When the prediction lives in writing from before launch, you lose the licence to rewrite history, and that is precisely the point. The friction is the feature.
A good experiment also has a single variable you are actually testing. If you change the headline, the hero image, the form length, and the call to action all at once and the number moves, you have learned that something worked, which is almost useless. You cannot repeat it, you cannot explain it, and you certainly cannot teach it. One change, one prediction, one clean read.
The mindset shift is from "let us try some things" to "let us settle a question". Every experiment exists to answer a specific question about your buyers or your funnel, and the answer becomes a permanent asset whether the test wins or loses. A team that runs ten experiments learns ten things. A team that runs ten tweaks learns nothing it can name.
INTERVIEW EWOUD: Tell the story of an experiment you ran where writing the hypothesis down beforehand changed the conclusion, the result was something you would have explained away if you had not committed the prediction first. What was the bet, and what did the number actually say?