FAQs

FAQs

How many experiments should we run at once? As many as you can read cleanly, which for most small teams means a small handful in parallel, not dozens. The limit is not ideas, it is your ability to keep each test's variable clean and its result honest. If two experiments touch the same surface and the same metric, they interfere and you cannot trust either read. Run as many as you can keep isolated, and no more.

Do I need statistical significance, or is that overkill for a small team? You need enough volume to trust the result, which is the practical core of significance, but you do not need a statistics degree to get there. The honest version is: decide your sample size and run length before launch, run across at least one full weekly cycle, and do not stop early on a peek. If your traffic is genuinely tiny, accept that you can only test changes large enough to show through the noise, and test bigger swings rather than fooling yourself with small ones on thin data.

What if I do not have enough traffic to run proper experiments? Then test bigger changes, fewer of them, and lean harder on qualitative signal. Low traffic means small effects are invisible, so testing a button colour is pointless, but testing a fundamentally different offer or message can still show through. Pair the quantitative tests you can run with direct buyer conversations and recorded sessions, which give you signal that does not need a large sample. Low traffic is a reason to change what you test, not a reason to stop.

How do I know whether to test a small tweak or a big change? Let the constraint and the expected value decide. If a stage is leaking badly, a big structural change to that stage is worth the build. If a stage is roughly fine, only a cheap test can earn its place there. As a rule, probe cheaply first to find where the signal is strong, then commit a big build only where the small tests have already shown the path. Big bets earned by evidence are good, big bets placed on a hunch are how quarters get burned.

What is a healthy win rate for experiments? Well below half, and that is the point. If most of your tests win, you are only testing the obvious and leaving the real wins untouched. A hit rate around one in three or one in four usually means you are testing genuinely uncertain ideas where the big lifts live. A high win rate is a warning sign that the programme has gone timid, not a badge of skill.

How is this different from just doing A/B testing? A/B testing is one tool inside experimentation, the mechanism for comparing two variants cleanly. Experimentation is the whole discipline around it: forming hypotheses, ranking a backlog by expected value, setting thresholds before launch, running long enough to trust the read, and logging every lesson so the system compounds. You can run A/B tests for years and learn little if the surrounding discipline is missing. The test is the easy part, the system around it is where the leverage is.