Most A/B test post-mortems focus on the wrong moment.
Teams look at results, see an inconclusive p-value or a suspiciously large lift, and ask: what went wrong? The answer is almost never in the results. It’s in the setup — the decisions made before a single impression was served.
Here’s where the failure actually happens.
The test was underpowered from day one
Statistical power is the probability that your test will detect a real effect if one exists. Most teams ignore it entirely.
The consequence: you run a test for two weeks, get a p-value of 0.08, call it inconclusive, and move on. What you don’t know is that you needed four times the sample size to have any reasonable chance of detecting the effect you were looking for.
Running an underpowered test isn’t neutral. It’s a waste of the time and traffic you spent on it, and it produces data that will mislead you anyway — because small samples produce unstable estimates that swing wildly from test to test.
Before any test runs, calculate the required sample size. You need three inputs: your baseline conversion rate, your minimum detectable effect (the smallest lift worth caring about), and your desired power level (80% is standard). If you can’t hit those numbers with your current traffic in a reasonable timeframe, you either need to increase traffic, widen your MDE, or kill the test before it starts.
That last option — choosing not to run a test — is an underrated decision. Not every question deserves an experiment.
The hypothesis was reverse-engineered from a hunch
A real hypothesis has a structure: If we change X, we expect Y to change, because of Z.
Most “hypotheses” in A/B testing are really just design preferences with post-hoc behavioral justifications attached. Someone on the team wants a bigger CTA button, so the hypothesis becomes “larger buttons are more visible and will increase clicks.” That’s not a hypothesis. That’s an instruction with a label on it.
The behavioral reasoning matters because it determines whether a result teaches you something generalizable or just tells you that this specific button on this specific page in this specific week performed slightly differently. Results without causal logic are single-use data points. Results that confirm or refute a behavioral model are assets you can apply across the entire funnel.
Write the hypothesis before you design the variant. If you can’t explain why the change should work in behavioral terms — attention, cognitive load, trust signals, decision friction — you’re guessing with extra steps.
You’re testing cosmetic changes against structural problems
There’s a category of A/B test that’s doomed to fail regardless of how well it’s set up: the test that optimizes surface details on a fundamentally broken experience.
If your landing page has a trust problem, changing the headline color won’t fix it. If your checkout flow has too many steps, a button copy test will move the needle by fractions of a percent while the structural issue bleeds conversion at scale.
This is where the prioritization framework matters more than the testing tool. Before running any test, answer two questions: what is the biggest drop-off point in this funnel, and what is the most likely behavioral reason for it? Start there. Save the cosmetic tests for when the fundamentals are solid.
The highest-ROI A/B tests are almost always the ones that feel too risky to run. Structural changes to pricing pages, radical simplifications of checkout flows, entirely rewritten value propositions. They’re high-variance and uncomfortable. They’re also the ones that produce 15–40% lifts instead of 1–3%.
Statistical significance is being used as a decision rule
This one is subtle and it’s everywhere.
Statistical significance — the p-value threshold, typically 0.05 — tells you one thing: the probability of seeing a result at least this extreme if there were actually no difference between the variants. That’s it. It doesn’t tell you whether the effect is real. It doesn’t tell you whether the lift is meaningful. It doesn’t tell you whether you should ship the variant.
The field has a replication crisis partly because teams (and researchers) treat “p < 0.05” as a binary pass/fail gate instead of one input into a broader decision. The result is publication bias in the worst case, and bad product decisions in the typical case.
Use significance as a filter, not a verdict. Pair it with effect size, confidence intervals, and business impact. A 0.3% lift at p = 0.04 on a page that gets 500 visits a month is not actionable. A 4% lift at p = 0.06 on your primary acquisition channel probably is. Context is the decision-making layer that statistics can’t provide.
There’s no measurement system — just a testing tool
Most teams conflate having a testing platform with having a testing practice.
A testing platform (Optimizely, VWO, AB Tasty, whatever) is infrastructure. A testing practice is a system: a prioritized backlog of hypotheses, a documented methodology for calculating sample sizes and runtime, a clear decision framework for reading results, and a repository of past tests that the whole team can learn from.
Without the system, tests run in isolation. Nobody knows what was tested six months ago or why. Variants that lose get discarded without extracting the learning. The same hypotheses get proposed again and again by different people who weren’t in the room.
The fix isn’t complicated, but it requires discipline: document every test with a standard template (hypothesis, baseline metric, required sample size, runtime, result, interpretation, next question). Build the backlog the same way you’d build a product roadmap — weighted by expected impact and cost of implementation, reviewed regularly, adjusted based on what you learn.
That’s what separates teams that run 50 tests a year and learn from them from teams that run 50 tests a year and spin in place.
The failure is almost always upstream. By the time you’re looking at results, the decision about whether this test could ever tell you anything useful was already made — weeks ago, in a planning conversation most people didn’t take seriously enough.
Start there.