A/B Testing & CRO: What to Test, How to Read Results, and What Most Teams Get Wrong
Photo by Arnold Francisca on UnsplashMost A/B tests are run wrong — not because people don't know the theory, but because they test the wrong things and read results too early. The biggest conversion lifts I've seen come from changing the offer or removing form fields, not from button colours. Run one variable at a time, wait for real statistical significance, and prioritise tests by how much they could plausibly move your number.
- ✓Change the offer or headline before touching any design element — that's where the real lift hides.
- ✓Peeking at results daily and stopping early is p-hacking — it produces false winners at a high rate.
- ✓Always segment by mobile vs. desktop — a winner on one can be a loser on the other.
- ✓A null result is still useful: it tells you that variable wasn't the bottleneck, so you can move on.
The test that cost two months and changed nothing
I watched a client's team spend eight weeks testing button colour on their primary lead form — green vs. orange vs. blue. They ran three sequential tests, ended each one when the dashboard turned green, and declared a winner. At the end of those eight weeks, conversion rate was exactly where it started. The button colour wasn't the problem. The form asked for nine fields, the headline was vague, and there was no proof anywhere on the page. They optimised a detail while ignoring the structure.
That's the most common CRO failure I see. Not bad testing mechanics — wrong priorities.
Test in this order
- 01The offer: 'Book a demo' vs. 'Get a free audit' on the same page can double your conversion rate. This is the highest-impact variable and the one teams test last.
- 02The headline: specific and outcome-focused beats clever and vague every time. '85% of our clients cut CPL in 90 days' beats 'Grow your business with better ads.'
- 03Social proof placement: a real testimonial with a name and company photo placed just above the CTA has moved conversion 20–40% for me more times than I can count.
- 04Form fields: remove every field you don't need at this stage. Phone number on a top-of-funnel form kills conversion. Get it on the confirmation call instead.
- 05CTA copy: 'Get started' is dead. Be specific about what happens next — 'Book your 20-minute audit' converts better because people know what they're agreeing to.
The mechanics: how to not fool yourself
Calculate your required sample size before you start — not after. A free tool like Evan Miller's calculator will tell you: for a baseline 4% conversion rate, detecting a 20% relative lift requires roughly 4,700 visitors per variant. If your page gets 200 visitors/week, that test takes 23 weeks. Most people don't run it that long. They stop at week three when one variant is 'up 15%' and ship the wrong winner.
- One variable per test — if you change headline and CTA simultaneously and get a lift, you don't know which drove it.
- Run at least two full business cycles (two weeks minimum) regardless of when you hit your sample size target — day-of-week effects are real.
- 95% confidence before calling a winner. Most tools default to showing results at lower thresholds.
- Segment results by mobile vs. desktop before shipping — I've seen tests that won on desktop by 25% and lost on mobile by 15%.
When you can't A/B test (low traffic)
Under 500 monthly conversions, you can't run valid A/B tests in any reasonable timeframe. The answer isn't to lower your confidence threshold — it's to use qualitative methods instead: session recordings (Hotjar, Microsoft Clarity), user interviews, and 5-second tests to identify friction. Then implement the highest-confidence change without a split test. Save A/B testing for pages that can reach significance.
The quality of your CRO programme is limited by the quality of your hypotheses, not your testing tool. If you can't explain why a change should work before running the test, you're just guessing with extra steps.
How long should an A/B test run?
Until you hit your pre-calculated sample size AND at least two full business weeks — whichever is longer. Never end a test just because the dashboard says 'significant.' Run the full sample. Day-of-week effects and external events (seasonality, promotions) can skew short windows badly.
What conversion rate lift is realistic from CRO?
Cosmetic changes: 0–5% relative lift. Copy and offer changes: 15–50% relative. Structural changes (removing form fields, changing the entire above-the-fold): 30–100%+. The ranges are wide because it depends entirely on how broken the current version is. A terrible page has more room to improve than a good one.
Which A/B testing tool should I use?
Google Optimize is shut down. For most businesses, VWO or Convert are the best paid options. Unbounce and Webflow have usable native testing for landing pages. For e-commerce, Shopify's native A/B doesn't exist — use a third-party like Neat A/B Testing or Intelligems. Whatever you pick, make sure it fires conversion events server-side so you're not missing iOS-blocked conversions in your results.
Want this kind of system built for your business? Let's talk — the first call is free, no pressure.