We Ran a 7-Way Checkout A/B Test. Here's What It Taught Us About Testing Itself
Earlier this month we wrapped up a checkout experiment on a SaaS product I work on. Seven variants, a couple of weeks, a few hundred users through the funnel. The conversion lift was big — but honestly, the more valuable output was what the test taught us about running tests. Most of what's written about A/B testing is about picking what to test. Almost nobody talks about how easy it is to read the results wrong.
Here's what we learned, with the numbers roughed out so I'm not publishing anyone's playbook.
1. The default embedded checkout was our biggest loser
Our control was the payment provider's stock embedded checkout — the drop-in iframe that every SaaS starts with because it takes an afternoon to integrate. Every other variant wrapped the same payment form in our own checkout page, with our own framing around it.
The stock embed converted checkout-to-trial at roughly a third of the rate of the best custom variants. Not a 10% lift. Three-x.
The lesson isn't "iframes bad." It's that the default checkout is optimized for the payment provider's constraints, not your funnel. If you've never tested owning that page, you're probably donating conversions to a default you picked for convenience in week one.
2. How you frame the price matters more than the price
Every variant charged exactly the same amount. The only thing that changed was the story the page told around the number — what got emphasized, what got de-emphasized, what anxiety got addressed. The spread between the best and worst custom framing was still more than 2x.
We changed zero pricing and nearly quadrupled the step conversion versus control. Before you touch your price, test the sentence next to it.
3. Two variants "won." The primary metric couldn't split them — the funnel behind it could
This was the big one. Two different variants finished statistically significant on the primary metric (checkout → trial start), with near-identical results. On the primary alone, it was a coin flip.
So we pulled the downstream funnel for each arm: trial → paid conversion. One winner converted its trials at around 60%. The other leaked — closer to 48%. Same trial volume, meaningfully different revenue. One variant was attracting committed buyers; the other was attracting people who liked the framing but not the product.
If we'd shipped on the primary metric alone, we had a 50/50 chance of shipping the worse business outcome while celebrating a "significant win."
4. A "losing" variant can be a winning one wearing a disguise
In an earlier test on the same funnel, a variant lost its primary metric badly — double-digit percentage drop. We nearly killed it on the spot. The full breakdown showed it produced identical paid conversions, but shifted the mix heavily toward the higher tier. Fewer trials, same customers, more revenue per customer.
The rule we took away: never call a winner or loser from one metric. Always pull exposure → checkout → trial → paid, plus plan-tier and billing-interval mix, before deciding anything. A primary metric is a hypothesis about what matters, not a verdict.
5. Write your ship/kill rules before the test starts
Mid-test, every result looks like a story you can argue either way. We now decide the rules up front, roughly:
Ship if the primary is significant and end-to-end paid isn't negative and the trial-quality guardrail isn't negative.
Kill if paid or the guardrail goes significantly negative — regardless of how good the primary looks.
Extend if it's trending but the paid sample is still small.
Boring, mechanical, and it removes the temptation to squint at a chart until it says what you want.
6. Know when you're making a judgment call — and say so
We shipped this winner with fewer paid conversions per arm than our own framework asks for. That's a judgment call: the primary was decisively significant, the guardrails were clean, and waiting had a real cost. The point isn't that rules should never be bent — it's that you should know you're bending them and write down why. A judgment call you've documented is a decision; a judgment call you've hidden from yourself is a bias.
7. Shipping isn't the end of the experiment
The winner is live at 100%, the flag is deleted, and there's a date in the calendar ~30 days out to check refunds, disputes, and paid retention for the cohort. A checkout change that lifts conversion by attracting worse-fit customers will show up there, and if it does, we roll back. A ship isn't permanent until it survives contact with a full billing cycle.
The meta-lesson
The variant we tested mattered less than the discipline around it. Same test, read lazily, ships the wrong winner half the time. If you take one thing from this: your primary metric tells you which page users prefer. Your downstream funnel tells you which page your business prefers. Read both, every time.