Email A/B testing is easy to start and hard to do well. Here’s the framework for tests that actually improve performance.
What to Test (Prioritized)
Highest impact: subject lines (affects open rate, which gates everything else). High impact: send time/day, preview text, from name. Medium impact: CTA text, email length, image vs text-only hero. Lower impact: colors, fonts, button size. Spend most testing effort on subject lines and send times — they move the biggest dial. Don’t waste list tests on minor design choices until you’ve optimized the fundamentals.
Sample Size and Statistical Significance
Don’t call a winner on 200 sends. Minimum for reliable email tests: 1,000–5,000 opens per variant (not sends — opens). For most lists that means 10K+ recipients per variant. Smaller lists need multiple tests averaged together. Use your ESP’s significance calculator; avoid ‘it’s winning, let’s ship it’ at 60% certainty — you’re often wrong.
One Variable at a Time
A/B tests must test one variable only. Changing subject line AND send time in the same test makes results uninterpretable. If you want to test multiple variables, run sequential tests or use multivariate testing (requires much larger lists). The discipline of one-variable-at-a-time is what separates rigorous testing from vibes-based marketing.
Test Metrics Beyond Opens
Subject line A might win on open rate but lose on conversion rate if it attracts curious clickers who don’t buy. Always measure downstream: open rate → click rate → conversion rate → revenue. Final measure is revenue per send, not opens. A ‘winning’ subject line that drops conversion rate is a losing test.
Common Mistakes
(1) Testing too many things — pick 3 big tests per quarter, not 30. (2) Testing with tiny samples and trusting results. (3) Calling a test after 1 hour — test for the full email life cycle (48+ hours for most sends). (4) Not controlling for day-of-week effects — emailing Monday vs Friday invalidates some tests. (5) Testing design theories your audience has already told you the answer to (open-rate data on last 90 days of subject lines tells you more than any test could).
What Winning Looks Like Long-Term
Maintain a test log: hypothesis, result, winning percentage, sample size. Over 50+ tests, patterns emerge specific to your audience. You’ll find your audience prefers short subject lines, or questions, or specific numbers. These patterns become templates for all future sends. That accumulated knowledge, more than any single test, is the real payoff of rigorous A/B testing.
Need help with email marketing?
Get a free audit of your email marketing setup. We’ll show you exactly where the opportunities are.
Get Free Audit →