Every newsletter issue is sent with two subject line variants (A/B split) to a 10-20% sample of the list; whichever variant has a higher open rate after a 2-hour measurement window goes out to the remaining 80-90% of subscribers. A variant needs at least a 1-2 percentage point open-rate edge to be declared a winner — smaller gaps are treated as noise and the plan defaults to the plainer, more literal variant. Tests run on subject line only; sender name and preview text are held constant so results aren't confounded.
At a glance
| Parameter | Setting |
|---|---|
| Sample size for test | 10-20% of list, split evenly A/B |
| Measurement window | 2 hours post-send |
| Winning threshold | ≥1-2 pt open-rate difference |
| Variable tested | Subject line only |
| Rollout of winner | Remaining 80-90% of list |
| Fallback if no clear winner | Plainer/more literal variant |
Why subject lines get tested every issue
Subject line is the single biggest lever on open rate — before a reader ever sees the content, the subject line has already decided whether the email gets opened at all. Testing every issue rather than occasionally means the newsletter accumulates a large, current dataset on what actually works for this specific audience, instead of relying on generic email-marketing advice that may not hold for automotive content.
What gets varied between A and B
- Question framing vs statement framing ("Is your EV losing range in the cold?" vs "Cold weather can cut EV range 20-30%")
- Number-led vs benefit-led framing
- Length — short (under 40 characters) vs descriptive (50-65 characters)
- Urgency/curiosity framing vs plain descriptive framing
The test-and-rollout process
- Draft two subject line variants that differ in exactly one dimension from the list above.
- Send Variant A to one half and Variant B to the other half of a 10-20% test sample.
- Measure open rate for both variants after a 2-hour window.
- If one variant leads by 1-2 percentage points or more, send it to the remaining list; otherwise default to the plainer variant.
- Log the result (topic, variants, winner, margin) to the running subject-line dataset.
What the accumulated data has shown
Patterns that repeat often enough to treat as a working rule of thumb: subject lines naming a specific number (a range figure, a dollar amount, a percentage) tend to outperform vaguer equivalents, and question-framed subject lines perform better on deep-dive Friday issues than on the Tuesday deals digest, where a direct "X deals this week" statement usually wins. These are treated as current tendencies to keep testing against, not fixed rules — audience response shifts over time and each issue is still tested individually.
What this plan deliberately avoids
- Misleading or clickbait subject lines that don't match the email's actual content
- Testing more than one variable per split (confounds the result)
- Declaring a winner on a sample too small to be statistically meaningful
Subject line testing optimizes wording, not truthfulness — every winning variant still has to accurately describe what's inside the email. A subject line that wins the test but misrepresents content is rejected regardless of its open-rate lift.
Frequently asked questions
How is the winning subject line decided?
What happens if neither variant clearly wins?
Does the test change anything besides the subject line?
Are misleading subject lines ever used to boost opens?
Is there a difference in what works for different issue types?
Sources & further reading
- Email marketing benchmarks and testing guidance — Mailchimp
- CAN-SPAM Act compliance guide — Federal Trade Commission
Figures, prices and policy details were current at the last-updated date above. Automotive pricing, incentives and regulations change frequently — verify time-sensitive details with the linked primary sources. Read our editorial policy and fact-checking standards.
