Goal: Another research partner wanted to increase the open rate of its email alerts. Subscribers opted to receive these email alerts about SEC filings from various companies.
Primary research question: Which subject line will produce the highest e–mail open rate?
Approach: This test ran 10 subject lines in pairs each day of the week. The same pairs were run on the same day of the week throughout the test to mitigate validity threats. We tested a number of stylistic variations in the subject line to identify the strongest styles and formats.
Other considerations: The test measured the open rate, clickthrough rate, and rate of email recipients that visited the site on the same day (conversion rate).
Email Subject Line Test Structure
Test pairs were determined by style variables:
- Formal vs. informal
- With research partner’s name vs. without
- Word placement (beginning vs. end)
- Urgency vs. no urgency
- Impersonal vs. personal
The Email Subject Lines We Tested
This is how the subject line copy was formatted with the customer–selected information shown in brackets:
1*: SECFilings.com Alert for: [Company Name]: [Form Type]
2: Your Alert for [Company Name]: Form [Form Type] now available at SECFilings.com
3: SECFilings.com Alert: [Form Type] report for [Company Name]
4: [Company Name] Alert: [Form Type] report from SECFilings.com
5: [Company Name] [Form Type] Alert
6: Just Filed: [Company Name] Form [Form Type] on [Form Date]
7: Alert for: [Company Name]: [Form Type]
8*: SECFilings.com Alert for: [Company Name]: [Form Type]
9: You asked us to tell you when [Company Name] filed form [Form Type]
10: Your notification of [Company Name] filing form [Form Type]
* SL1 & SL8 both used the control subject line, due to the variation pairings
Here are examples of how these subject lines would actually read if a subscriber had elected to receive 8–K alerts for Apple:
1*: SECFilings.com Alert for: Apple: 8–K
2: Your Alert for Apple: Form 8–K now available at SECFilings.com
3: SECFilings.com Alert: 8–K report for Apple
4: Apple Alert: 8–K report from SECFilings.com
5: Apple 8–K Alert
6: Just Filed: Apple Form 8–K on 8/26/2008
7: Alert for: Apple: 8–K
8 *: SECFilings.com Alert for: Apple: 8–K
9: You asked us to tell you when Apple filed form 8–K
10: Your notification of Apple filing form 8–K
Results
What you need to understand: SL7, an alert that didn’t use the partner’s name, outperformed the control by 84.05% in open rate. SL4, containing alert information and research partner name, outperformed the control by 44.74% in conversion.
Conclusions
Let’s examine how we can use these results to determine follow–up tests.
Two different subject lines gave the best performance for two different objectives. This requires further strategic analysis:
- What is the best objective, open rate or conversion?
- Determine the business impact of each
- Quantify value of the relative increase in open rate and conversion.
- Compare these results to determine the value of highest ROI impact.
Use follow–up tests to optimize further, dig deeper:
- Best Objective: Open Rate
- Test the best subject lines by open rate in each pair against the control to determine the strongest subject line; optimize by testing “from” line.
- Best Objective: Conversion
- Test the best subject lines by conversion in each pair against the control to determine the strongest subject line; focus on body copy elements.
Key Points:
- Keep an eye on objectives when testing. Example: Don’t let low open rates eclipse a higher conversion rate. Focus on prime movers that will increase ROI.
- Use follow–up tests strategically to gain deeper insights, rather than simply testing different word combinations.
- Identify and test distinct variations in headline styles, such as: personal vs. impersonal, formal vs. informal, urgency vs. no urgency, questions vs. declarative statements, long vs. short, headline case vs. sentence case, etc.
