You know what’s more exciting than getting a regular award? Getting an award that’s shaped like a probability distribution.
We were so excited to be chosen as the winners of the Expy Awards this year, and to be the proud co-owners of the most adorable (and most nerdy) trophy that any of us have ever seen. We were even more excited about the experiment that won us the award–a pretty neat little test on the effectiveness of a welcome email that was designed by our data science team and the AFL-CIO, and analyzed by data science whiz Andy Zack.
Why’d we do it?
The idea behind the experiment was that lots of organizations currently send out welcome emails to new subscribers on their email lists. The theory is that the welcome email increases the new subscribers’ long-term engagement. Makes sense, right? The only problem with the theory was that we couldn’t find any scientifically rigorous experiments that had put that theory to the test. So… we ran one ourselves.
How the heck did the experiment work?
We worked together with our BFFs at the AFL-CIO on this experiment. We used a group of 4,540 new supporters on the AFL-CIO’s list. Each subscriber was randomly assigned to receive one of three emails:
- “Add your name: Tell Mitch McConnell to support sentencing reform” This was a previously used petition email that didn’t perform very well.
- “Sign now: Supreme Court vacancy must be filled!” This was a previously used petition email that had performed well.
- “Welcome—here’s your next step” This was that same Supreme Court email, but with a welcoming subject line and first sentence.
We sent off the emails, and then eagerly checked back on each group’s performance like the true nerds we are. We were looking to see how many actions the new supporters had taken (not counting the initial email action), and whether they had unsubscribed from the list at the end of nine months.
At the end of the nine months, we did some math.
(If you’re curious about exactly what kind of math we used, hit us up! We’d be happy to explain.)
So what’d we find out?
It turns out that including welcome language in the first email actually negatively affected both the number of actions that the new supporters took AND significantly increased the number of new supporters who unsubscribed. Yikes! Turns out people don’t really like it when you roll out the welcome mat for them.
Including welcome language lowered the number of long term actions per user by an estimated -16.4%, with a p-value of 0.035. As for unsubscribing, including welcome language decreased the percentage of supporters who remained subscribed after nine months by 4.2 percent.
Our theory is that including welcome language on an email alerts people that they just signed up for a new email list, and they might then think “I hate getting emails! Time to unsubscribe.” Of course, there are folks who think that unsubscriptions are a good thing–that they mean that the people who are on your list actually want to be there. If that’s your mindset, then a welcome email might not be a bad thing. But if you’re trying to aggressively grow your list, you might want to start off with a high-performing prior email instead.
Questions? Comments? Wanna tell us that you’re coming for our Expy next year? (Never!) Let us know in the comment section below.