A/B Testing Emails in Sitecore EXM
Quick Links:
This tutorial is Part 6 of a mini-series on Sitecore's EXM functionality.
EXM Part 1 - Intro to Sitecore EXM (FTFM)
EXM Part 2 - Creating Contact Lists in Sitecore
EXM Part 3 - Creating an Email Campaign in Sitecore
EXM Part 4 - Crafting your message in EXM
EXM Part 5 - Personalise your message in EXM
Transcript
Send a quick test of an email campaign
To make sure that you can successfully send and receive a specific email campaign, in the Email Experience Manager (EXM), you can send the email campaign to a few known email addresses, such as your own or a co-workers email address. In this way, you can quickly verify that the email campaign layout looks good in your own email client, and that spam filters don’t block your email campaign.
To send a quick test of an email campaign:
1. In EXM, open the relevant email campaign and click the "Review" tab.
2. Click "Send quick test" and enter one or more email addresses in the field. Separate the email addresses with a comma.
3. If your email message has multiple variants, you must first select the variants that you want to send in the quick test.
4. Click "Send" and review the email message in the relevant email inbox.
OK, so you’ll have noticed that I mentioned the word “Variant” there, but that’s a topic we haven’t covered yet. In Sitecore it's possible to set up an A/B Test for your email campaigns, and we call these multivariate tests because you’re not just limited to an "a" and a "b" version, you can test a "c,d,e,f,g" version too, if you’d like!
Why use A/B Testing for emails?
Before we dive in to show you how the a/b testing functionality works, let's take a second to talk about why we would bother setting up an AB test in the first place.
You see, the success of your email campaign is highly dependent on how well you know your customers. When you create an email campaign, you base the content and layout of the message on an assumption.
But what if your assumptions aren’t accurate...recipients might not respond to your campaign in the way you expect them to.
With A/B testing, you can test and validate your assumptions on a small group of your customers before you send the final email campaign to the rest of your mailing list. In that way, you can ensure that you always send the email campaign that statistically attracts the most customers and generates the most value for your business.
How to create a message variant
When you run an A/B test, you test different variations of the same message. For each variation in the message that you want to test, you need to create a new variant of the original message. The original message is the first variant (Variant A), and any new variants that you create is named Variant B, Variant C, and so on.
To create a new variant of a message:
1. Open the message that you want to perform an A/B test on.
Important
A/B tests cannot be performed on automated messages or on these message types: Simple HTML message, Plain text message, or Existing page.
2. You can choose to add a new variant or duplicate the original variant and all its content:
o To add a new variant, on the Message tab, click "Add Variant".
This creates a new message variant based on the same template as the original message but where no content is replicated. Choose this option if you want to test comprehensive differences between the message variations, such as larger components or the layout of the message.
o To duplicate the original variant, click "Actions" and then "Duplicate this variant". This creates a new variant of the original message by duplicating it and all of its content. Choose this option if you want to test minor differences between the variants, such as the subject line, images, or the name of a button.
3. The message variants are displayed on individual tabs where you can easily toggle between the variants.
You can create as many variants of the message as you like, but be aware that when you add more variants, you need a larger pool of recipients for the test results still to be clear and reliable. You should test a maximum of two to three variants.
4. Make the necessary changes to the new message variant, for example, you can change the message layout, change the images, or add new components.
Note
If you base your A/B test message on an imported HTML template, only the Variant A contains the layout. For the other variants, you have to add the message layout manually.
5. When you have created all the message variants for the test, click Save.
Start an A/B test
After creating the message variants for the A/B test, you need to determine how you want to send the messages.
To send an A/B test:
1. Open the relevant message and on the Delivery tab, click the variants that you want to include in the delivery. In this example, the variants A and B are selected.
2. In the Size of the test drop-down menu, select the percentage of recipients that you want to include in this test. This set of recipients is randomly generated from the full list of recipients. Generally speaking I’d recommend sending each variant to about 150 people (so 300 people total if youre running 2 variants) and we do that in order to get more reliable test results. So the figure you choose here will depend on the size of your contact list.
3. Under Select winner, specify how you want the winner to be chosen. Select one of the following options:
o Automatic – the system selects the winning variant for you, based on either best value per visit or highest open rate. In the Automatically select winner after fields, specify how long you want the test to run before the winner is selected.
If you click Best value per visit, the winning variant is the one generates the most value on the website per visitor.. The value is calculated according to the engagement value points that are set up on your website.
If you click Highest open rate, the winning variant is the one that was opened most often.
o Manual – you review the reports of the A/B test and select the winner variant based on your own criteria.
4. Under Schedule delivery, select:
o Send message now to start the A/B test immediately.
o Schedule message delivery to schedule the delivery of the A/B test message for a later time. Specify the date and time for when you want to start the A/B test.
o Schedule a recurring delivery to schedule the A/B test message to send recurrently at a certain interval. Specify the interval, the date, and time for when you want to send the recurring A/B test.
Note
When you schedule a recurring delivery of an A/B test message, a recurring message is only sent if the winner of the previously sent test message has been selected, either automatically or manually.
5. If you want to send a notification to certain email addresses when the dispatch of the A/B test has completed, under Notification, select the check box and enter the relevant email addresses. If you chose to have the winning variant selected and sent automatically, the notification is sent when the final message has been sent.
6. If you want to send the message in the recipients' preferred language, click the More link and under Multi-language, select Use preferred language.
7. Click Start A/B test (or click Schedule test if you scheduled the A/B test).
Important
If you select Use preferred language in the Multi-language section, be sure to verify the default language in the Message delivery confirmation dialog that appears.
The A/B test starts and changes the status of the message to Sending. When the system finishes sending the message to the limited set of recipients, the button’s label changes to "Resume" but remains inactive until the winning variant is selected.
If you choose to schedule the A/B test, you can cancel the message up until the date and time it is sent, by clicking Cancel scheduling on the Delivery tab of the message.
Select and send the winning message
You can choose to select the winning message manually or have it sent automatically to the rest of the recipients in the recipient list. The winning message will not be sent to any of the recipients that were included in the A/B test.
Automatically select the A/B test winner
If you choose to have the winner selected automatically, when the specified time has passed, EXM dispatches the winning variant automatically to the rest of the recipients in the recipients list.
If the result of the A/B test is even, the system automatically selects Variant A as the winner and sends it to the rest of your recipient list.
Note
You should always review the reports of the A/B test. It is a great opportunity for you to learn more about your customers and their preferences.
Manually select the A/B test winner
To select the winner manually, you should review the A/B test reports yourself and select the variant that you think is the best. Make sure that you give your recipients enough time to read and react to the message.
To select the A/B test winner manually:
1. In the navigation menu, under Email campaigns, click "In progress".
In the list, you can see the status of the message. The messages with the "Select winner" status are ready for you to select a winner.
2. Open the message and under the Message Preview section, click the "Full message" link.
3. Click the Delivery tab and under Select winner, you can see the results of the A/B test so far.`
Each variant is listed with the current statistics:
o Open rate – the percentage of recipients that have opened the message.
o Click rate – the percentage of recipients that have clicked a link in the message.
o Value – the number of engagement value points that the recipients have generated on the website after opening the message.
4. Based on these results, determine which variant you would like to send to the rest of the message recipients and click Choose as winner.
5. When you are ready to send the winning message to the rest of the recipients, click Resume.
Need a helping hand?
Call 01189 314196 today to get in touch with 3chillies - the Experienced Technology Experts.
