Transactional A/B Testing

Sailthru allows you to continuously improve your transactional emails by A/B testing different versions. What subject line is most welcoming on a Welcome email? Where should you place a link to get an invitee most likely to click to join your site? Using A/B testing, you can find out.

The basic idea of A/B testing is simple: create two (or more) different versions of your template, and determine which one gets a better response from your users (usually measured by click and open rate). Sailthru makes this easy.

Assuming you have already created one version of the template, pull it up in the Template Editor. Click the A/B Test… button and hit OK at the confirmation prompt.

Now two versions of your template will be created with different Sample names – one A and one B. You can change these Sample names if you like by editing the Sample field for each one.

These two template versions can be individually edited as you wish. Try out changes to the subject line and content.

Now that you have two versions, every time you send a transactional email using the send API call, Sailthru will pick randomly between the different A/B versions. Over time, you will be able to compare the results and determine the most effective template.

If you want to temporarily stop sending one version or the other without deleting it altogether, change the Status of the template from Active to Paused. This will cause Sailthru to always pick the other one. Be careful not to pause both versions, as this will cause Sailthru to send neither.

Just click on the Transactional A/B Testing Report link in the Reports section. You will see a chart, updated daily, that compares the different versions. Pick the version with the highest click and open rates and continue improving!

If you're satisfied that you've got the better template and you want to stop testing, just delete the less effective version. The remaining template will no longer be an A/B test and will function as normal.

To delete the unwanted template, select it from the drop down menu at the top of the page, and click the Delete button to the right.