Creating Static A/B Testing Email Campaigns in MoEngage

In this article, you will learn how to

  • create A/B and Multivariate email campaigns to experiment with and improve your conversion rates.
  • use control groups to measure conversion impact of a specific change
  • Figure out the best way to get your audience to engage better with your campaigns

Before we dive deeper into creating and editing multivariate email campaigns, you might want to know what multivariate testing is.

  1. Creating Multivariate Experiment & Adding Control Group
  2. Editing Multivariate Experiment

Creating Multivariate Experiment

Creating a multivariate experiment for Emails is very simple and intuitive. Let us take a sample experiment:

Sample Hypothesis: Having an emoji in the email subject line gives better CTRs

Experiment: Create two email message variations and use emoji in Variation 1 Title and send plain message in Variation 2 Title

Measure: Compare CTRs of Variation 1 & Variation 2 . If CTRs of Variation 1 are better than Variation 2, there are high chances you have a valid hypothesis

Step 1: Choose the target audience

Picture4.pngStep 2: Messaging of Email campaign

  1. Create Message variations
  2. Decide User Distribution among variations

Create Message Variations

Variation 1: With Emoji

Picture1.pngOnce done with Variation 1, you can Add another Variation to create A/B Test by clicking:Picture11.png

You can add a new as well as copy from existing variations. Also, you can use any field under the Compose Email tab (e.g. Subject, Sender name, From address, message content) to create variations for your campaign.

Picture5.png

For the current example, since we are only experimenting with subject line, we can use Copy Variation 1 which will copy all the payload fields from Variation 1. Change the subject for Variation 2 accordingly.

Variation 2: Without Emoji

Picture2.pngFor the example above, we have created only two variations to test only one variable - emoji. One can create up-to 5 variations to test one or more hypothesis.

Add Control Group

Read why you need a control group in your experiment. To add a control group, you can click on Add button.

Picture6.png

You can then set the percentage of users you want to add to control group (start your experiments with 2-5% allocated to control) and click ADD.

Picture8.png

Once set, you will see a confirmation like the one below. Once added, you can choose to remove or edit the control group split.

Picture7.png

Decide User Distribution among variations

Once the control group, and message variations are decided, you can then set the split across variations for the campaign. By default, the split is evenly distributed across the available number of variations. However, you can choose to change the distribution to your preference.Picture3.png

Important Note: The allocation of user to any of the group - test or control is non-sticky in nature i.e. for any active campaign, if users qualifies twice for receiving the campaign, we do not guarantee that if User A has received Variation 1 once, she would or wouldn't receive it again.

Complete Step 3 (Scheduling and Goal setting) and finish creating the campaign and measure results by comparing variation performances on Campaign Analytics.

You should be able to see analytics as below where you can compare variation performance and deduce the best performing variate.Picture9.png

You can also view a graphical comparison:Picture10.png

 

Editing Multivariate experiment

Campaign Type

Status

Allowed edit options

General Email

Scheduled but not sent yet

- Add/remove variations;

- Edit variation message ;

- Edit the distribution %age of variations

General Email - Periodic

Scheduled and not even 1 instance is sent

- Add/remove variations;

- Edit variation message ;

- Edit the distribution %age of variations

General Email - Periodic

Scheduled but at-least 1 instance is executed

- Edit variation message but cannot add/remove a variation;

- Edit the distribution %age of variations

Smart Triggers

Active/Paused

- Edit variation message but cannot add/remove a variation;

- Edit the distribution %age of variations

Note: This feature is currently in closed beta and hence is available only for few clients. If you wish to have access to this feature, please contact your Customer Success Manager.

Was this article helpful?
0 out of 0 found this helpful