Creating A/B, Multivariate experiment for Push



MoEngage allows you to:

  • create A/B and Multivariate experiments to deliver messages that convert
  • use control group to measure conversion uplift due to notification 
  • analyze what works with your audiences

In this article we will talk about how you can create and edit a multivariate experiment for Push Notifications. Before this, you might want to know what multivariate testing is.

  1. Creating Multivariate Experiment & Adding Control Group
  2. Editing Multivariate Experiment

Creating Multivariate Experiment

Creating a multivariate experiment for pushes is very simple and intuitive. Let us take a sample experiment:

Sample Hypothesis: Having an emoji in message title give better CTRs
Experiment: Create two message variation and use emoji in Variation 1 Title and send plain message in Variation 2 Title
Measure: Compare CTRs of Variation 1 & Variation 2 . If CTRs of Variation 1 are better than Variation 2, there are high chances you have a valid hypothesis

Step 1: Choose the platform and target audience

Step 2: Messaging of Push campaign 

  1. Create Message variations
  2. Decide User Distribution among variations

Create Message Variations

Variation 1: With Emoji

Once done with Variation 1, you can Add another Variation to create A/B Test by clicking: 

You can add a new as well as copy from existing variations.

For the example, as we are only experimenting with title, we can use Copy Variation 1 which will copy all the payload fields from Variation 1. Change the title for Variation 2 accordingly.

Variation 2: Without Emoji

For the example above, we have created only two variations to test only one variable - emoji. One can create up-to 5 variations to test one or more hypothesis.

Add Control Group

Read why you need a control group in your experiment. Start your experiments with 2% allocated to control. To add a control group, you can click on Add button.

You can then set the percentage of users you want to add to control group (start your experiments with 2-5% allocated to control) and click ADD.

Once set, you will see a confirmation like the one below. Once added, you can choose to remove or edit the control group split.

Decide User Distribution among variations

Once done setting the message for variation,Click Next to set split percentage to distribute users among the variations. There are currently two ways to do that:

1. Static Multivariate Testing -  In this mechanism, marketer distributes users across variations as a fixed ratio e.g. sending 50% to Variation 1 and sending 50% to Variation 2 to test which variation performs best.

For the example below, we have divided the users 50% across each variation.


2. Dynamic Multivariate Testing (powered by Sherpa) - In this mechanism, the variation split is decided on the run-time as per variation performance till that time. Read how Sherpa dynamically optimizes your multivariate campaigns.


Important Note: The allocation of user to any of the group - test or control is non-sticky in nature i.e. for any active campaign, if users qualifies twice for receiving the campaign, we do not guarantee that if User A has received Variation 1 once, he would or wouldn't receive it again.

Complete Step 3 (Scheduling and Goal setting) and finish creating the campaign and measure results by comparing variation performances on Campaign Analytics.

You should be able to see analytics as below where you can compare variation performance, platform wise. The column with crown represents the Winner variation - the variation with maximum Click Through Rate.

You can also view a graphical comparison:


 Editing Multivariate experiment

Campaign Type


Allowed edit options

General Push

Scheduled but not sent yet

 - Add/remove variations;
 - Edit variation message ;
 - Edit the distribution %age of variations

General Push - Periodic

Scheduled and not even 1 instance is sent

 - Add/remove variations;
 - Edit variation message ;
 - Edit the distribution %age of variations

General Push - Periodic

Scheduled but at-least 1 instance is executed

 - Edit variation message but cannot add/remove a variation;
 - Edit the distribution %age of variations

Smart Triggers


 - Edit variation message but cannot add/remove a variation;
 - Edit the distribution %age of variations

Next in series: Sample experiments that you can create

Was this article helpful?
5 out of 5 found this helpful