Creating A/B, Multivariate experiment for Push

 

Uplift allows you to:

  • create A/B and Multivariate experiments to deliver messages that convert
  • use control group to measure conversion uplift due to notification 
  • analyze what works with your audiences

In this article we will talk about how using Uplift, you can create and edit a multivariate experiment for Push Notifications. Before this, you might want to know what multivariate testing is.

  1. Creating Multivariate Experiment & Adding Control Group
  2. Editing Multivariate Experiment

Creating Multivariate Experiment

Creating a multivariate experiment for pushes is very simple and intuitive. Let us take a sample experiment:

Sample Hypothesis: Having an emoji in message title give better CTRs
Experiment: Create two message variation and use emoji in Variation 1 Title and send plain message in Variation 2 Title
Measure: Compare CTRs of Variation 1 & Variation 2 . If CTRs of Variation 1 are better than Variation 2, there are high chances you have a valid hypothesis

Step 1: Choose the platform and target audience

Step 2: Messaging of Push campaign 

  1. Create Message variations
  2. Decide User Distribution among variations

Create Message Variations

Variation 1: With Emoji

Once done with Variation 1, you can Add another Variation to create A/B Test by clicking: 

You can add a new as well as copy from existing variations.

For the example, as we are only experimenting with title, we can use Copy Variation 1 which will copy all the payload fields from Variation 1. Change the title for Variation 2 accordingly.

Variation 2: Without Emoji

For the example above, we have created only two variations to test only one variable - emoji. One can create up-to 5 variations to test one or more hypothesis.

Add Control Group

Read why you need a control group in your experiment. Start your experiments with 2% allocated to control. To add a control group, you can click on Add button.

You can then set the percentage of users you want to add to control group (start your experiments with 2-5% allocated to control) and click ADD.

Once set, you will see a confirmation like the one below. Once added, you can choose to remove or edit the control group split.

Decide User Distribution among variations

Once done setting the message for variation,Click Next to set split percentage to distribute users among the variations. For the example below, we have divided the users 50% across each variation.

Important Note: The allocation of user to any of the group - test or control is non-sticky in nature i.e. for any active campaign, if users qualifies twice for receiving the campaign, we do not guarantee that if User A has received Variation 1 once, he would or wouldn't receive it again.

Complete Step 3 (Scheduling and Goal setting) and finish creating the campaign and measure results by comparing variation performances on Campaign Analytics.

You should be able to see analytics as below where you can compare variation performance, platform wise. The column with crown represents the Winner variation - the variation with maximum Click Through Rate.

You can also view a graphical comparison:

 

 Editing Multivariate experiment

Campaign Type

Status

Allowed edit options

General Push

Scheduled but not sent yet

 - Add/remove variations;
 - Edit variation message ;
 - Edit the distribution %age of variations

General Push - Periodic

Scheduled and not even 1 instance is sent

 - Add/remove variations;
 - Edit variation message ;
 - Edit the distribution %age of variations

General Push - Periodic

Scheduled but at-least 1 instance is executed

 - Edit variation message but cannot add/remove a variation;
 - Edit the distribution %age of variations

Smart Triggers

Active/Paused

 - Edit variation message but cannot add/remove a variation;
 - Edit the distribution %age of variations

Next in series: Sample experiments that you can create

Have more questions? Submit a request

Comments