Sample A/B experiments for Push

You should use multivariate testing to test how small changes to your campaigns can impact your results in a big way. You can then plough back the learnings from one experiment into the next campaign.

Multivariate experiments can be done across multiple ‘variables’ and one can compare how one variation performs better for a given conversion goal. As customer will be served either of the variations, their engagement with each experience is measured and collected to be shown as analytics. If possible, you should use Control Group to create a baseline. You can go as low as 2-5% for your control group allocation.

You can start experimenting with your messages:

1. Emojis

Sample Hypothesis: Having an Emoji in message give better CTRs

Experiment: Use Emoji in Variation 1 and send plain message in Variation 2

Measure: Compare CTRs of Variation 1 & Variation 2 . If CTRs of Variation 1 are better than there are high chances you have a valid hypothesis.

2. Personalization

Sample Hypothesis: Using customer’s First Name or Full Name results in better CTRs

Experiment: Use First Name attribute based personalization in Variation 1 and create plain message in Variation 2

Measure: Compare CTRs of Variation 1 & Variation 2 . If CTRs of Variation 1 are better than there are high chances you have a valid hypothesis.

3. Action Buttons

Sample Hypothesis: Using action buttons allow result in higher click-throughs

Experiment: Use notification action in Variation 1 while use only the default notification click in Variation 2. Set the conversion goal as per what conversion event you want to track.

Measure: Compare CTR for Variation 1 & Variation 2 . If CTR of Variation 1 is better than Variation 2, there are high chances you have a valid hypothesis.

4. Images

 Sample Hypothesis: Using images in Android push notifications result in higher click-throughs

Experiment: Use image in Variation 1 while do not use any image  in Variation 2

Measure: Compare CTRs of Variation 1 & Variation 2 . If CTRs of Variation 1 are better than that of Variation 2 then there are high chances you have a valid hypothesis.

5. Coupon Code

Sample Hypothesis: Sending Coupon Codes with push notifications result in higher conversions

Experiment: Create experiment where Variation 1 is using the Coupon Code while Variation 2 is not using it.Set the conversion goal as per what conversion event you want to track.

Measure: Compare conversions of Variation 1 & Variation 2 . If conversions of Variation 1 are better than 1, there are high chances you have a valid hypothesis.

6. Landing pages

Sample Hypothesis: Taking new customers to any deal page results in higher conversion rate

Experiment: Create experiment where Variation 1 is using the discount code of 10-15% OFF and taking user to Home page while Variation 2 does not serve any discount code but takes users to your deals page. 

Measure: Compare conversions of Variation 1 & Variation 2 . If conversions of Variation 2 are better than 1, there are high chances you have a valid hypothesis.

You can similarly experiment:

- if using a custom sound gets you more user attention resulting in higher CTRs

- if using a carousel of images gets more clicks in comparison to single image

Do let us know what else are you experimenting with.

Have more questions? Submit a request

Comments