FAQs: Uplift, Multivariate Testing, Dynamic Multivariate Testing

Here we have curated a few FAQs regarding Static and Dynamic Multivariate Testing

Q. What is multi-variate testing?
Ans. Multivariate testing refers to presence of more than two variations of a communication that are sent to different users chosen at random to compare the performance of message variations.

Q. How are the users distributed across variations?
Ans. We assign users to each variation by a random selection which is the best method for representative sampling. Hence, for a moderate target audience size, you can assume that users under every variation are statistically similar to the members of the entire group and will have similar demographic/behavioral traits.

Q. What is a Control Group?
Ans. A control group, is a subset of the customers we are targeting with a particular campaign, who will not receive the campaign and will hence serve as a baseline to compare campaign performance.

Q. How do we measure campaign Uplift?
Ans. Campaign Uplift measures the impact of your message variations to drive conversion goal. Control Group performance serves as base-line to calculate the Uplift. You can check this in detail here.

Q. How can I identify the users who received a certain variation of my general push campaign?
Ans. You can find these users by making a segmentation query from Segmentation Page . The query will be:

has executed Notification Received Android where campaign id contains <cid>_F_T_GP_AB_<VariationSerialNumber>

e.g for cid:57f112bc2b8dbc480b457ffe

Variation 1 can be identified with key: 57f112bc2b8dbc480b457ffe_F_T_GP_AB_1
Variation 2 can be identified with key: 57f112bc2b8dbc480b457ffe_F_T_GP_AB_2
Variation 3 can be identified with key: 57f112bc2b8dbc480b457ffe_F_T_GP_AB_3
 
The resultant query for Variation 1 will be: has executed Notification Received Android where campaign id contains 57f112bc2b8dbc480b457ffe_F_T_GP_AB_1
 
Note: Currently we do not allow to identify members of Control Group.
 
Q. How can I identify the users who clicked a certain variation of my general push campaign?
Ans. You can find these users by making a segmentation query from Segmentation Page . The query will be:

has executed Notification Clicked Android where campaign id contains <cid>_F_T_GP_AB_<VariationSerialNumber>

OR 

has executed Notification Clicked iOS where campaign id contains <cid>_F_T_GP_AB_<VariationSerialNumber>

e.g for cid:57f112bc2b8dbc480b457ffe

Variation 1 can be identified with key: 57f112bc2b8dbc480b457ffe_F_T_GP_AB_1
Variation 2 can be identified with key: 57f112bc2b8dbc480b457ffe_F_T_GP_AB_2
Variation 3 can be identified with key: 57f112bc2b8dbc480b457ffe_F_T_GP_AB_3
 
The resultant query for Variation 1 will be: has executed Notification Clicked Android where campaign id contains 57f112bc2b8dbc480b457ffe_F_T_GP_AB_1
 
Q. What is our statistical approach - Frequentist or Bayesian?
Ans. We use Bayesian Testing because the results of Bayesian Testing are easily understandable for marketers plus absorb the prior learning which refine the result over time. You can read a primer on Bayesian Testing here.
 
 
 
Have more questions? Submit a request

Comments