Here we have curated a few FAQs regarding Static and Dynamic Multivariate Testing
Q. What is multi-variate testing?
Ans. Multivariate testing refers to presence of more than two variations of a communication that are sent to different users chosen at random to compare the performance of message variations.
Q. How are the users distributed across variations?
Ans. We assign users to each variation by a random selection which is the best method for representative sampling. Hence, for a moderate target audience size, you can assume that users under every variation are statistically similar to the members of the entire group and will have similar demographic/behavioral traits.
Q. Is the message variation sticky in nature e.g. if a user received Variation 1 of any periodic campaign on Day 1 and is re-eligible to receive the same campaign on any day later, will he/she receive only Variation 1 ?
Our A/B Testing feature is not sticky in nature. Every time, a periodic instance is scheduled to run, we fetch all the eligible users and divide them randomly across message variations. Hence the user may or may not receive the Variation 1.
Similarly, if a user becomes re-eligible to receive an already received active campaign, same scenario will be repeated.
Q. What is a Control Group?
Ans. A control group is a subset of the customers we are targeting with a particular campaign, who will not receive the campaign and will hence serve as a baseline to compare campaign performance.
Q. How do we measure campaign Uplift?
Ans. Campaign Uplift measures the impact of your message variations to drive conversion goal. Control Group performance serves as base-line to calculate the Uplift. You can check this in detail here.
Q. How can I identify the users who received a certain variation of my general push campaign?
Ans. You can find these users by making a segmentation query by navigating to MoEngage Dashboard > Segment > Create Segment. The query will be:
has executed Notification Received Android where campaign id contains <cid>_F_T_GP_AB_<
e.g for cid:57f112bc2b8dbc480b457ffe
has executed Notification Clicked Android where campaign id contains <cid>_F_T_GP_AB_<
OR
has executed Notification Clicked iOS where campaign id contains <cid>_F_T_GP_AB_<
e.g for cid:57f112bc2b8dbc480b457ffe
Ans. We use Bayesian Testing because the results of Bayesian Testing are easily understandable for marketers plus absorb the prior learning which refines the result over time. You can read a primer on Bayesian Testing here.