Control Group in flows is currently a Beta Feature, if you wish to try out this feature please reach out to firstname.lastname@example.org.
Control Group in MoEngage Flows is the percentage of total user entries that you configure that would not receive any of the configured action campaigns of the flow.
Initially, restricting some of the users entering the flow from receiving any of the messages may seem counter-intuitive at first but by not delivering any message to control group users we establish the baseline of the organic behavior of the Targetable audience by tracking the behavior of the CG users after they enter the flow. The flow's performance when compared to the baseline established by the control group helps the marketers validate the below assumptions with which they publish a flow:
The configured flow can only lead to higher conversions.
There is a need for engaging the users with the current flow.
If the conversion ratio(CVR) of the control group is higher than that of the flow that would mean that users are organically more likely to convert than after receiving the campaigns from the configured hence the assumptions enlisted as not valid that flow and hence the flow stop not be continued.
However, in case the CVR of control is less than that of the flow, then the efficacy of the flow can be understood by calculating the conversion uplift that flow has fetched with respect to the control group.
Uplift analysis the most common way to analyze the extent with which a MoEnagege Flow has nudged your users towards a conversion. Uplift is measured against the organic conversions from the control group users and conversions of the users who did receive at least one of the action campaigns of the flow.
Uplift = [(Flow CVR - Control group CVR )/Control Group CVR] x 100
A 50% conversion uplift signifies that the configured flow has fetched 50% more conversions than what would be possible had the Flow was never configured.
How to configure Control Group?
Option to add control group is available in the Entry Condition node and well as in the flow setting page. The control group is disabled by default and can be added anytime during the flow lifecycle. The users will become part of the control group only after the flow is published with the control group setting enabled.
Control group users are randomly selected out of the total users entering the trip out of all the users in the Flow's target audience.
In order to maintain the sanctity of the experiment we ensure that once a user enters the flow as part of the control group, they would continue to be part of the control in all their subsequent entries into the flow until the control group is removed. You can choose to disable this if required.
How big should be the control group?
You can make the control group as big as 95% or as small as 1%, depending on your target audience. For example, if you expect 100k users entering the flow then 1% CG would be enough. The smaller the target group the larger the control group should be configured to effectively measure the efficacy of the flow.
How does it work?
Users are randomly allocated to control group at the time of entering into the flow. The control group users after entering the flow traverse through the flow as per their behavior during the trip and eventually exit the flow.
Conversion tracking for Control group
Unlike the flow conversion tracking, control group users conversion tracking starts as soon as they enter the flow and we continue to track the conversion for a period of configured attribution window for the flow after they exit the flow from one of the flow exit nodes.
Since the control group users will skip all the action campaigns, we get an interesting problem "what should be conversion attribution window be flow in such a case?" We can try one of the following solutions
- Calculating the longest path of the flow and keep waiting for that much period for control group users to convert after entering the flow. But this assumes that all the control group users had not been part of the control group would have gone traversed the flow through the longest path, which seems very unlikely.
- Wait for the attribution window period for control group users to convert after entering the flow. This approach gives an unfair advantage to the control group users if the configured flow is long. For example, if the largest path of the flow takes 3 days to traverse and exit, and if the attribution window of the flow is 24 hours. Then, a control group can enter and convert for the flow of approximately thrice before the flow user completes one trip through the longest path.
- We let to control group users to traverse through any of the possible paths based on their behavior after they enter the flow and continue to track the conversions. After they exit the flow from one of the nodes we track the conversion for an additional period that is equal to the attribution window fo the flow. This way we let the control group user behavior govern the period till which the conversion should be tracked.
We identified the third approach to be the lease assumptive and most intuitive. If you have any concerns or suggestions please reach out to email@example.com.
Editing the control group
While Marketers can choose to change the Control group variation at any time during the lifetime of the flow, but we recommend against it as changing the traffic distribution may invalidate the experiment results.
For Example, let's say you publish your flow with a 50% control group on Day 1 and 2000 user entries happen on that day. 1000 Flow user entries result in 250 converted trips and 1000 control group(CG) user entries fetch 200 conversions. Hence, the CG CVR is 20% whereas flow CVR is 25% which is a 25% uplift from the control group.
Now the next day if you change the CG variation to 10% and 1000 user entries happen on the next day. Now out of 900 regular flow entries, 45 lead to at least one conversion( i.e. become converted trips), and out of 100 CG user entries only 4 converts. Now for Day 2, CG CVR is 4% and Flow CVR is 5% which is again a 25% uplift from the CG.
Now if combine the Day 1 and Day 2 results you will get CG CVR as 18.55% [(204/1100)x100) whereas the flow CVR during the same period drops down to 15.52% [(295/1900)x100] which gives an overall negative uplift of -16.33%.
Even though Flow CVR is better on both the days individually, the Control group wins the experiment. Changing the CG distribution has eventually altered the results of the experiment, this in statistical theory is known as Simpson's Paradox. Because of this, we strongly recommend fixing the control group variation before publishing flow and not changing unless you absolutely have to or you have validated the efficacy of that flow over time.
Removing the control group
If you are observing conversion uplift for your flows over a significant period of time, we recommend you disable the Control group setting for your flow. Once this is done, all the users entering the flow can potentially receive the action campaigns configured in the flow.
Control group users if re-enter the flow they will also be eligible to receive the action campaigns. The list of previous control group users is also purged, so if the control group is added again the CG user distribution will start from scratch without any influence of the previous control group.
What happens if I remove the Control group by accident?
In such a scenario, there is no way to restore the earlier configuration but you can choose to add the CG again and restart analyzing the flow performance.
Once you have removed the control group, you might want to target the users that were part of the control group and did not receive any of the flow action campaigns events after going through the flow.
You can create a segment with all the users who have performed "User Added to Control Group" event with the Flow Id Attribute value being the flow-id of your flow.