Overview
Control Group in MoEngage Flows is the percentage of total user entries that you configure that would not receive any of the configured action campaigns of the flow.
Initially, restricting some of the users entering the flow from receiving any of the messages may seem counter-intuitive at first but by not delivering any message to Control Group (CG) users we establish the baseline of the organic behavior of the Targetable audience by tracking the behavior of the CG users after they enter the flow. The flow's performance when compared to the baseline established by the control group helps the marketers validate the below assumptions with which they publish a flow:
-
The configured flow can only lead to higher conversions.
-
There is a need for engaging the users with the current flow.
If the conversion ratio(CVR) of the control group is higher than that of the flow that would mean that users are organically more likely to convert than after receiving the campaigns from the configured hence the assumptions enlisted as not valid that flow and hence the flow stop not be continued.
However, in case the CVR of control is less than that of the flow, then the efficacy of the flow can be understood by calculating the conversion uplift that flow has fetched concerning the control group.
Conversion Uplift
The uplift analysis is the most common way to analyze the extent to which a MoEnagege Flow has nudged your users toward a conversion. Uplift is measured against the organic conversions from the control group users and conversions of the users who did receive at least one of the action campaigns of the flow.
Uplift = [(Flow CVR - Control group CVR )/Control Group CVR] x 100
| info |
Why?
|
Understanding Uplift Percentage
Example
Let's take a Flow that has a 36-hour attribution window and has been configured for a Push and an Email campaign.
Default scenario of 36 hr attribution window:
- Hour 0 - User Enters flow (90 TG, 10 CG)
- All of them are sent the push campaign Push-1, assuming all of them have received (100% impression/view-through)
- 0 to 24 hours - 5 users (4 TG, 1 CG) convert. These users are attributed only to the push campaign Push-1.
- Hour 24 - Email campaign Email-1 is sent out, assuming all of them have opened the email (100% impression/view-through)
- 24 to 36 hours - 8 users (8 TG, 0 CG) convert. These users are attributed to both the push campaign Push-1 and the Email campaign Email-1.
- Hours 36 to 60 - 5 users (4 TG, 1 CG) convert. These users are attributed only to the email campaign Email-1.
Uplift Percentages and CVR at the Child Level
| Channel |
Metrics |
| Push |
TG CVR = 12/90 = 13.3% CG CVR = 1/10 = 10% Uplift = 33% |
|
TG CVR = 12/90 = 13.3% CG CVR = 1/10 = 10% Uplift = 33% |
The Uplift Percentage at the Flow level
- Total TG converted users = 16 / 90 = 17.7%
- Total CG converted users = 2 / 10 = 20%
- Uplift = -11.5%
How to configure Control Group?
The option to add a Control Group is available in the Entry Condition node as well as on the flow setting page. The control group is enabled by default and can be changed anytime during the flow lifecycle. The users will become part of the control group only after the flow is published with the control group setting enabled.
Control group users are randomly selected out of the total users entering the trip out of all the users in the Flow's target audience.
In order to maintain the sanctity of the experiment, we ensure that once a user enters the flow as part of the control group, they would continue to be part of the control in all their subsequent entries into the flow until the control group is removed. You can choose to disable this if required.
How Control Group users are allocated?
Due to the scale at which MoEngage operates, we split users randomly into subgroups and apply the control group split percentage to each of the groups on a daily basis. The above process is made with the assumption that an equal number of users can be allotted to the different subgroups on any given day. So there is a need to have a minimum number of users to enter the flow for the split to be maintained. Also, with time the system scales down/up the number of users allotted as CG depending on the condition so that the % of users in CG is always close to the defined value.
When marketers check the “Users part of Control Group should continue to be part of it for their subsequent entries”(as shown in the above image), the system makes sure that a user marked as CG will always be a CG user. So, if CG users repeatedly enter the Flow more than the other type of users, then the % between CG trips and overall trips is going to be greater than the defined value. However, the % split on the unique user level will be as defined.
For example, let's take the case where a marketer defines a Flow with a 20% Control Group split and wants “Users part of the Control Group should continue to be part of it for their subsequent entries”. Let users U1, U2, U3, U4, and U5 enter 1,2,3,4,5 times, respectively. Let U5 be marked as the Control Group. In this situation:
Total Trips = 15
CG trips = 5
However, the CG split has been maintained at 20% as only 1 in 5 users was marked as the Control Group.
How big should the control group be?
You can make the control group as big as 95% or as small as 1%, depending on your target audience. For example, if you expect 100k users to enter the flow then 1% CG would be enough. The smaller the target group the larger the control group should be configured to effectively measure the efficacy of the flow.
How do Control Groups work?
Users are randomly allocated to the control group at the time of entering into the flow. The control group users after entering the flow traverse through the flow as per their behavior during the trip and eventually exit the flow. These users are not sent any campaign during their flow trip.
Conversion tracking for the Control group
Similar to the flow conversion tracking, control group users' conversion tracking starts as soon as they move past any one of the action campaigns configured in the flow. We assume that these users have received the campaign and we start tracking their conversion for the configured attribution window. As with the non-control group users also, conversions performed by CG users are attributed to the action campaigns as well as the flow.
| info |
Takeaway Since the control group users will skip all the action campaigns, we get an interesting problem "What should the conversion attribution window be for the flow in such a case?" We can try one of the following solutions:
We identified the third approach to be the least assumptive and most intuitive. If you have any concerns or suggestions please reach out to support@moenage.com. |
Editing the control group
While Marketers can choose to change the Control group variation at any time during the lifetime of the flow, we recommend against it as changing the traffic distribution may invalidate the experiment results.
For Example, let's say you publish your flow with a 50% control group on Day 1 and 2000 user entries happen on that day. 1000 Flow user entries result in 250 converted trips and 1000 control group(CG) user entries fetch 200 conversions. Hence, the CG CVR is 20% whereas flow CVR is 25% which is a 25% uplift from the control group.
Now the next day if you change the CG variation to 10% and 1000 user entries happen on the next day. Now out of 900 regular flow entries, 45 lead to at least one conversion( i.e. become converted trips), and out of 100 CG user entries only 4 converts. Now for Day 2, CG CVR is 4% and Flow CVR is 5% which is again a 25% uplift from the CG.
Now if combine the Day 1 and Day 2 results you will get CG CVR as 18.55% [(204/1100)x100) whereas the flow CVR during the same period drops down to 15.52% [(295/1900)x100] which gives an overall negative uplift of -16.33%.
Even though Flow CVR is better on both days individually, the Control group wins the experiment. Changing the CG distribution has eventually altered the results of the experiment, this in statistical theory is known as Simpson's Paradox. Because of this, we strongly recommend fixing the control group variation before publishing the flow and not changing it unless you absolutely have to or you have validated the efficacy of that flow over time.
Removing the control group
If you are observing conversion uplift for your flows over a significant period of time, we recommend you disable the Control group setting for your flow. Once this is done, all the users entering the flow can potentially receive the action campaigns configured in the flow.
If the Control group users re-enter the flow, they will also be eligible to receive the action campaigns. The list of previous control group users is also purged, so if the control group is added again, the CG user distribution will start from scratch without any influence from the previous control group.
| warning |
What happens if I remove the Control group by accident? In such a scenario, there is no way to restore the earlier configuration, but you can choose to add the CG again and restart, analyzing the flow performance. |
Finding the users marked as the control group
Once you have removed the control group, you might want to find them to target the users that were part of the control group and did not receive any of the flow action campaign events after going through the flow.
Users Marked as the Control Group for the Flow
You can create a segment with all the users who have performed the "User entered flow" event with the Flow Id Attribute value being the flow-id of your flow, with the "with control group" attribute as "true" and "Control Group Type" as Flow.
Users Marked as the Global Control Group
You can create a segment with all the users who have performed the "User entered flow" event with the Flow Id Attribute value being the flow-id of your flow, with the "with control group" attribute as "true" and "Control Group Type" as Global.