Control Group In Flows

Overview

Control Group in MoEngage Flows is the percentage of total user entries that you configure that would not receive any of the configured action campaigns of the flow. 

Initially, restricting some of the users entering the flow from receiving any of the messages may seem counter-intuitive at first but by not delivering any message to control group users we establish the baseline of the organic behavior of the Targetable audience by tracking the behavior of the CG users after they enter the flow. The flow's performance when compared to the baseline established by the control group helps the marketers validate the below assumptions with which they publish a flow:

  1. The configured flow can only lead to higher conversions.

  2. There is a need for engaging the users with the current flow. 

If the conversion ratio(CVR) of the control group is higher than that of the flow that would mean that users are organically more likely to convert than after receiving the campaigns from the configured hence the assumptions enlisted as not valid that flow and hence the flow stop not be continued. 

However, in case the CVR of control is less than that of the flow, then the efficacy of the flow can be understood by calculating the conversion uplift that flow has fetched concerning the control group. 

Conversion Uplift

The uplift analysis is the most common way to analyze the extent to which a MoEnagege Flow has nudged your users towards a conversion. Uplift is measured against the organic conversions from the control group users and conversions of the users who did receive at least one of the action campaigns of the flow. 

Uplift = [(Flow CVR - Control group CVR )/Control Group CVR] x 100

Flow_Stats.png

 

Takeaway

    A 50% conversion uplift signifies that the configured flow has fetched 50% more conversions than what would be possible had the Flow never been configured.

Understanding Uplift Percentage

The uplift calculation is dependent on the stage and the timeframe in consideration. Thus, the upliftment for the flow shown at the top of the Detailed Stats section might vary from the upliftment shown in the Engagement Trends in the Detailed stats as the latter is calculated only for the chosen Channel while the former is calculated for the entire flow.

Example

Let's take a Flow that has a 36-hour attribution window and has been configured for a Push and an Email campaign.
Default scenario of 36 hr attribution window:

Timeline
  •  Hour 0 - User Enters flow (90 TG, 10 CG)
  • All of them are sent the push campaign Push-1, assuming all of them have received (100% impression/view-through)
  • 0 to 24 hours - 5 users (4 TG, 1 CG) convert. These users are attributed only to the push campaign Push-1.
  • Hour 24 - Email campaign Email-1 is sent out, assuming all of them have opened the email (100% impression/view-through)
  • 24 to 36 hours - 8 users (8 TG, 0 CG) convert. These users are attributed to both the push campaign Push-1 and the Email campaign Email-1.
  • Hours 36 to 60 - 5 users (4 TG, 1 CG) convert. These users are attributed only to the email campaign Email-1. 
The uplift percentage is calculated as shown below for this example.

Uplift Percentages and CVR at the Child Level

Channel

Metrics

Push

TG CVR = 12/90 = 13.3%

CG CVR = 1/10 = 10%

Uplift = 33%

Email

TG CVR = 12/90 = 13.3%

CG CVR = 1/10 = 10%

Uplift = 33%

The Uplift Percentage at the Flow level

  • Total TG converted users = 16 / 90 = 17.7%
  • Total CG converted users = 2 / 10 = 20%
  • Uplift = -11.5%

How to configure Control Group?

The option to add Control Group is available in the Entry Condition node as well as on the flow setting page. The control group is enabled by default and can be changed anytime during the flow lifecycle. The users will become part of the control group only after the flow is published with the control group setting enabled. 

WhoWillEnterTheFlow.png

Control group users are randomly selected out of the total users entering the trip out of all the users in the Flow's target audience. 

In order to maintain the sanctity of the experiment we ensure that once a user enters the flow as part of the control group, they would continue to be part of the control in all their subsequent entries into the flow until the control group is removed. You can choose to disable this if required.

How big should the control group be?

You can make the control group as big as 95% or as small as 1%, depending on your target audience. For example, if you expect 100k users to enter the flow then 1% CG would be enough. The smaller the target group the larger the control group should be configured to effectively measure the efficacy of the flow.

How do Control Groups work?

Users are randomly allocated to the control group at the time of entering into the flow. The control group users after entering the flow traverse through the flow as per their behavior during the trip and eventually exit the flow. These users are not sent any campaign during their flow trip.

Conversion tracking for the Control group

Similar to the flow conversion tracking, control group users' conversion tracking starts as soon as they move past any one of the action campaigns configured in the flow. We assume that these users have received the campaign and we start tracking their conversion for the configured attribution window. As with the non-control group users also, conversions performed by CG users are attributed to the action campaigns as well as the flow.

 

Why?

Since the control group users will skip all the action campaigns, we get an interesting problem "What should the conversion attribution window be for the flow in such a case?" We can try one of the following solutions:

  • Approach 1: Calculate the longest path of the flow and keep waiting for that much period for control group users to convert after entering the flow. But this assumes that all the control group users who had not been part of the control group would have gone traversed the flow through the longest path, which seems very unlikely.
  • Approach 2: Wait for the attribution window period for control group users to convert after entering the flow. This approach gives an unfair advantage to the control group users if the configured flow is long. For example, if the largest path of the flow takes 3 days to traverse and exit, and if the attribution window of the flow is 24 hours. Then, a control group can enter and convert for the flow approximately thrice before the flow user completes one trip through the longest path.
  • Approach 3: We let control group users traverse through any of the possible paths based on their behavior after they enter the flow and continue to track the conversions. After they exit the flow from one of the nodes we track the conversion for an additional period that is equal to the attribution window for the flow. This way we let the control group user behavior govern the period till which the conversion should be tracked.

We identified the third approach to be the least assumptive and most intuitive. If you have any concerns or suggestions please reach out to support@moenage.com.

Editing the control group

While Marketers can choose to change the Control group variation at any time during the lifetime of the flow, we recommend against it as changing the traffic distribution may invalidate the experiment results. 

For Example, let's say you publish your flow with a 50% control group on Day 1 and 2000 user entries happen on that day. 1000 Flow user entries result in 250 converted trips and 1000 control group(CG) user entries fetch 200 conversions. Hence, the CG CVR is 20% whereas flow CVR is 25% which is a 25% uplift from the control group.

Now the next day if you change the CG variation to 10% and 1000 user entries happen on the next day. Now out of 900 regular flow entries, 45 lead to at least one conversion( i.e. become converted trips), and out of 100 CG user entries only 4 converts. Now for Day 2, CG CVR is 4% and Flow CVR is 5% which is again a 25% uplift from the CG. 

Now if combine the Day 1 and Day 2 results you will get CG CVR as 18.55% [(204/1100)x100) whereas the flow CVR during the same period drops down to 15.52% [(295/1900)x100] which gives an overall negative uplift of -16.33%. 

Even though Flow CVR is better on both days individually, the Control group wins the experiment. Changing the CG distribution has eventually altered the results of the experiment, this in statistical theory is known as Simpson's Paradox. Because of this, we strongly recommend fixing the control group variation before publishing the flow and not changing it unless you absolutely have to or you have validated the efficacy of that flow over time.

Removing the control group

If you are observing conversion uplift for your flows over a significant period of time, we recommend you disable the Control group setting for your flow. Once this is done, all the users entering the flow can potentially receive the action campaigns configured in the flow. 

If the Control group users re-enter the flow, they will also be eligible to receive the action campaigns. The list of previous control group users is also purged, so if the control group is added again the CG user distribution will start from scratch without any influence from the previous control group.  

 

What happens if I remove the Control group by accident?

In such a scenario, there is no way to restore the earlier configuration but you can choose to add the CG again and restart analyzing the flow performance.

Once you have removed the control group, you might want to target the users that were part of the control group and did not receive any of the flow action campaign events after going through the flow. 

You can create a segment with all the users who have performed the "User entered flow" event with the Flow Id Attribute value being the flow-id of your flow and with the "with control group" attribute as "true".

Flows_Segmentation.png

 

Was this article helpful?
2 out of 2 found this helpful