Growth teams need to perform a lot of data analysis for their daily operations. For this analysis, they need to have the right data access at the right time. MoEngage Open Analytics enables growth teams to access the latest data on the MoEngage platform in real-time directly from the data storage. MoEngage enables customers to directly query on the data warehouse and get desired data.
This article strictly explains Open Analytics using AWS Athena. If you are looking for Open Analytics using Google BigQuery, click here.
Capabilities of Open Analytics
MoEngage open analytics with Athena can help with -
- Connecting to Visualization tools and generate all the reports on one dashboard
- Read all of the data using SQL queries through console or code
- Export all of the desired data into your system
This doc helps customers in accessing the data from the MoEngage warehouse using queries.
All data received at the MoEngage system can be categorized into 3 kinds –
- Device data,
- User Data, and
- Event data
The data is being collected using the SDK and S2S events and processed and stored with events.
At MoEngage AWS Warehouse, we process all the device attributes & event attributes and store them in one event table. Each Event has its own table consisting of Event Attributes and Device Attributes.
Tables Name - Event Name
|attribute 1||attribute 2||attribute 3||--||attribute n-1||attribute n|
Time-window for analysis can be selected using attribute - day. This partition time works for all tables. Must always use this in your queries to limit the costs. For Example
Select server_time, count(server_time) as metric
where day >= '2018-09-23'
group by 1
order by 1
Event Tables can be joined using common key/attributes such as - user_id
(Select user_id as user1 from "account_name"."uptete_spa") table1 INNER JOIN
(Select user_id as user2 from "account_name"."moe_user_merge" ) table2
ON table1.user1 = table2.user2
Event & Attribute Name Structure
To avoid data loss from similar names and few security purposes, the names of events and attributes present in MoEngage Warehouse will be cryptic.
Event names have structure as ‘EventName_XXX’, for example - moe_app_opened_828
Attribute names have structure as ‘Datatype_AttributeName_XXX’, for example - string_activity_name_880
We have provided an excel file with all the readable names and names present in the warehouse for ease of use.
MoEngage open analytics have a user table. While event tables have user attributes of that specific event execution time, this user table has all the updated & latest user attributes. This table refreshes once per day.
Table Name - base_user_xxx for example - base_user_224466
Column for MoEngage generated User Id - user_id
|attribute 1||attribute 2||attribute ...||attribute n|
Please request a Data Dictionary from your CSM for your account. This will have all the readable names for events, event attributes, and user attributes.
Important Attribute Information
(These attributes should be present in all of the events)
- day - Time-window for analysis can be selected using this attribute. This partition time works for all tables. This should be considered as the date attribute for Tables.
- server_time - UTC/GMT time at which events reach MoEngage servers.
- user_id - This is a MoEngage generated User Id.
- string_uid_125 - This is the User Id generated by your App/Site.
- original_event_name - Event Name
- user_time - Event time in local time zone
Open analytics Usage Best Practices
- Must always use day in your queries to keep the cost minimum. If the day is not used, Athena will scan all the data, and scanning costs will be higher.
- Athena works on a columnar database, hence use only the columns ( attributes ) that you require for analysis. If unnecessary columns are used, cost increases and query performance decreases.
- Please do read this information in details -
- A general guide to writing Athena SQL queries -
Connecting with Visualization Tools
All SQL queries work on MoEngage Warehouse. Data can be accessed using multiple platforms such as Tableau. Customers can also query directly on Athena as well.
Connecting Tableau with MoEngage Warehouse -
Service Account Access
To read data using python or java code, Growth teams will require service account access from MoEngage. Get in touch with your CSM to get it enabled for your account.
Also, check out the docs for the required process - https://docs.moengage.com/docs/open-analytics-setup-for-aws-athena
To transfer data out of the MoEngage Warehouse, you can use MoEngage streams, details given in the link below.
To get started with open analytics few of the queries are given above in the doc. Below are the links for a few of the frequently asked queries.
Open Analytics is an add-on feature to MoEngage analytics. Get in touch with the Support Team and CSM to get it enabled for your account.