S3 Exports

Overview

All user events and campaign events that have been received or generated by MoEngage can be exported using S3 Exports. The exported data can be analyzed further to glean insights into user behavior, preferences, and what have you.

S3 Integration for Data Exports

S3 is a highly scalable storage system offered by Amazon Web Services. Integration with the S3 buckets will allow MoEngage to share periodic exports of required data in JSON format. To get started with S3 integration, follow the below steps:

Step 1: Create S3 Bucket

Even if you have an S3 bucket already, we still recommend creating a new bucket specifically for MoEngage so you can limit permissions.

To create a bucket for your app, open the S3 console and follow the instructions to Sign in or Create an Account with AWS.
Once signed in, select “S3” from the “Storage & Content Delivery” category.
Select “Create Bucket” on the next screen and you will be prompted to create your bucket and select a region.

Step 2: Create Policy

Navigate to the Policies tab in the navigation bar and select “Get Started” then “Create Policy”. This will allow you to add permissions for your user. Select “Create Your Own Policy”. This will give limited permissions so we only can access the bucket that you specify.

Input the code below when creating your own policy. Specify a “Policy Name” of your choice, and input the code below into the “Policy Document” section. Be sure to replace the INSERTSID with your Sid, INSERTPOLICYID with your policy id, and INSERTBUCKETNAME with your bucket name.

{
    "Version": "2012-10-17",
    "Id": <INSERTPOLCIYID>,
    "Statement": [
        {
            "Sid": <INSERTSID >,
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::612427630422:role/service-role/core_s3_export-role-e1qw8ulf"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::<INSERTBUCKETNAME>/*"
            }
        ]
}

Step 3: Enable S3 Exports on the MoEngage App Marketplace

On your MoEngage Dashboard, go to the App Marketplace.
Search for "Amazon S3".
Go to the "Integrate" tab, and click on "+ Add Integration".

Fill out the connection details of your S3 bucket:

Field	Description	Example
Connection Name	Give this connection an identifiable name.	`My S3 Export`
Bucket name	This is the name of your S3 bucket. How to find this: In your AWS Console, go to S3. Click on your bucket from the list, and the name of the bucket is mentioned at the top. You can also find this in your bucket Properties tab. Look at your S3 ARN, the part after `arn:aws:s3:::` is your bucket name:	`test-bucket`
Folder path	The path of the folder in which your files will be dumped. Do not begin or end with a slash. The final path of the dump will be: `s3://<your-bucket-name>/[<folder_path>_]<connection_name>/export_day=<DD-MM-YYYY>/event_exports_<export_timestamp>.json.gz` The export timestamp is the EPOCH timestamp of when the file was generated in the UTC timezone. Your final path will be shown on the UI for your visual confirmation. How to find this: From the list of objects in your S3 bucket, open the folder you to dump to. In the Properties tab, look at the folder's S3 URI. Your folder path is the part after `s3://[your-bucket-name]/` excluding the trailing slash. In the above example, your folder path is: `my-dump/to_import`	`my_folder` `my_folder/sub_folder`
Bucket region	Optionally, you can provide MoEngage with the region in which your bucket is hosted. How to find this: You can find the region of your bucket under the bucket's Properties. Click here to know more.	`Asia Pacific (Mumbai)` (ap-south-1)
Select events	Select which events you want to export to your S3 buckets. You can either choose to export "All events" (existing and new events coming into MoEngage in the future) or cherry-pick events manually and export only those events.
User properties	You can also select which user properties (User properties and device properties) you want to export along with your events.
Frequency	Select how often you want to export these events. We offer three frequencies: Every 15 minutes - We will dump files at 15-minute intervals. Daily - We will dump files at the end of each day, 00:00 UTC. Weekly - We will dump files on every Monday, 00:00 UTC. Please note that it is not recommended to frequently change your export frequency as this might result in data loss or data duplication.

Once you have filled in all the required details, you can test the connection to verify if everything is configured correctly.
Once you are ready, save the connection details and MoEngage will begin exporting events shortly to your S3 bucket.

Once the setup is complete, the files will start flowing into the specified folder path based on your selected export frequency.

Events and attributes

With the MoEngage and S3 integration, you can export all events coming into MoEngage to your S3 buckets. By default, MoEngage tracks some standard events and user properties through its SDKs. You can read about our standard events and attributes here. You can also export any custom user-defined event you tracked in MoEngage.

Data Export Format

File Format: JSON

Compression Type: GZip

Export Frequency: Every 15 mins / Daily / Weekly

Sample Folder Path:

s3://<your-bucket-name>/[<folder_path>_]<connection_name>/export_day=<DD-MM-YYYY>/event_exports_<export_timestamp>.json.gz

The <export_timestamp> is the EPOCH timestamp of when the file was generated in the UTC timezone.

Sample Export Data


    {
        "app_name": "App Name",
        "export_hour": "10",
        "db_name": "MoEngage App Name",
        "event": {
            "uid": "user id",
            "event_type": "USER_ACTION_EVENT",
            "event_code": "VIEW_RADAR",
            "event_name": "VIEW_RADAR",
            "user_attributes": {
                "moengage_user_id": "userid1",
                "user_attr1": "value1",
                "user_attr2": "value2"
            },
            "event_source": "MOENGAGE",
            "event_attributes": {
                "event_attr1": "value1",
                "event_attr2": "value2"
            },
            "event_uuid": "62045d15-8d5e-4cfd-ac49-5919bffb401d",
            "event_time": 1652820710,
            "device_attributes": {
                "moengage_device_id": "device id"
            }
        },
    		"export_day": "2022-05-20"
    }


    {
    "app_name": "App Name",
    "export_hour": "6",
    "db_name": "MoEngage App Name",
    "event": {
        "uid": "user id",
        "event_type": "CAMPAIGN_EVENT",
        "event_code": "MOE_EMAIL_SENT",
        "event_name": "Email Sent",
        "user_attributes": {
            "moengage_user_id": "userid1",
            "user_attr1": "value1",
            "user_attr2": "value2"
        },
        "event_source": "MOENGAGE",
        "email_id": "test@xyz.com",
        "event_attributes": {
            "campaign_name": "Campaign name",
            "moe_campaign_tags": [
                "Tag1",
                "Tag2"
            ],
            "campaign_type": "GENERAL",
            "moe_campaign_channel": "Email",
            "moe_delivery_type": "One Time",
            "campaign_id": "moengage campaign id",
            "campaign_channel": "EMAIL"
        },
        "event_uuid": "3b918064-08dc-47ae-b674-c23797475a8d",
        "event_time": 1653022310,
        "device_attributes": {
            "moengage_device_id": "device id if applicable"
        }
    },
    "export_day": "2022-05-20"
}

You can download a sample file from here. This is a GZIPPED file and needs to be extracted before the files can be viewed.

Frequently Asked Questions

arrow_drop_down How soon will I start getting the data in my S3 bucket?

It typically takes up to 30 minutes for the automated data exports to be set up for your account and for you to start seeing data there.

arrow_drop_down Can I export additional events at a later time?

Yes, if you need to export new events at a later time, you can always edit your configuration on the App Marketplace and get the new events added to your exports.

arrow_drop_down Can I change my old S3 Export's frequency as well?

If you are already using S3 Exports and wish to change your frequency, you can go to the App Marketplace and edit your S3 Exports configuration. Please note that changing the frequency to either Daily or Weekly will result in your file path being changed permanently. The new file path will be shown on the UI for your confirmation. You will not be able to go back to the previous file path once you make the switch.

arrow_drop_down What happens when I change my export frequency?

If you move from a more frequent export to a less frequent export, you will have temporary data duplication when you first make this change. If you move from a less frequent export to a more frequent export, there might be a rare possibility of data loss in case of high volume.

arrow_drop_down Can I export historical data using Automated Data Exports?

As of now, you cannot export data before configuring Automated data exports. Once configured, you will start seeing data for each data in your S3 bucket.

arrow_drop_down Is there any pricing involved for Automated Data Exports?

If your billing plan does not cover this already and you need to get this enabled at a later time, you would need to work with your Customer Success Manager to get this included in your billing plan. Based on the amount of data that MoEngage will place on your S3 bucket on a monthly basis, this could lead to additional charges.