AWS Data Imports

Overview

MoEngage allows customers to import users and events through flat files. To import data into MoEngage, you need to define the following:

  1. File location or Configure one or more file import destinations
  2. Import method

MoEngage supports imports from AWS S3 buckets only.

Supported File Types

The import usually takes place in two steps:

  1. One-Time Historical File Import
  2. Daily File Import

All the historical data is imported as a one-time upload post which users can import the data on a daily basis.

The following are the file types:

  1. Historical File : This is a one time import that allows users to dump all the historical data from the beginning of time to the present day. 
  2. Daily File : This is a daily import carried out on a regular basis.

For example, when importing data into MoEngage, a user can upload historical data for say five or six years as per their need or based on the amount of historical data they have using the Historical File. Once this one time upload is completed, they can they go about importing their Daily File every day.

File Naming Conventions

The file name convention for users and events is as follows:

Events

  1. Historical file format: {event_name}_{yyyymmdd}
  2. Daily file format: {event_name}_{yyyymmdd}

Here are a few sample event data file names:

  • Purchase Summary_20220206.csv
  • Purchase Transaction_20220207.csv

Users

  1.  Historical file format: {registered_userdata}_{yyyymmdd}, {anonymous_userdata}_{yyyymmdd}
  2. Daily file format: {registered_userdata}_{yyyymmdd}, {anonymous_userdata}_{yyyymmdd}

Here are a few sample User data file names:

  • registered_userdata_20220427.csv
  • anonymous_userdata_20220427.csv

Steps for Importing Data from S3 Buckets

 

PREREQUISITES

  1.  Details of the number of unique events to be imported along with the volumes for the same.

  2.  Information regarding the volume of historical data to be retained.
  3. Details of the mappings to MoEngage's events and user data

Step 1: Manifest Files

MoEngage requires the manifest files that contain the mappings between the source column, the moe attribute, and the data type of the column.

The number of manifest files needed would depend on the number of unique events to be imported. For example, if there are two unique events, then there would be two manifest files.

Name of the Manifest File

The name of the manifest file should be named <Event_name>_manifest.json

 

NOTE

  1. If we have the following multiple user files in the folder: registered_userdata_daily1_YYYYMMDD and registered_userdata_daily2_YYYYMMDD, we do not require multiple manifest files to enable mapping. We can have one "Master" manifest file which has all the columns (union of all the different user files), to enable the mapping.
  2. The number of columns in the manifest file should be greater than or equal to the number of columns in the individual user files.
  3. The user id (UID) should be a common column in all the files as it is used as the primary identifier. 

Sample Manifest Files ( Events and Users) 

An example of an event manifest file (Purchase Details_manifest.json)  is shown below: 

JSON

{
    "mapping": [
        {
            "column": "mobileno",
            "moe_attr": "u_em",
            "type": "STRING",
            "is_customer_id_field": true
        },
        {
            "column": "Product ID",
            "moe_attr": "Product ID",
            "type": "STRING"
        },
        {
            "column": "Product Name",
            "moe_attr": "Product Name",
            "type": "STRING"
        },
        {
            "column": "Purchase date",
            "moe_attr": "Purchase date",
            "type": "DATETIME",
            "format": "%-m/%-d/%-y %-I:%M %p"
        },
        {
            "column": "created_on",
            "moe_attr": "Created on",
            "type": "DATETIME",
            "format": "%-m-%-d-%Y"
        }
    ]
}

An example of a user manifest file (Purchase Details_manifest.json)  is shown below:

JSON

{
    "mapping": [
        {
            "column": "mobileno",
            "moe_attr": "u_em",
            "type": "STRING",
            "is_customer_id_field": true
        },
        {
            "column": "City Name",
            "moe_attr": "City",
            "type": "STRING"
        },
        {
            "column": "State Name",
            "moe_attr": "State",
            "type": "STRING"
        },
        {
            "column": "Customer Type",
            "moe_attr": "Customer Type",
            "type": "STRING"
        },
        {
            "column": "Registration date & Time",
            "moe_attr": "Registration Date",
            "type": "DATETIME",
            "format": "%-m/%-d/%-y %-I:%M %p"
        },
        {
            "column": "created_on",
            "moe_attr": "Created on",
            "type": "DATETIME",
            "format": "%-m-%-d-%Y"
        }
    ]
}

Manifest File Definition

Field Description
column Represents the column name from the source file.
moe_attr The MoEngage attribute to which the source file column has to be mapped.
type The data type of the column. For example, the data type can be String, Date and Time, and so on.
is_customer_id _field This needs to be added for the column which would be marked as user identifier.
format Applicable for Date and Time fields and denotes the date time format.

Step 2: Share the CSV template

Share the CSV template following the format below (please include the manifest file). The fields and data types should be consistent in every file.  

Example CSV File

If there are two events Purchase and Purchase detail, the sample files would be as follows.

Purchase File Format with headers

SOURCE Website
ID xxx@gmail.com
EMAIL xxx@gmail.com
PURCHASE_DATE_TIME 2021-03-01T00:00:00Z
RECEIPT_NUMBER ORD-123455
CUSTOMER_NUMBER CRSG-123456
MEMBER_GROUP XXXX
TOTAL_QUANTITY 4
TOTAL_AMOUNT 104
TOTAL_LOYALTY_AMOUNT
 

Purchase Detail Format with headers

SOURCE

Website

Website

ID

xxx@gmail.com

xxx@gmail.com
EMAIL

xxx@gmail.com

xxx@gmail.com
PURCHASE_DATE_TIME

2021-03-01T00:00:00Z

2021-03-01T00:00:00Z
CUSTOMER_NUMBER

CRSG-123456

CRSG-123456
RECEIPT_NUMBER

ORD-123455

ORD-123455

PRODUCT_CODE 86969 86969
PRODUCT_DESCRIPTION Creative SXFI AIR GAMER

Sound Blaster GC7

QUANTITY 4 1
UNIT_PURCHASE_PRICE 52 52
UNIT_RETAIL_PRICE 52 52
UNIT_DISCOUNT_AMOUNT 0 0
CATEGORY_CODE 600 100
CATEGORY_DESCRIPTION Headset Sound Blaster
SUB_CATEGORY_DESCRIPTION

Headset Gaming

Sound Blaster Gaming
BRAND_CODE 424 424
BRAND_DESCRIPTION PRIMERS PRIMERS
 

NOTE

  • The maximum file size supported is 100MB per file. In the case of a bigger file, the file needs to be split into smaller files that can be up to 100MB file size each.

Step 3: AWS S3 Credentials

Share the following AWS S3 details so that MoEngage can pull the required data.

  1. Bucket Name
  2. Access Key
  3. Region
  4. Secret Key

Step 4: Place the files in the S3 Folder

Place all files into Amazon S3 Folder. The folder structure should be as shown in the picture below.

Sample Path:

S3_Folder_Structure.png

The files inside the historical folder should be as shown below.

Sample Path:

HistoricalFilesFolderStructure.png

Step 5: Validate the Data and Manifest

Before the import begins for all the files, it is necessary to validate the data in the dashboard for 1-2 files. Your MoEngage CSM will validate one of the files uploaded and the manifest file before the run.  In case of any issues, your CSM will reach out to you. 

For more information, please reach out to support@moengage.com

Limits on Imports

The following are the maximum limits supported for AWS S3 Imports.

Limits for the CSV file

  1. Rows: We allow a maximum of 1 Million rows in a CSV file.
  2. Columns/Attributes: We allow 100 columns ( attributes) for users and 100 columns ( attributes) per event.
  3. File size: The preferred file size should be less than 100MB per CSV.

Import Limits

Import Type Users Events

One-Time Historical import

20 Million

72 Million

Daily Import

2 Million/day

10 Million/day

 

NOTE

  • These are the default limits. If there is a requirement that exceeds the above limits, you can get in touch with MoEngage to increase the same. We recommend that clients run higher volume imports during non-business hours i.e 10 PM - 4 AM IST.
Was this article helpful?
0 out of 0 found this helpful