Overview

MoEngage allows customers to import users and events through files stored inside S3 Buckets. MoEngage Dashboard enables you to configure multiple imports directly by defining file locations, configuring one or more than one file import destinations, and import methods.

Types of Imports

MoEngage can import the following from your S3 buckets:

1. Registered Users: These are users who are already registered on MoEngage.
2. Anonymous Users: These are users who are not yet registered on MoEngage.
3. Events (Standard and User Defined): MoEngage can import standard events like Campaign Interaction Events as well as your own user-defined events.

Preparing the files

Before configuring the imports on the MoEngage dashboard, ensure that all your files are in the expected format and within the sanctioned limits.

Supported File Types

We only support CSV files (in UTF-8 file encoding) as of now.

Naming Conventions

The file name is made up of two parts: <import type>_<date time format>.csv. The final name will depend on the type of import, and the date-time format you pick.

Import Type

• Registered Users
  • File names should start with the prefix registered_user_data_
  • Example: registered_user_data_01311997.csv
• Anonymous Users
  • File names should start with the prefix anonymous_user_data_
  • Example: anonymous_user_data_jan_2022.csv
• Events
  • File names should start with the prefix <event name>_
  • In certain cases, Event Name and Event Display Name can be different. You can manage your events by going to the Data Management page in MoEngage.
  • For example, if the event name is "Purchase Summary": Purchase Summary_31011997.csv
  • For example, if the event name is "Purchase_Transaction": Purchase_Transaction_19970131_1259.csv

Date Time Format

Your file names should end with the suffix <date time format>.csv. We support multiple date-time formats:

Type	Description	Example
{mm}_{yyyy}	mm: Month from 01 to 12 yyyy: 4-digit year	12_2022
{ddmmyyyy}	dd: Days from 01 to 31 mm: Month from 01 to 12 yyyy: 4-digit year	31012022
{yyyymmdd}	yyyy: 4-digit year mm: Month from 01 to 12 dd: Days from 01 to 31	20220112
{mon_yyyy}	mon: 3-letter abbreviation of the month from jan to dec (in lowercase) yyyy: 4-digit year	jan_2022
{yyyymmdd}_{hhmm}	yyyy: 4-digit year mm: Month from 01 to 12 dd: Days from 01 to 31 hh: 2-digit hours from 00 to 23 mm: 2-digit minutes from 00 to 59	20220131_1259

You can configure the date time formats for the file names on the MoEngage dashboard while setting up the import.

Valid File Name Examples

Few examples of valid file names:

• Registered Users Import with date time format {yyyymmdd}_{hhmm} will have file names like registered_user_data_20220131_1259.csv
• Anonymous Users Import with date time format {mon_yyyy} will have file names like anonymous_user_data_feb_2022.csv
• "Item Purchased" event import with date time format {ddmmyyyy} will have file names like Item Purchased_31012022.csv

File Structure

We support CSV files only. Here are some rules that your files need to adhere to:

1. The first row of your file needs to be the column names. MoEngage will always treat the first row of your file as the header row.
2. Please ensure there are no duplicate column names in your file. Do note that column names are case-sensitive.
3. Please ensure there are no missing column names.
4. Avoid empty rows (either the row is blank, or all the columns in that row are blank).

MoEngage will recognize the first row of your files as the header file and parse the column names. You can then map them to MoEngage Attributes on the MoEngage Dashboard or choose to upload a manifest file.

For each type of import, we require you to have a column that can be mapped to a mandatory attribute:

1. Registered Users Import: Your files should have a column that can be mapped to user ID (user identifier to identify User Profile in MoEngage).
2. Anonymous Users Import: Your files should have a column (email, mobile, etc) that can be mapped to an anonymous ID.
3. Event Import: Your files should have a column that can be mapped to Event Time and a column that can be mapped to user ID (user identifier to identify User Profile in MoEngage).

Here are some sample files:

1. Import Registered Users Sample File
2. Import Anonymous Users Sample File
3. Import Events Sample File
4. Sample Manifest/Mapping File

Manifest Files

Using the MoEngage Dashboard, you can map the columns in your CSV files to attributes inside MoEngage. Or optionally, you can also choose to upload a manifest file.

A manifest file contains mappings between the source column and a MoEngage Attribute. It also contains the data type of the column. This file needs to be in JSON format.

Sample Manifest File

An example of a manifest file is shown below:

{
  "mapping": [
    {
      "column": "ID",
      "moe_attr": "uid",
      "type": "string",
      "is_skipped": false
    },
    {
      "column": "First Name",
      "moe_attr": "u_fn",
      "type": "string",
      "is_skipped": false
    },
    {
      "column": "First Seen",
      "moe_attr": "cr_t",
      "type": "datetime",
      "datetime_format": "YYYY-MM-DD hh:mm:ss",
      "is_skipped": false
    },
    {
      "column": "LTV",
      "moe_attr": "t_rev",
      "type": "double",
      "is_skipped": false
    },
    {
      "column": "Install Status",
      "moe_attr": "installed",
      "type": "bool",
      "is_skipped": false
    }
  ]
}

For each column, you need to mention the following fields:

1. “column”: (required) Represents the column name from the source file. 
2. “moe_attr”: (required) The MoEngage Attribute to which you want to map the source file column. You can find all the attributes and their names on the Data Management page. Ensure each column is mapped to a unique "moe_attr".
3. “type”:  (optional) This field represents the data type of the column. Refer to the table below for the acceptable "type" formats for attributes. Refer to the Data Management page for more information.
4. “datetime_format”: (optional) This field indicates the date and time format. It is mandatory for DateTime fields only. Refer to the table below for the complete list of supported date-time formats.
5. “is_skipped”: (optional) This field represents whether MoEngage should import the column or not. It is a boolean field and any attribute marked as true will be skipped during import.

Supported User Attribute Types

Type	Description	Value in Manifest
String	Any string value. Examples of acceptable values: ABC, ABC XYZ, ABC123, etc.	"type": "string"
Double	Any decimal value. Examples of acceptable values: 3.14159, 241.23, -123.1	"type": "double"
Boolean	Examples of acceptable values: true, false	"type": "bool"
Date Time	Any date time value. The supported formats are covered in the next table. Example of acceptable values: 2019/02/22 17:54:14.933	"type": "datetime"

Supported Datetime Formats

Datetime Format	Examples
“datetime_format”: "YYYY-MM-DD"	2022-01-22
“datetime_format”: "YYYY/MM/DD"	2022/01/22
“datetime_format”: "DD/MM/YYYY"	22/01/2022
“datetime_format”: "DD-MM-YYYY"	22-01-2022
“datetime_format”: "DD-MM-YYYY hh:mm:ss"	31-12-2022 12:10:33
“datetime_format”: "DD/MM/YYYY hh:mm:ss"	31/12/2022 12:10:33
“datetime_format”: "YYYY-MM-DD hh:mm:ss"	2019-02-22 17:54:14
“datetime_format”: "YYYY/MM/DD hh:mm:ss"	2019/02/22 17:54:14
“datetime_format”: "DD-MM-YYYYThh:mm:ss.s"	31-12-2022T12:10:33.882
“datetime_format”: "DD/MM/YYYYThh:mm:ss.s"	31/12/2022T12:10:33.882
“datetime_format”: "DD-MM-YYYYThh:mm:ssTZD"	31-12-2022T12:10:33Z 31-12-2022T12:10:33+08:00 31-12-2022T12:10:33-08:00
“datetime_format”: "DD/MM/YYYYThh:mm:ssTZD"	31/12/2022T12:10:33Z 31/12/2022T12:10:33+08:00 31/12/2022T12:10:33-08:00
“datetime_format”: "YYYY-MM-DD hh:mm:ss.s"	2019-02-22 17:54:14.933
“datetime_format”: "YYYY/MM/DD hh:mm:ss.s"	2019/02/22 17:54:14.933
“datetime_format”: "YYYY-MM-DDThh:mm:ssTZD"	2019-11-14T00:01:02Z 2019-11-14T00:01:02+08:00 2019-11-14T00:01:02Z-08:00
“datetime_format”: "YYYY/MM/DDThh:mm:ssTZD"	2019/11/14T00:01:02Z 2019/11/14T00:01:02+08:00 2019/11/14T00:01:02-08:00
“datetime_format”: "YYYY-MM-DDThh:mm:ss.sTZD"	2019-02-22T17:54:14.957Z 2019-02-22T17:54:14.957299-08:00 2019-02-22T17:54:14.957299+08:00
“datetime_format”: "YYYY/MM/DDThh:mm:ss.sTZD"	2019/02/22T17:54:14.957Z 2019/02/22T17:54:14.957299-08:00 2019/02/22T17:54:14.957299+08:00

Once you are ready with the files, you need to place them inside a folder on your S3 bucket.

Folder Structure

Place all files into an Amazon S3 Folder. You can configure the folder path while setting up the import.

Note that MoEngage will not look for files inside any sub-folders.

Required Access Permissions

You will need to create a new policy so that MoEngage can access your buckets and import files correctly. We request the following permissions:

1. s3:Get* - This will grant us the permissions required to download your files.
2. s3:List* - This will grant us the permissions required to view and list your files.

Create a new policy

Navigate to the Policies tab in the navigation bar and select “Get Started” then “Create Policy”. This will allow you to add permissions for your user. Select “Create Your Own Policy”. This will give limited permissions so we only can access the bucket that you specify.

Input the code below when creating your own policy. Specify a “Policy Name” of your choice, and input the code below into the “Policy Document” section. Be sure to replace the <INSERTSID> with your Statement ID, <INSERTPOLICYID> with your Policy ID, and <INSERTBUCKETNAME> with your bucket name. Replace the Principal as well (make sure the access key ID and access secret key belongs to this user/account).

{
    "Version": "2012-10-17",
    "Id": <INSERTPOLCIYID>,
    "Statement": [
        {
            "Sid": <INSERTSID>,
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<ACCOUNTID>:<USER>/<USERNAME>"
            },
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::<INSERTBUCKETNAME>/*",
                "arn:aws:s3:::<INSERTBUCKETNAME>"
            ]
        }
    ]
}

You can find more information about how to create policies here.

Setup Imports from S3

You can set up the S3 Imports from the MoEngage Dashboard >> Segment >> S3 Imports page.

Clicking on "+Import" button will allow you to create a new import for either Users (Registered/Anonymous can be picked in the next step) or Events.

Step 1: Configure your S3 credentials and file format

User Imports

You will now have the option to select whether you want to import Registered users or Anonymous users:

Choose which one you want to import. Both registered users import and anonymous users import expect different file names. Refer to the Naming Conventions section for more information.

Event Imports

For Event imports, you will now have the option to select which event it is that you want to import:

Based on the event you pick, your file names will be different. Your file names must begin with the Event Name of your chosen event. In the "Select event" dropdown, we will show you the Display Names of all the available events in your MoEngage app. Some of these events might have different Display Names and Event Names. When you're selecting an event, MoEngage will show you what is the file name it expects for that particular event in the Event Description box:

In the above example, "App/Site Opened" is the Display Name of the chosen event and its Event Name is MY_APP_OPENED, so the file names should begin with MY_APP_OPENED_. Refer to the Naming Conventions section for more information.

You can also create a new event by clicking on the "+ Create event" option at the end of the dropdown:

Give your event a name that is unique. By default, your Display Name will be the same as the Event Name. You can go to the Data Management page to view or edit this event and also your other MoEngage events. Do note that any new events you create now will only be available in your app after the first successful import.

Adding your S3 Credentials

In this first step, "Source and format", you will also need to tell MoEngage how to connect to your S3 bucket and where to look for the files.

Refer to the below table to know how to fill these fields:

Field	Description	Example
S3 bucket name	This is the name of your S3 bucket. How to find this: In your AWS Console, go to Amazon S3. Click on your bucket from the list, and the name of the bucket is mentioned at the top. You can also find this in your bucket Properties tab. Look at your S3 ARN, the part after arn:aws:s3::: is your bucket name:	users_to_import
S3 access key	This is the Access Key ID of the account with which you want MoEngage to connect to your bucket. You can either choose to provide the Access Key ID of an existing user (programmatic access or console access) or create a new user with Programmatic Access just for MoEngage. How to find this: In your AWS Console, go to IAM Console >> Users. You can manage your keys for each user here. For an existing user, under the "Security credentials" tab, scroll down to the "Access keys" section. Either copy an existing access key or create a new one (recommended). Enter the Access Key ID here. Click here to know more.	23478207027842073230762374023
S3 secret key	AWS provides a pair of Access Key IDs and secret keys for each user. Enter the corresponding Secret Access Key for the Access Key ID of the account above. How to find this: Your Secret Access Key is provided when you make a new Access Key ID. Click here to know more.	wJa...lr/.../bPxRf...LEKEY
Region	Optionally, you can provide MoEngage with the region in which your bucket is hosted. How to find this: You can find the region of your bucket under the bucket's Properties. Click here to know more.	Asia Pacific (Mumbai) (ap-south-1)
Folder path	The path of the folder in which your files will be present. MoEngage does not look inside sub-folders. Do not include trailing slashes. Please note that we do not support root-folders at the moment so ensure that all your files are inside a folder. How to find this: From the list of objects in your S3 bucket, open the folder you to import from. In the Properties tab, look at the folder's S3 URI. Your folder path is the part after s3://[your-bucket-name]/ excluding the trailing slash. In the above example, your folder path is: my-dump/to_import	my_folder my_folder/sub_folder

Import Name

Enter a name for this import to easily identify it on the Imports Dashboard.

File configurations

Select the DateTime format that will be in your file name. As per the Naming Conventions defined above, your file name should end with _<date time format>.csv. Refer to the table provided earlier to see which DateTime formats we support. The info message below will update according to your chosen DateTime format so that you can easily cross-verify if your files names are as per expectations:

After you've configured your S3 Credentials and your file names format, MoEngage will try to connect to your S3 bucket and fetch the most recent file available with the specified format. Ensure you have at least one file available in the specified location otherwise, you will not be able to continue. Once you're ready, click the "Fetch file" button to fetch a recent file.

If your S3 credentials and chosen file format are correct, then MoEngage will show you the preview of the top 5 rows from a recent file:

However, sometimes there can be an error while fetching the file from your S3 buckets. These are some common errors:

• Given S3 Access Key is not valid - This means that your S3 credentials are incorrect. Please enter the correct credentials and try to fetch the file again.
• Please check the folder path and try again - This means that either your folder path is incorrect or there are no CSV files present inside the given folder path.
• Found zero data rows in the file - This means that the file MoEngage fetched did not have any rows after the header row (the total row count of the file is 1). We expect at least 2 rows (1 header row + 1 data row) in the file since the first row is always treated as the header row. Learn more about the File Structure.

Once you are satisfied with the preview of the file, click Next to continue to the mapping step.

Step 2: Map your columns to MoEngage Attributes

In this step, you need to map the columns of your CSV files to the attributes present in MoEngage. All your columns will be shown one below the other:

1. Column name - This is the column name (picked from the first row of the fetched file in the previous step) to be mapped. Below the column name, we also show a sample value (picked from the second row of the fetched file in the previous step) for your reference.
2. Map attribute - Here you need to pick which MoEngage attribute you want to map the CSV column to. You can also choose to create a new attribute. Some attributes support ingestion from multiple data types, so you need to pick the data type of the column as well. For the "DateTime" columns, you also need to pick the format.
3. More actions - You can optionally choose to skip the column. The skipped column will not be imported.

Depending on the type of import, there are a few mandatory mappings required:

1. User imports - You need to select the column having:
   1. Registered - 
      1. User ID - Your file should mandatorily have a column that contains a unique user identifier (that identifies a user’s account in your system).
   2. Anonymous - 
      1. Anonymous ID - You need to mark a column from your file (email, mobile number, etc.) as an identifier of these users.
2. Event imports - You need to select the column having:
   1. User ID - This column will be used to match the users in MoEngage to your CSV files.
   2. Event Time - You need to map either Event Time (UTC) or Event Time (App Timezone). If you have chosen Event Time (UTC), then the "Event Time" of the imported event will be converted to the timezone chosen in your MoEngage Dashboard settings:

Once a mandatory mapping is marked, it will reflect against the column name in the mapping table and you will no longer be able to mark the column as skippable. 

Just like new events, you can also create a new attribute from the dropdown by clicking on "+ Create attribute":

Give your attribute a name and select its Data type. You can edit this as well as existing attributes from the Data Management page. Do note that any new attributes you create now will only be available in your app after the first successful import.

Manifest Files

Optionally, you can choose to auto-map these columns by uploading a manifest file. 

To upload a manifest file, click the "Upload mapping file" option on the top-right of the mapping table:

Upload your manifest file and click Done. Your mappings will be auto-configured accordingly. Any columns with non-MoEngage attributes will be left blank and can either manually map the column or create a new attribute for it. Make sure your manifest file follows the expected conventions as mentioned.

Any additional columns that are present in your Manifest File but not there in your CSV file will be ignored. Also, if the mapping for an existing CSV column is not present in the manifest file, MoEngage will keep the mapping blank for you to manually configure it.

Do note that if a column in the manifest file is mapped to a non-existent MoEngage Attribute, then the mapping will be blank and you will need to first manually create a new attribute from the UI and then map it.

Sending Import Notifications

You can, optionally, choose to get notified about your imports' statuses via email. Select up to 10 emails to send the status emails to. You will receive an email in the following events:

1. An import was successful.
2. An import failed, along with the possible cause of failure.

When you are satisfied with all mappings, click Next. If there are any errors, correct them to move forward.

Step 3: Select the Import frequency

In this step, you need to define when to import files from your S3 bucket. We support the following types of imports:

1. One-Time Imports: You can either choose to run the import as soon as possible or at a later date and time (scheduled)
2. Periodic: You can choose to run your imports daily, weekly or monthly, or with intervals and advanced configurations.

For each run, MoEngage will try to fetch a file with the configured DateTime format so ensure your file names are configured properly.

Optionally, you can specify whether the import should end after a specified set of occurrences, or at a particular date. Click "Done" when ready.

Duplicate Imports

You can only configure one unique import at a time. An import is considered a duplicated import when all of these are the same:

1. Import Type - Users / Events
2. Import Sub-Type - Event Name / Registered / Anonymous - If the event name is the same, or there already exists a registered/anonymous users import with the same S3 path and bucket name.
3. S3 Folder Path
4. Bucket Name

As long as any one of the above is different, the import is considered unique.

Imports Dashboard

The S3 Imports Dashboard contains all the information you need to keep track of your imports. You can view the most important information at a glance:

Name	Description
Name	The name of the import you gave while setting it up.
Type	The source of the imports, right now we only support S3. The type of imports, One-Time or Periodic, is also mentioned in the brackets.
Created by	The email ID of the account which created the import.
Created at	The date and time of the import when it was created.
Status	The last status of the import. One-Time Imports: The current status of the import. Scheduled: The import is scheduled to run in the future. Processing: The import is currently ongoing. Successful: The import was successful. Partial Success: At least one file was imported with Partial Success. Failed: The import has failed. Recurring Imports: The current status of the most latest scheduled import. Scheduled: The import is scheduled to run in the future. The scheduled time is given below. Processing: The import is currently ongoing. An estimated time of completion is also mentioned. Successful: The import was successful. The completion time is also provided. Partial Success: When at least one file was imported with Partial Success. Failed: The import has failed. The failure time is given for reference.
Files Processed	The number of files processed to date.
Actions	You can choose to View Details, Edit, Duplicate or Delete an import.

You can also view the details of each import by clicking on the three-dot actions menu against it, and selecting "View Details":

Name	Description
Scheduled At	The name of the import you gave while setting it up.
Status	The status of the import. Successful Failed - Hover to see the failure reason. Partial Success - At least 1 row failed to import in the file.
Rows in file	The number of data rows present in the file.
Events Added	The number of events that were imported from the file.
Events Failed	The number of events that failed to import.
Users Updated	The number of users that were updated.
Users Created	The number of users that were created during the import.
Users Failed	The number of users that failed to import.
Rows Skipped	The number of rows that were skipped during import.
Actions	You can choose "Export File" to download a copy of the file that was imported.

File Limits

File Size Limit

The maximum file size we support is 200MB per file. In case of a bigger file, the file needs to be split into 200MB file sizes each.

Other Limits

1. Rows: We allow max 1M rows in a CSV file.
2. Columns/Attributes: We would allow 100 columns (attributes) for users and 100 columns (attributes) per event 
3. File size: The preferred file size should be <200MB  (per CSV)
4. Import limits: Please refer to the below table. 

Import Type	Users	Events
One-Time Historical import	20M	72M
Daily Import	2M/day	10M/day