Overview

MoEngage allows customers to import users and events through files stored inside S3 Buckets.

Types of Imports

MoEngage can import the following from your S3 buckets:

1. Registered Users: Users who are already registered on MoEngage.
2. Anonymous Users: Users who are not yet registered on MoEngage.
3. Events (standard and user-defined): MoEngage can import standard events like campaign interaction events and your user-defined events.
4. Auxiliary Data: Users who are added on a temporary basis.

Prepare the files

For information on supported file types, naming conventions, and file structure, consult the File Imports help documentation. Once your files are prepared, place them in a folder within your S3 bucket

Folder Structure

You must configure the folder path when setting up the import. Place all files into an Amazon S3 Folder. 

Note that MoEngage does not look for files inside any sub-folders

Required Access Permissions

You must create a new policy so MoEngage can access your buckets and import files correctly. MoEngage requests the following permissions:

1. s3:Get* - This grants the permissions required to download your files.
2. s3:List* - This grants the permissions required to view and list your files.

Create a new policy

To create a new policy, perform the following steps:

1. Navigate to the Policies tab in the navigation bar.
2. Click Get Started and then click Create Policy.
   This allows you to add permissions for your user.
   
3. Select Create Your Own Policy.
   This gives you limited permissions so MoEngage can only access the bucket that you specify.

Input the code below when creating your own policy. Specify a “Policy Name” of your choice, and input the code below into the “Policy Document” section. Be sure to replace the <INSERTSID> with your Statement ID, <INSERTPOLICYID> with your Policy ID, and <INSERTBUCKETNAME> with your bucket name. Replace the Principal as well (make sure the access key ID and access secret key belong to this user/account).

{
    "Version": "2012-10-17",
    "Id": <INSERTPOLCIYID>,
    "Statement": [
        {
            "Sid": <INSERTSID>,
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<ACCOUNTID>:<USER>/<USERNAME>"
            },
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::<INSERTBUCKETNAME>/*",
                "arn:aws:s3:::<INSERTBUCKETNAME>"
            ]
        }
    ]
}

You can find more information about how to create policies here.

Set Up Imports from S3

To set up S3 Imports, perform the following steps:

1. On the left navigation menu in the MoEngage dashboard, click Data > Data imports.
2. Click + Import in the upper-right corner and select Users or Events to create a new import.
   
3. Click the S3 tile.
4. Click Continue.
   

Step 1: Source and format

In this step, configure the S3 connection and specify the file format. In the Import name box, enter a name for this import to easily identify it on the Imports Dashboard.

Based on the type of import selected, your next steps might vary:

User Imports Events Imports

You can now select whether to import Registered users or Anonymous users. You can also choose to import both together. In the Select user type section, select either Registered users or Anonymous users. Both user types require different file names. For more information, refer to Naming Conventions: 

For Event imports, you can select which event it is that you want to import. From the Select event list, you can select which event you want to import. 

Based on the event you pick, your file names will be different. Your file names must begin with the Event Name of your chosen event. In the Select event list, MoEngage displays the names of all the available events for your app. Some of these events might have different display names and event names. When you hover over an event, MoEngage will show you what the file name it expects for that particular event

For example, the event App/Site Opened will have the value MOE_APP_OPENED, so the file names should begin with MOE_APP_OPENED. For more information, refer to Naming Conventions.  

To create a new event, perform the following steps:

1. Click the + Create event at the end of the Select event list. The Create new event dialog box is displayed.
2. In the Event name box, type a unique name for your event. By default, your Display Name is same as the Event Name.
   

You can go to the Data Management page to view or edit the generated event and your other MoEngage events. 

Add your S3 Credentials

In this first step, Source and format, you must specify MoEngage, which S3 connection to use, and the table from which to import. To get started, provide the following details: 

File Configurations

Select the DateTime format which is in your file name. As per the Naming Conventions, your file name should end with _<date time format>.csv or _<date time format>.json.  For more information, refer here to see which DateTime formats that MoEngage supports. The info message will update according to your chosen DateTime format so that you can easily cross-verify if your file names are as per expectations.

After you've configured your S3 Credentials and your file name format, MoEngage tries to connect to your S3 bucket and fetch the most recent file available with the specified format. Ensure you have at least one file available in the specified location otherwise, you cannot continue. After you're ready, click Fetch file(s)*  to fetch a recent file.

If your S3 credentials and chosen file format are correct, then MoEngage shows you the preview of the top five rows from a recent file:

These are some common errors that you can encounter while fetching the file from your S3 buckets:

• Given S3 Access Key is not valid: This indicates that your S3 credentials are incorrect. Enter the correct credentials and try to fetch the file again.
• Please check the folder path and try again: This indicates that either your folder path is incorrect or there are no CSV or JSON files present inside the given folder path.
• Found zero data rows in the file: This indicates that the fetched file either contained no rows beyond the header (for CSV files) or was entirely empty (for JSON files). MoEngage expects at least two rows (one header + one data row) for CSV files, or at least one data row for JSON files. For more information, refer to File Structure.

After you are satisfied with the preview of the file, click Next to continue to the mapping step.

Step 2: Import configuration and action

In this step, after configuring the S3 connection and file format, you will map your data columns to MoEngage attributes. The interface will list all columns from your file (derived from the CSV header or JSON keys), allowing you to map each one to its corresponding attribute in MoEngage

In the Map Columns section, you have the following:

1. Column name: This is the column name (picked from the first row of the fetched file in the previous step) to be mapped. Below the column name, MoEngage also shows a sample value (picked from the second row of the fetched file in the previous step) for your reference.
2. Map column attribute: You must select which MoEngage attribute you want to map the column to. You can also choose to create a new attribute. Some attributes support ingestion from multiple data types, so you need to pick the data type of the column as well. For the "DateTime" columns, you also need to pick the format. For the "DateTime" columns, you also need to pick the format
3. Action: You can optionally choose to skip the column. The skipped column will not be imported.

Depending on the type of import, there are a few mandatory mappings required:

User imports

• Registered Users  
  • • User ID: Your file should mandatorily have a column that contains a unique user identifier (that identifies a user’s account in your system).
• Anonymous Users 
  • Anonymous ID: You must mark a column from your file (email, mobile number, etc.) as an identifier of these users.
• Event imports
  • User ID: This column is used to match the users in MoEngage to your files.
  • Event Time: You must map either Event Time (UTC) or Event Time (App Timezone). If you have chosen Event Time (UTC), then the "Event Time" of the imported event will be converted to the timezone chosen in your MoEngage Dashboard settings:

After a mandatory mapping is marked, it reflects against the column name in the mapping table, and you will no longer be able to mark the column as skippable.

You can also create new Object attributes by clicking the + Create new attribute in the Select attribute list under the Map attribute mapping section.

Give your attribute a name and select its Data type. You can edit this as well as existing attributes from the Data Management page. Do note that any new attributes you create now is available in your app after the first successful import.

Add Fields

You can also add fields directly from the dashboard by clicking +Add Column. This creates a new Enter column name at the bottom of the list below where you can enter the attribute mapping manually.

Manifest Files

Optionally, you can choose to auto-map these columns by uploading a Manifest file. To upload a manifest file:

1. Click the Upload mapping file in the upper-right of the mapping table.
   
2. On the Upload mapping dialog box, upload your manifest file.
   
3. Click Done.

If it is a fresh import, you can find the sample manifest file, which can be used as a reference point as you create an appropriate mapping file for your import. If you duplicate an existing import and upload a mapping file, you will find your existing mapping file already added, which can be downloaded. Your mappings will be auto-configured accordingly. For any additional attributes mentioned in the manifest file but not currently present in MoEngage, a modal will open up, which will list down the attributes for you to create the attributes.

Any columns with non-MoEngage attributes will be left blank, and you can either manually map them or create a new attribute for them. Make sure your manifest file follows the expected conventions, as mentioned.

Any additional columns present in your Manifest File but not in your file will be ignored. Also, if the mapping for an existing column is not present in the manifest file, MoEngage will keep the mapping blank so that you can manually configure it.

Make sure if a column in the manifest file is mapped to a non-existent MoEngage Attribute, then the mapping will be blank, and you will need to first manually create a new attribute from the UI and then map it.

Send Import Notifications

To receive an email notification about the status of your imports, perform the following steps:

1. Select the Send import status to check box.
2. In the Select email id list, select the email ID. You can select up to 10 emails to send the status emails to.
   
   

   The import status email contains info about the following events:

   • • An import is created
     • An import is successful
     • An import has failed
3. After completing all mappings, click Next

Step 3: Scheduling

In this step, you must define when to import files from your S3 bucket. 

MoEngage supports the following types of imports:

• One time imports: You can run the import as soon as possible or at a later date and time (scheduled).
• Periodic: You can run your imports hourly, daily, weekly, or monthly, or with intervals and advanced configurations.

For each run, MoEngage will try to fetch a file with the configured DateTime format, so ensure your file names are configured properly. Optionally, you can specify whether the import should end after a specified set of occurrences or at a particular date. Click Done when ready.

Duplicate Imports

A duplicate import is considered when the:

• Users/ Events import types are the same.
• Event name/ Registered/ Anonymous/ All users import subtypes are the same.
• Event name/ Registered/Anonymous/All users import types with the same S3 folder path.
• Event name/ Registered/Anonymous/All users import types with the same Bucket name.

FAQ