Overview

MoEngage allows you to import users and events through files stored inside your SFTP folders.

Types of Imports

MoEngage can import the following from your SFTP folders:

1. Registered Users: These are users who are already registered on MoEngage.
2. Anonymous Users: These are users who are not yet registered on MoEngage.
3. Events (Standard and User Defined): MoEngage can import standard events like Campaign Interaction Events as well as your own user-defined events.
4. Auxiliary Data: These are users added on a temporary basis.

Prepare the Files

Refer to the File Imports help doc to understand the supported file types, naming conventions, file structure, etc. Once you are ready with the files, you need to place them inside an SFTP folder.

Folder Structure

Place all files into a folder. You can configure the folder path while setting up the import.

Note that MoEngage will not look for files inside any sub-folders.

Set Up Imports from SFTP

To set up imports from SFTP, perform the following steps:

1. On the left navigation menu in the MoEngage dashboard, click Data, and then click Data Imports.
2. Click + Import, and then select your source to create a new import for either Users or Events.
   

   info	Information Contact your MoEngage Customer Success team to enable the Data menu option on the left navigation in the MoEngage dashboard.

3. Click the SFTP tile and click Continue.
   

Step 1: Select Your SFTP Connection and File Format

User Imports

You will now have the option to select whether you want to import Registered users or Anonymous users:

Choose which one you want to import. Both registered users import and anonymous users import expect different file names. Refer to the Naming Conventions section for more information.

Event Imports

For Event imports, you will now have the option to select which event it is that you want to import:

Based on the event you pick, your file names will be different. Your file names must begin with the Event Name of your chosen event. In the "Select event" dropdown, we will show you the Display Names of all the available events in your MoEngage app. Some of these events might have different Display Names and Event Names. When you're selecting an event, MoEngage will show you what is the file name it expects for that particular event in the Event Description box:

In the above example, "App/Site Opened" is the Display Name of the chosen event and its Event Name is MOE_APP_OPENED, so the file names should begin with MOE_APP_OPENED_. Refer to the Naming Conventions section for more information.

You can also create a new event by clicking on the "+ Create event" option at the end of the dropdown:

Give your event a name that is unique. By default, your Display Name will be the same as the Event Name. You can go to the Data Management page to view or edit this event and also your other MoEngage events. Do note that any new events you create now will only be available in your app after the first successful import.

Select Your SFTP Credentials

In the first step, Source and format, you must notify MoEngage which SFTP connection to use and the folder to import from. To get started, perform the following steps:

1. In the SFTP connection list, select the SFTP connection to use for this import. If you have not yet created an SFTP connection, click + Add connection at the end of the SFTP connection list. For more information, refer to Configure a New SFTP Connection.
   
2. Enter the following details:

Field	Description
Folder URL	Type the folder URL. This is the path of the folder where your files exist. MoEngage does not look inside sub-folders. Do not include trailing slashes. Note: MoEngage does not currently support root folders, so ensure all your files are located within a single folder. How do I find this? Your folder path is the complete path beginning with sftp://[host-name]/ excluding the trailing slash. Example: sftp://files.hostname.com/my_folder sftp://127.0.0.1/my_folder/sub_folder
Is your file encrypted?	If you are importing encrypted files, select the Is your file encrypted? check box. MoEngage supports importing encrypted files. It currently supports PGP Encryption. For more information, refer to How Does MoEngage Handle PGP Encryption?
Decryption Key	Type the decryption key. This field will be visible when you select the Is your file encrypted? check box. This is your private PGP key. You can generate the PGP keys by yourself or have MoEngage generate them. Example: -----BEGIN PGP PRIVATE KEY BLOCcK----- xcMGBGVqq74BCADKpWRLSAdcTtDKpT+LCUCqDoZQ2dlgeBiXNovBVyTPlAOPQSqX bnNGs+U4RpHebjvTxaWCLDBGYOvtZbtv6WvPIF7lzXrIv+DcgLMBQp2b3t+ohRBi 41Ch8CZ04F2lxbVBo -----END PGP PRIVATE KEY BLOCK-----
Signing Key	Type the signing key. This field will be visible when you select the Is your file encrypted? check box. If a signing key is configured, enter it in the Signing key section in the following format, for example: -----BEGIN PGP PUBLIC KEY BLOCK----- xqq74BCADKpWRLSAdcTtD45nl24KpT+LCUCqDoZQ2dlgeBiXNovBVyTPlAOPQSqX bnNGs+U4RpHebjvTxaWCLDBGYOvtZbtv6WvPIF7lzXrIv+DcgLMBQp2b3t+ohRBi 41Ch8CZ04F2lxboksCwnl31laEuQkk8MERlcthB9AiKvSEf6jihLcPSVVanrQUQN77UZf sFCynJXweqijZxtmUl9V1838sUVOoTzfmGwu7PzV99tAiOlxBjYF9SXIYY/GOoJN Zuz8uOaXWtFPmV536uQT -----END PGP PUBLIC KEY BLOCK-----
Key passphrase	Type the key passphrase, which is generated with the encryption key pair.

Import Name

Enter a name for this import to identify it on the Imports Dashboard easily.

File configurations

Select the DateTime format that will be in your file name. As per the Naming Conventions defined above, your file name should end with _<date time format>.csv. Refer to the table provided earlier to see which DateTime formats we support. The info message below will update according to your chosen DateTime format so that you can easily cross-verify if your files names are as per expectations.

After you've configured your SFTP Credentials and your file names format, MoEngage will try to connect to your SFTP server and fetch the most recent file available with the specified format. Ensure you have at least one file available in the specified location otherwise, you will not be able to continue. Once you're ready, click the "Fetch file" button to fetch a recent file.

If your SFTP credentials and chosen file format are correct, then MoEngage will show you the preview of the top 5 rows from a recent file:

However, sometimes, there can be an error while fetching the file from your SFTP servers. These are some common errors:

• Given credentials are not correct - This means that your SFTP credentials are incorrect. Please enter the correct credentials and try to fetch the file again.
• Please check the folder path and try again - This means that either your folder path is incorrect or there are no CSV files present inside the given folder path.
• Found zero data rows in the file - This means that the file MoEngage fetched did not have any rows after the header row (the total row count of the file is 1). We expect at least 2 rows (1 header row + 1 data row) in the file since the first row is always treated as the header row. Learn more about the File Structure.

Once you are satisfied with the preview of the file, click Next to continue to the mapping step.

Step 2: Map your columns to MoEngage Attributes

In this step, you need to map the columns of your CSV files to the attributes present in MoEngage. All your columns will be shown one below the other:

1. Column name - This is the column name (picked from the first row of the fetched file in the previous step) to be mapped. Below the column name, we also show a sample value (picked from the second row of the fetched file in the previous step) for your reference.
2. Map attribute - Here, you need to pick which MoEngage attribute you want to map the CSV column to. You can also choose to create a new attribute. Some attributes support ingestion from multiple data types, so you need to pick the data type of the column as well. For the "DateTime" columns, you also need to pick the format.
3. More actions - You can optionally choose to skip the column. The skipped column will not be imported.

Depending on the type of import, there are a few mandatory mappings required:

1. User imports - You need to select the column having:
   1. Registered - 
      1. User ID - Your file should mandatorily have a column that contains a unique user identifier (that identifies a user’s account in your system).
   2. Anonymous - 
      1. Anonymous ID - You need to mark a column from your file (email, mobile number, etc.) as an identifier of these users.
2. Event imports - You need to select the column having:
   1. User ID - This column will be used to match the users in MoEngage to your CSV files.
   2. Event Time - You need to map either Event Time (UTC) or Event Time (App Timezone). If you have chosen Event Time (UTC), then the "Event Time" of the imported event will be converted to the timezone chosen in your MoEngage Dashboard settings:

Once a mandatory mapping is marked, it will reflect against the column name in the mapping table, and you will no longer be able to mark the column as skippable.

Adding Fields

You can also add fields directly from the dashboard by clicking on +Add Field. This will create a new field at the bottom of the list below where you can enter the attribute mapping manually.

Manifest Files

Optionally, you can choose to auto-map these columns by uploading a manifest file.

To upload a manifest file, click the "Upload mapping file" option on the top-right of the mapping table.

If it is a fresh import, you can find the sample manifest file, which can be used as a reference point as you create an appropriate mapping file for your import. If you duplicate an existing import and upload a mapping file, you will find your existing mapping file already added, which can be downloaded.

Upload your manifest file and click Done. Your mappings will be auto-configured accordingly. For anyadditional attributes mentioned in the manifest file but not currently present in MoEngage,a modal will open up, which will list down these attributes for you to create the attributes.

Any columns with non-MoEngage attributes will be left blank, and you can either manually map the column or create a new attribute for it. Make sure your manifest file follows the expected conventions as mentioned.

Any additional columns that are present in your Manifest File but not in your CSV file will be ignored. Also, if the mapping for an existing CSV column is not present in the manifest file, MoEngage will keep the mapping blank for you to manually configure it.

Do note that if a column in the manifest file is mapped to a non-existent MoEngage Attribute, then the mapping will be blank, and you will need to first manually create a new attribute from the UI and then map it.

Sending Import Notifications

You can, optionally, choose to get notified about your imports' statuses via email. Select up to 10 emails to send the status emails to. You will receive an email in the following events:

1. An import was successful.
2. An import failed, along with the possible cause of failure.

When you are satisfied with all mappings, click Next. If there are any errors, correct them to move forward.

Step 3: Select the Import frequency

In this step, you need to define when to import files from your SFTP server. We support the following types of imports:

1. One-Time Imports: You can either choose to run the import as soon as possible or at a later date and time (scheduled)
2. Periodic: You can choose to run your imports hourly, daily, weekly, or monthly, or with intervals and advanced configurations.

For each run, MoEngage will try to fetch a file with the configured DateTime format, so ensure your file names are configured properly.

Optionally, you can specify whether the import should end after a specified set of occurrences or at a particular date. Click "Done" when ready.

Duplicate Imports

You can only configure one unique import at a time. An import is considered a duplicated import when all of these are the same:

1. Import Type - Users / Events
2. Import Sub-Type - Event Name / Registered / Anonymous - If the event name is the same, or there already exists a registered/anonymous users import with the same SFTP path.
3. SFTP Folder Path - The complete folder path, including hostname. 

As long as any one of the above is different, the import is considered unique.

FAQ

1. My imports have failed. How do I check what went wrong?

   Click the ellipsis on the right and click View to look up the Import details. Hover over the Failed Status to learn the reason.

2. What happens when there is an error in fetching the files from the S3/SFTP folder?

   By default, imports are not retried when there is a failure. You can, however, configure to receive an email alert upon failure.
   

3. What if a scheduled import adds the data into a recently archived segment?

   In such cases, the new data will still be added to the archived segment. You can unarchive the segment as required.

4. How does MoEngage pick up new files in a periodic run?In a scheduled run for a given workspace, MoEngage picks up all the new files identified in the S3/ SFTP folder based on file name prefix and processes them. Data import in a given MoEngage workspace uniquely identifies these files with the combination of Folder location and File name. 
5. Will a periodic Data import pick up an earlier processed file?No. A file once processed will not be processed again if the file name remains the same and the in the same folder location. However if the same file is also placed in a different folder location, a new data import setup in MoEngage with the new folder location configured will pick up the file.
6. Can a CSV file contain both User and Event data?No. Data imports in your MoEngage workspace dashboard have to be setup separately for User data and Event data, hence the files should either contain User attribute data or Event data.