SFTP Imports

Overview

You can import users and events with MoEngage through files stored in your SFTP folders.

Types of Imports

MoEngage can import the following from your SFTP folders:

Registered users: Users who are already registered on MoEngage.
Anonymous users: Users who are not yet registered on MoEngage.
Events (standard and user-defined): Standard events like campaign interaction events and your user-defined events.
Auxiliary data: Users who are added on a temporary basis.

Prepare the Files

For more information about supported file types, naming conventions, file structure, and so on, refer to the File Imports help doc. After your files are ready, place them in an SFTP folder.

Folder Structure

Place all files into a folder. You can configure the folder path when you set up the import.
MoEngage does not look for files inside any sub-folders.

Set Up Imports from SFTP

info	Prerequisites Ensure you have an SFTP connection set up in the App Marketplace with relevant permissions. For more information, here. Contact your MoEngage Customer Success team to enable the Data menu option on the left navigation in the MoEngage dashboard.

To set up imports from SFTP, perform the following steps:

On the left navigation menu in the MoEngage dashboard, click Data, and then click Data Imports.
Click + Import, and then select your source to create a new import for either Users or Events. The Import users dialog box appears.
Click the SFTP tile and click Continue.

Step 1: Source and Format

In this step, configure the SFTP connection and specify the file format. In the Import name box, enter a name for this import to easily identify it on the Imports Dashboard.

Based on the type of import selected, your next steps might vary:

You can now select whether to import Registered users or Anonymous users. You can also choose to import both together. In the Select user type section, select either Registered users or Anonymous users. Both user types require different file names. For more information, refer to Naming Conventions.

For Event imports, you can select which event it is that you want to import. From the Select event list, you can select which event you want to import

Based on the event you pick, your file names will be different. Your file names must begin with the Event Name of your chosen event. In the Select event list, MoEngage displays the names of all the available events for your app. Some of these events might have different display names and event names. When you hover over an event, MoEngage will show you what the file name it expects for that particular event:

For example, the event App/Site Opened will have the value MOE_APP_OPENED, so the file names should begin with MOE_APP_OPENED. For more information, refer to Naming Conventions.

To create a new event, perform the following steps:

Click the + Create event at the end of the Select event list. The Create new event dialog box is displayed.
In the Event name box, type a unique name for your event. By default, your Display Name will be the same as the Event Name.

You can go to the Data Management page to view or edit this event and your other MoEngage events.

info	Information The newly created events do not appear on the Data Management page until the initial import is successful.

Select Your SFTP Credentials

In the first step, Source and format, you must notify MoEngage which SFTP connection to use and the folder to import from. To get started, perform the following steps:

In the SFTP connection list, select the SFTP connection to use for this import. If you have not yet created an SFTP connection, click + Add connection at the end of the SFTP connection list. For more information, refer to Configure a New SFTP Connection.
Enter the following details:

Field	Description
Folder URL	Type the folder URL. This is the path of the folder where your files exist. MoEngage does not look inside sub-folders. Do not include trailing slashes. Note: MoEngage does not currently support root folders, so ensure all your files are located within a single folder. How do I find this? Your folder path is the complete path beginning with `sftp://[host-name]/` excluding the trailing slash. Example: `sftp://files.hostname.com/my_folder` `sftp://127.0.0.1/my_folder/sub_folder`
File Type	Select the type of file you want to import. MoEngage supports importing the following files: Comma-separated Values (.`CSV`) Newline-delimited JSON files (.`JSON`)
Is your file encrypted?	If you are importing encrypted files, select the Is your file encrypted? check box. MoEngage supports importing encrypted files. It currently supports PGP Encryption. For more information, refer to How Does MoEngage Handle PGP Encryption?
Decryption Key	Type the decryption key. This field will be visible when you select the Is your file encrypted? check box. This is your private PGP key. You can generate the PGP keys by yourself or have MoEngage generate them. Example: -----BEGIN PGP PRIVATE KEY BLOCcK----- `xcMGBGVqq74BCADKpWRLSAdcTtDKpT+LCUCqDoZQ2dlgeBiXNovBVyTPlAOPQSqX bnNGs+U4RpHebjvTxaWCLDBGYOvtZbtv6WvPIF7lzXrIv+DcgLMBQp2b3t+ohRBi 41Ch8CZ04F2lxbVBo` -----END PGP PRIVATE KEY BLOCK-----
Signing Key	Type the signing key. This field will be visible when you select the Is your file encrypted? check box. If a signing key is configured, enter it in the Signing key section in the following format, for example: -----BEGIN PGP PUBLIC KEY BLOCK----- `xqq74BCADKpWRLSAdcTtD45nl24KpT+LCUCqDoZQ2dlgeBiXNovBVyTPlAOPQSqX bnNGs+U4RpHebjvTxaWCLDBGYOvtZbtv6WvPIF7lzXrIv+DcgLMBQp2b3t+ohRBi 41Ch8CZ04F2lxboksCwnl31laEuQkk8MERlcthB9AiKvSEf6jihLcPSVVanrQUQN77UZf sFCynJXweqijZxtmUl9V1838sUVOoTzfmGwu7PzV99tAiOlxBjYF9SXIYY/GOoJN Zuz8uOaXWtFPmV536uQT` -----END PGP PUBLIC KEY BLOCK-----
Key passphrase	Type the key passphrase, which is generated with the encryption key pair.

File Configurations

Select the DateTime format which is in your file name. As per the Naming Conventions, your file name should end with _<date time format>.csvor _<date time format>.json. Refer to the table provided earlier to see which DateTime formats MoEngage supports. The info message will update according to your chosen DateTime format so that you can easily cross-verify if your file names are as per expectations.

After you've configured your SFTP credentials and file name format, MoEngage will try to connect to your SFTP server and fetch the most recent file available with the specified format. Ensure you have at least one file available in the specified location; otherwise, you will not be able to continue. After you're ready, click Fetch file(s)* to fetch a recent file.

If your SFTP credentials and chosen file format are correct, MoEngage will show you the preview of the top five rows from a recent file:

These are some common errors that you can encounter while fetching the file from your SFTP servers:

Given credentials are not correct: This indicates that your SFTP credentials are incorrect. You must enter the correct credentials and try to fetch the file again.
Please check the folder path and try again: This indicates that either your folder path is incorrect or there are no CSV or JSON files present inside the given folder path.
Found zero data rows in the file: This indicates that the fetched file either contained no rows beyond the header (for CSV files) or was entirely empty (for JSON files). MoEngage expects at least two rows (one header + one data row) for CSV files, or at least one data row for JSON files. For more information, refer to File Structure.

After you are satisfied with the preview of the file, click Next to continue to the mapping step.

Step 2: Import Configuration and Action

In this step, after configuring the SFTP connection and file format, you will map your data columns to MoEngage attributes. The interface will list all columns from your file (derived from the CSV header or JSON keys), allowing you to map each one to its corresponding attribute in MoEngage:

In the Map Columns section, you have the following:

Column name: This is the column name (picked from the first row of the fetched file in the previous step) to be mapped. Below the column name, MoEngage also shows a sample value (picked from the second row of the fetched file in the previous step) for your reference.
Map column to attribute: You must select which MoEngage attribute you want to map the column to. You can also choose to create a new attribute. Some attributes support ingestion from multiple data types, so you need to pick the data type of the column as well. For the "DateTime" columns, you also need to pick the format. For the "DateTime" columns, you also need to pick the format.
Action: You can optionally choose to skip the column. The skipped column will not be imported.

Depending on the type of import, there are a few mandatory mappings required:

User imports

Registered Users
- - User ID: Your file should mandatorily have a column that contains a unique user identifier (that identifies a user’s account in your system).
Anonymous Users
- Anonymous ID: You must mark a column from your file (email, mobile number, etc.) as an identifier of these users.
Event imports
- User ID: This column is used to match the users in MoEngage to your files.
- Event Time: You must map either Event Time (UTC) or Event Time (App Timezone). If you have chosen Event Time (UTC), then the "Event Time" of the imported event will be converted to the timezone chosen in your MoEngage Dashboard settings:

After a mandatory mapping is marked, it reflects against the column name in the mapping table, and you will no longer be able to mark the column as skippable.

Add Fields

You can also add fields directly from the dashboard by clicking +Add Column. This creates a new Enter column name at the bottom of the list below where you can enter the attribute mapping manually.

Manifest Files

Optionally, you can choose to auto-map these columns by uploading a Manifest file. To upload a manifest file:

Click the Upload mapping file in the upper-right of the mapping table.
On the Upload mapping dialog box, upload your manifest file.
Click Done.

If it is a fresh import, you can find the sample manifest file, which can be used as a reference point as you create an appropriate mapping file for your import. If you duplicate an existing import and upload a mapping file, you will find your existing mapping file already added, which can be downloaded. Your mappings will be auto-configured accordingly. For any additional attributes mentioned in the manifest file but not currently present in MoEngage, a modal will open up, which will list down the attributes for you to create the attributes.

CreateNewAttributes (1).png

Any columns with non-MoEngage attributes will be left blank, and you can either manually map them or create a new attribute for them. Make sure your manifest file follows the expected conventions, as mentioned.

Any additional columns present in your Manifest File but not in your file will be ignored. Also, if the mapping for an existing column is not present in the manifest file, MoEngage will keep the mapping blank so that you can manually configure it.

Make sure if a column in the manifest file is mapped to a non-existent MoEngage Attribute, then the mapping will be blank, and you will need to first manually create a new attribute from the UI and then map it.

Send Import Notifications

To receive the email notification about the status of your imports, perform the following steps:

Turn on the Send import status toggle.
In the Select email id list, select the email ID. You can select up to 10 emails to send the status emails to.

The import status email contains info about the following events:
- - An import is created
  - An import is successful
  - An import has failed
After completing all mappings, click Next.

Step 3: Scheduling

In this step, you must define when to import files from your SFTP server.

We support the following types of imports:

One time imports: You can run the import as soon as possible or at a later date and time (scheduled).
Periodic: You can run your imports hourly, daily, weekly, or monthly, or with intervals and advanced configurations.

For each run, MoEngage will try to fetch a file with the configured DateTime format, so ensure your file names are configured properly. Optionally, you can specify whether the import should end after a specified set of occurrences or at a particular date. Click Done when ready.

Duplicate Imports

A duplicate import is considered when the:

Users/ Events import types are the same.
Event name/ Registered/ Anonymous/ All users import subtypes are the same.
Event name/ Registered/Anonymous/All users import types with the same SFTP folder path.

FAQ

arrow_drop_down My imports have failed. How do I check what went wrong?

Click the ellipsis on the right and click View to look up the Import details. Hover over the Failed Status to learn the reason.

arrow_drop_down What happens when there is an error in fetching the files from the S3/SFTP folder?

By default, imports are not retried when there is a failure. You can, however, configure to receive an email alert upon failure.

arrow_drop_down What if a scheduled import adds the data into a recently archived segment?

In such cases, the new data will still be added to the archived segment. You can unarchive the segment as required.

arrow_drop_down How does MoEngage pick up new files in a periodic run?

In a scheduled run for a given workspace, MoEngage picks up all the new files identified in the S3/SFTP folder based on file name prefix and processes them. Data import in a given MoEngage workspace uniquely identifies these files with the combination of Folder location and File name.

arrow_drop_down Will a periodic Data import pick up an earlier processed file?

No. A file once processed will not be processed again if the file name remains the same and in the same folder location. However if the same file is also placed in a different folder location, a new data import setup in MoEngage with the new folder location configured will pick up the file.

arrow_drop_down Can a CSV file contain both User and Event data?

No. Data imports in your MoEngage workspace dashboard have to be setup separately for User data and Event data, hence the files should either contain User attribute data or Event data.

arrow_drop_down Can an import be stopped while it's running?

Once an import process starts, it can't be stopped midway. This is because the data goes through several steps, and interrupting it could lead to incomplete or inconsistent results. It's best to let the current import finish.

arrow_drop_down How can I stop future runs of a scheduled import?

Yes, you can stop future scheduled imports from running automatically. To do this, find the import schedule and select the Archive option from the Actions menu on the Data Imports dashboard. This will prevent it from running on its next scheduled time.

arrow_drop_down What is NDJSON, and why is it required?

NDJSON (Newline-Delimited JSON) means each line in your .json file is a complete JSON object, separated by a new line. This format is required to process extensive JSON datasets line by line, allowing for scalable and robust imports.

arrow_drop_down Are compressed files (e.g., .gz, .zip) supported?

No, we don’t support importing compressed files.

arrow_drop_down If I change the file type of an existing import, what happens to my mappings?

If you change the file type, the existing preview will be removed, and the list of available files will be re-fetched based on the new file type. You will then need to select a file for preview again, and your mappings will need to be configured based on the new file's structure.

arrow_drop_down What should I do if an import seems stuck or is taking a long time?

If an import appears to be stuck or is taking longer than usual, it's best to wait. The system has checks in place to handle these situations automatically and will retry if necessary. Manually starting the same import again while it's still processing can cause conflicts and may prevent the original import from completing successfully.