Introduction

Databricks is a platform that allows organizations to store, analyze, and process large volumes of structured and semi-structured data in a highly scalable and efficient manner. With its unique architecture, organizations can consolidate their data, perform quick analytics, and gain valuable data-driven insights accessible to all users.

MoEngage <> Databricks

With this integration, you can export data from campaign interaction events to your Databricks tables.

Integration 

If not done already, you need to set up a MoEngage <> Databricks integration for your account, refer to Databricks Integration help doc.

Export events from MoEngage to Databricks

Step 1: Make sure your user's assigned role has required permissions

MoEngage will need WRITE access to your Databricks database to store data. You have the option to either grant these permissions to an existing database user or create a new dedicated database user specifically for MoEngage.

GRANT USE CATALOG ON CATALOG `<CATALOG_NAME>` TO `<EMAIL_ID_OF_THE_USER>`;
GRANT USE SCHEMA, CREATE TABLE ON SCHEMA `<CATALOG_NAME>`.`<SCHEMA_NAME>` TO `<EMAIL_ID_OF_THE_USER>`;

Remember to change the following values:

1. <CATALOG_NAME> - The name of your catalog.
2. <SCHEMA_NAME> - The name of your database/schema.
3. <EMAIL_ID_OF_THE_USER> - The email ID of the user who created the token.

Step 2: Add a New Databricks Export on the MoEngage App Marketplace

To add a new Databricks export, perform the following steps:

1. On the left navigation menu in the MoEngage dashboard, click App marketplace.
2. On the App Marketplace page, search for Databricks.
3. Click the Databricks tile.
4. On the Databricks page, go to the Integrate tab and click +Add Export.
5. Enter the following details:
   

   Field	Required	Description
   Export name	Yes	Type a name for the Databricks export.
   Databricks connection	Yes	Select your Databricks connection.
   Export into schema	Yes	Select the schema that MoEngage uses to create new tables and export your events into.
   User properties	Optional	You can select the user properties (user properties and device properties) you want to export with your events.
   Select events	Yes	Select the events you want to export to Databricks. You can choose to export All events, including existing and new ones coming into MoEngage, or manually select specific events to export.

   
   
6. Click Save export.

Upon clicking Save export, MoEngage starts dumping(appending) events to the newly created table every hour.

Export Frequency

We will dump the data at hourly intervals.

Sample Table Structure

MoEngage will create the following two tables in your schema:

1. test_connection_moengage - We will use this table to verify if your connection details are valid from time to time.
2. moe_events_<table_creation_timestamp> - We will dump all your events in this table. All exports set up under a single connection will be appended to the same table.

The overall structure of the table will remain fixed for all types of events. Any new user property or event attribute that is exported will be part of the user_attributes or event_attributes column respectively. All the events will be dumped in a single table containing necessary information such as event names, event times, etc for easy analysis.