Warehouse Sync API v2 Overview

Use the Warehouse Sync API to enrich user profiles in mParticle with insights derived from data in Snowflake, Google BigQuery, or Amazon Redshift. The Warehouse Sync API allows you to build a headless data infrastructure.

To sync with your data warehouse, use the API to:

  1. Define an ingest connection.
  2. Define an ingest data model. The data model defines the data structure and important columns including the SQL query to perform.
  3. Define an ingest data pipeline: connect a connection and model together and specify when and how often to execute a sync.

Prerequisites to Accessing the API

To authenticate when using the Warehouse Sync API, you will need a new set of API credentials.

  1. Navigate to API Credentials in the mParticle UI, and click Add Credential.
  2. Give your new credential a descriptive name, check the box next to Platform and select Admin from the Permissions dropdown menu.
  3. Click Save, and copy the Client ID and Client Secret. You will use these when fetching an OAuth access token.

Authentication

After creating your new API credential by following the steps above, you can authenticate by issuing a POST request to mParticle’s SSO token endpoint.

https://sso.auth.mparticle.com/oauth/token

The JSON body of the request must contain:

  • client_id - your Client ID that you saved when creating your new API credential
  • client_secret - your Client Secret that you saved when creating your new API credential
  • audience - set to a value of "https://api.mparticle.com"
  • grant_type - set to a value of "client_credentials"

Curl Syntax

curl --request POST \
  --url https://sso.auth.mparticle.com/oauth/token \
  --header 'content-type: application/json' \
  --data '{"client_id":"...","client_secret":"...","audience":"https://api.mparticle.com","grant_type":"client_credentials"}'

Sample Raw HTTP Request

POST /oauth/token HTTP/1.1
Host: sso.auth.mparticle.com
Content-Type: application/json

{
  "client_id": "your_client_id",
  "client_secret": "your_client_secret",
  "audience": "https://api.mparticle.com",
  "grant_type": "client_credentials"
}

Using your Bearer Token

A successful POST request to the token endpoint will result in a JSON response as follows:

{
  "access_token": "YWIxMjdi883GHBBDnjsdKAJQxNjdjYUUJABbg6hdI.8V6HhxW-",
  "expires_in" : 28800,
  "token_type": "Bearer"
}

Subsequent requests to the API can now be authorized by setting the Authorization header as follows:

Authorization: Bearer YWIxMjdi883GHBBDnjsdKAJQxNjdjYUUJABbg6hdI.8V6HhxW-

Versioning

Once you have authenticated, the API resources can be accessed at https://api.mparticle.com/platform/v2/. Subsequent updates to the API that introduce breaking changes will be published with a new version number in the URL.

HTTP Methods

This API uses the HTTP methods GET, POST, PUT, and DELETE.

Headers

This API accepts and sometimes requires the following headers:

Header Required Method Notes
Authorization Required GET, POST, PUT, DELETE
Content-Type Optional GET, POST, PUT, DELETE

Request Bodies

All POST/PUT requests should send JSON as the Request Payload, with Content-Type set to application/json.

Limits

In addition to the standard default service limits, note the following limits specific to the Warehouse Sync API:

Limit Value Notes
Max number of Active Pipelines 5
Historical Record Limit 24 million For new interval based pipelines, there is a 24 million record limit while retrieving records before the schedule start time. See sync mode from and until to filter data to load.
Column limit 100
Record count limit per hourly interval 1 million
Record count limit per daily interval 24 million
Record count limit per weekly interval 40 million
Record count limit per monthly interval 40 million
Record count limit per once request 40 million
Record count limit per on-demand request 24 million Applicable when the trigger API is used

Was this page helpful?