Integrations

Amazon Kinesis Firehose
CategoriesAll

Audience

Amazon Kinesis is a platform for streaming data on AWS, offering powerful services to make it easy to load and analyze streaming data, and also providing the ability for you to build custom streaming data applications for specialized needs.

The Amazon Kinesis Firehose Event Integration supports data from all platforms.

Prerequisites

mParticle’s Audience Integration with Amazon Kinesis sends Audience data to a Kinesis Delivery Stream. All audience messages contain an ‘Audience Name’, so it is possible to send multiple audiences to the same Delivery Stream. To set up a Delivery Stream, follow Amazon’s instructions here. To set up the integration, you will need to:

  1. Create a Stream
  • The Kinesis Stream Name and Kinesis Service Region are required for mParticle setup.
  • When you create your stream, be sure to set the Source to Direct PUT or other sources .
  1. Create an IAM user
  • Be sure to save the credentials file which contains the Access Key Id and Secret Access Key required for mParticle setup.
  1. Create a Custom Policy. Use one of the following methods to create the policy:

    1. Manual Policy Creation
    • Select Firehose as the AWS Service
    • Include the PutRecord and DescribeDeliveryStream permissions
    • Enter the ARN of your stream.
    • Click Add Statement
    1. Create Policy from JSON
    • Paste the template policy below into the “Policy Document” field. Be sure to replace the {region}, {account-id} and {stream-name} with your specific values.
    {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "firehose:DescribeDeliveryStream",
                   "firehose:PutRecord"
               ],
               "Resource": [
                   "arn:aws:firehose:{region}:{account-id}:deliverystream/{stream-name}"
               ]
           }
       ]
    }
  2. Assign Customer Policy to User.

Data Format

mParticle will forward event data to Amazon Kinisis Firehose as base64 encoded JSON. An example is below (for readability the example has been left unencoded). You can control which events and attributes are forwarded in the Connection Settings.

{
    "DeliveryStreamName": "my-stream",
    "Record": {
        "Data": {
           
           "events":[{"data":{"event_name":"Level_Acheived"},"event_type":"custom_event"}],"unique_id":"abc123123","message_id":"521e0ae1-164d-  4495-8e41-95be88fa7e42","message_type":"events","schema_version":1,"device_info":{"ios_advertising_id":"68b7032b-1c2d-4533-92b8-3fb143ebeadf","android_advertising_id":"f9f1b997-f228-415f-99e1-ed7c223e238d","limit_ad_tracking":false,"is_dst":false},"application_info":{"application_version":"1.3","package":"test-app-id-550854415"}}
        }
    }
}

Rate Limits

Amazon Kinesis Firehose imposes standard rate limits that vary depending on your Service Region. If your throughput is close to or in excess of these limits, mParticle will make retries in an exponential backoff pattern. If you continually exceed the limit, you will see 500 - Internal Server Error and Retry Limit Exceeded errors in your System Alerts dashboard. If this occurs, you may need to request a limit increase from Amazon.

Configuration Settings

Setting Name Data Type Default Value Description
Access Key ID string This is your IAM user’s Access Key Id, which can be found on your IAM dashboard, or in the credentials.csv file that you might have downloaded after creating the IAM user.
Secret Access Key string This is your IAM user’s Secret Access Key, which can be found in the credentials.csv file that you might have downloaded after creating the IAM user.

Connection Settings

Setting Name Data Type Default Value Description
Stream Name string This is your Kinesis Firehose Delivery Stream name.
Kinesis Service Region string us-east-1 Endpoint for the Kinesis Firehose instance. Defaults to US East (N. Virginia) also known as us-east-1
Unique ID string An optional string identifier for your app that will be forwarded with each event batch. Standard app identifiers (Apple OS Bundle ID, Android Package Name) will be forwarded regardless of this setting.
Send Anonymous Users bool True If enabled, users that are only identified by their mParticle ID will be sent.
Send Lifecycle Events bool True If enabled, lifecycle events (application start/stop, session start/end) will be forwarded.
Send Screen Views bool True If enabled, screen view events will be forwarded.
Send Crash Events bool True If enabled, app crashes will be forwarded.
Send Network Performance Events bool True If enabled, network performance events will be forwarded.
Send Custom Events bool True If enabled, custom app events will be forwarded.
Send Push Registrations and Receipts bool True If enabled, push registration and receipt notifications will be forwarded.
Send as Batch bool True If enabled, this setting will cause your app’s events to be sent in (roughly) 10-minute batches per device. If disabled, mParticle will POST each event to you individually, as its received. This setting is ignored if “Wait for Complete Batch” is enabled.
Wait for Complete Batch bool False If enabled, mParticle will POST events to you in batches only after a user session has ended, so that each event batch you receive will represent a full session of user activity within your app.
Include Location Information bool True If enabled, location data will be forwarded with event data whenever possible.
Include User Identities bool True If enabled, user identity information will be forwarded with event batches.
Send Profile Change Events bool True If enabled, mParticle will forward ID profile events, such as user sign ups, logins logouts, updates, and deletes.
Send Commerce Events bool True If enabled, commerce events will be forwarded.
Include Metadata bool True If enabled, the following metadata - application_info, device_info and source_info will be forwarded.
Include User Attribute Change Events bool False If enabled, User Attribute Change Events will be forwarded.
Include User Identity Change Events bool False If enabled, User Identity Change Events will be forwarded.

Was this page helpful?