Event

Amazon S3 provides a simple web-services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web.

Supported Features

• Event Forwarding

Prerequisites

In order to take advantage of the Amazon S3 integration, you’ll need the name of the S3 bucket in which you want to store your data. You’ll also need to provide access for mParticle to write data to your bucket. There are two ways to do this:

Option 1 is role-based authentication. In this option, you apply a policy to the bucket itself, granting mParticle access to write to your bucket. Most organizations choose this path.

Option 2 is user-based authentication. In this option, you create a user under your own AWS account, give the user permission to write to your bucket, and provide the credentials for that user to mParticle.

Option 1 - Role-Based Authentication

To use role-based authentication, you’ll need to create an S3 bucket.

The bucket name must begin with mp-forwarding-, and setup a Bucket Policy. Use this link docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#arn-syntax-s3 for information on S3 ARN Syntax.

Sample ARN syntax for S3 is: arn:aws:s3:::**bucket_name**/**key_name*.

To set up a bucket policy:

1. Create an S3 Bucket. The bucket name must begin with mp-forwarding-.

2. Create a New Policy from the template below and be sure to replace the word {bucket} with the name of the S3 bucket created in step 1.

3. Assign Policy to Bucket. In the Bucket Policy Editor

   1. Click Properties
   2. Click Permissions
   3. Click Add bucket policy
   4. Paste the JSON from step 2 above

AWS S3 Bucket Policy Template

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::{bucket}/*",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::338661164609:role/Prod-Role-DG12-Default"
                ]
            }
        },
        {
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::{bucket}/*",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::338661164609:role/role-lambda-verifyrequest"
                ]
            }
        }
    ]
}

Option 2 - User-Based Authentication

To use user-based authentication, you’ll need the name of an S3 bucket and the credentials of an Identity and Access Management (IAM) user that has access to S3. Use this link docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#arn-syntax-s3 for information on S3 ARN Syntax. Sample ARN syntax for S3 is: arn:aws:s3:::**bucket_name**/**key_name*.

Refer to the steps below for Amazon setup if you are Assigning Customer Policies to a User:

1. Create an S3 Bucket

   • The Bucket Name is required for mParticle setup and for creating the custom policy

2. Create an IAM user

   • Be sure to save the credentials file which contains the Access Key Id and Secret Access Key required for mParticle setup.

3. Create a Custom Policy

   Use one of the following methods to create the policy:

   • Create a Policy from Scratch

     1. Select Amazon S3 as the AWS Service.
     2. Include the following actions: ListAllMyBuckets.
     3. Enter the following ARN: arn:aws:s3:::*
     4. Click Add Statement.
     5. Include the following actions: ListBucket, GetBucketLocation.
     6. Enter the following ARN: arn:aws:s3:::{bucket}. Be sure to replace the word “{bucket}” with the name of the S3 bucket created in step 1.
     7. Click Add Statement.
     8. Include the following actions: GetObject,ListBucket, PutObject
     9. Enter the following ARN: arn:aws:s3:::{bucket}/*. Be sure to replace the word “{bucket}” with the name of the S3 bucket created in step 1.
     10. Click Add Statement.

   • Create a Policy from JSON.

     1. Paste the policy template shown below into the Policy Document field. Be sure to replace the word {bucket} with the name of the S3 bucket created in step 1:
4. Assign Custom Policy to User

AWS IAM Policy Template

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListAllMyBuckets"
      ],
      "Resource": [
        "arn:aws:s3:::*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::{bucket}"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:PutObject",
        "kms:Decrypt"
      ],
      "Resource": [
        "arn:aws:s3:::{bucket}/*"
      ]
    }
  ]
}

Event Data Format

File Contents

The event data will be forwarded as plain text files containing new-line delimited JSON objects. Each line in the file will correspond with a single event batch formatted as a JSON object. Please refer to the mParticle JSON documentation for a detailed description of the data format used for each batch.

For example, if two batches are forwarded at once, the file will contain something similar to the following (note some fields were removed for brevity):

{"events":[{"data":{"event_name":"MyTestEvent","timestamp_unixtime_ms":"1595542763908","event_id":"1234"},"event_type":"custom_event"}],"user_identities":[{"identity_type":"customer_id","identity":"TestCustomerId1"}],"environment":"production"}
{"events":[{"data":{"event_name":"MyOtherTestEvent","timestamp_unixtime_ms":"1595543017730","event_id":"2345"},"event_type":"custom_event"}],"user_identities":[{"identity_type":"customer_id","identity":"TestCustomerId2"}],"environment":"production"}

File and Folder Names

File Names

Each file should be named using the following format:
<App Name>_<UTC timestamp set as message is forwarded>_<Random 5 digit number>.txt.

For example: mPTravel_20171017170911644493_34523.txt.

Folder Names

Optionally, you can store files in a folder within your S3 bucket by providing a Folder Name in mParticle’s Amazon S3 Integration Connection Settings.

Date-Based Folders

You can further organize files into date-based folders, with the name format:
YYYY-MM-DD.

Files can be sorted into folders in several ways according to how you set Store Data in Folders By Date in the mParticle’s Amazon S3 Integration Connection Settings:

• None - files will not be stored in date folders
• Store data in folders by the event time - each file will be sorted into a folder according to the timestamp of the first event in the batch. However, if the batch has no event or event timestamp, the text file will be stored at the base folder, not in a date-labeled folder.
• Store data in folders by upload time - each file will be sorted into a folder according to the upload time of the batch
• Store data in folders by first event time or upload time if no events - each file will be sorted into a folder according to the timestamp of the first event in the batch, or the upload time of the batch if there are no events

Note that files will be sorted into folders depending on when when the events occurred, or were received by mParticle, but the files themselves are named according to the time the file is created. If there is a delay in forwarding, it is possible that the file and folder timestamps will not match. File/folder timestamp mismatches do not indicate an error.

Configuration Settings

Connection Settings