{"componentChunkName":"component---src-templates-markdown-template-jsx","path":"/guides/analytics/analytics-deprecated/developer-docs/data-warehouses-lakes/aws-kinesis/","result":{"data":{"pageMetadata":{"title":"AWS Kinesis (Snowplow)","path":"/guides/analytics/analytics-deprecated/developer-docs/data-warehouses-lakes/aws-kinesis/","sourceCode":null,"metadataParent":{"title":"Data Warehouses and Data Lakes","path":"/guides/analytics/analytics-deprecated/developer-docs/data-warehouses-lakes/","sourceCode":null,"hasVersionPicker":null,"metadataChildren":[{"title":"Advanced Data Warehouse Settings","path":"/guides/analytics/analytics-deprecated/developer-docs/data-warehouses-lakes/advanced-data-warehouse-settings/","order":5.1,"label":null},{"title":"AWS Kinesis (Snowplow)","path":"/guides/analytics/analytics-deprecated/developer-docs/data-warehouses-lakes/aws-kinesis/","order":5.1,"label":null},{"title":"AWS Redshift (Define Your Own Schema)","path":"/guides/analytics/analytics-deprecated/developer-docs/data-warehouses-lakes/aws-redshift/","order":5.2,"label":null},{"title":"AWS S3 Integration (Define Your Own Schema)","path":"/guides/analytics/analytics-deprecated/developer-docs/data-warehouses-lakes/aws-s3-define-schema/","order":5.3,"label":null},{"title":"AWS S3 (Snowplow Schema)","path":"/guides/analytics/analytics-deprecated/developer-docs/data-warehouses-lakes/aws-s3-snowplow/","order":5.3,"label":null},{"title":"BigQuery (Snowplow Schema)","path":"/guides/analytics/analytics-deprecated/developer-docs/data-warehouses-lakes/bigquery-snowplow-schema/","order":5.4,"label":null},{"title":"BigQuery Firebase Schema","path":"/guides/analytics/analytics-deprecated/developer-docs/data-warehouses-lakes/bigquery-firebase-schema/","order":5.5,"label":null},{"title":"BigQuery (Define Your Own Schema)","path":"/guides/analytics/analytics-deprecated/developer-docs/data-warehouses-lakes/bigquery-define-schema/","order":5.6,"label":null},{"title":"GCP BigQuery Export","path":"/guides/analytics/analytics-deprecated/developer-docs/data-warehouses-lakes/gcp-bigquery-export/","order":5.7,"label":null},{"title":"Snowflake (Snowplow Schema)","path":"/guides/analytics/analytics-deprecated/developer-docs/data-warehouses-lakes/snowflake-snowplow-schema/","order":5.8,"label":null},{"title":"Snowplow Schema Overview","path":"/guides/analytics/analytics-deprecated/developer-docs/data-warehouses-lakes/snowplow-schema/","order":5.8,"label":null},{"title":"Snowflake (Define Your Own Schema)","path":"/guides/analytics/analytics-deprecated/developer-docs/data-warehouses-lakes/snowflake-define-schema/","order":5.9,"label":null}],"metadataParent":{"title":"Developer Docs","path":"/guides/analytics/analytics-deprecated/developer-docs/","metadataChildren":[{"title":"Introduction","path":"/guides/analytics/analytics-deprecated/developer-docs/introduction/","order":1,"label":null},{"title":"Integrations","path":"/guides/analytics/analytics-deprecated/developer-docs/integrations/","order":4,"label":null},{"title":"Data Warehouses and Data Lakes","path":"/guides/analytics/analytics-deprecated/developer-docs/data-warehouses-lakes/","order":5,"label":null},{"title":"APIs","path":"/guides/analytics/analytics-deprecated/developer-docs/apis/","order":6,"label":null},{"title":"SDKs","path":"/guides/analytics/analytics-deprecated/developer-docs/sdks/","order":6,"label":null},{"title":"Developer Basics","path":"/guides/analytics/analytics-deprecated/developer-docs/developer-basics/","order":7,"label":null}],"metadataParent":{"title":"Analytics (Deprecated)","path":"/guides/analytics/analytics-deprecated/","metadataChildren":[{"title":"Introduction","path":"/guides/analytics/analytics-deprecated/introduction/","order":1.1,"label":null},{"title":"Developer Docs","path":"/guides/analytics/analytics-deprecated/developer-docs/","order":4,"label":null},{"title":"Identity Providers","path":"/guides/analytics/analytics-deprecated/identity-providers/","order":null,"label":null},{"title":"Settings","path":"/guides/analytics/analytics-deprecated/settings/","order":null,"label":null}],"metadataParent":{"title":"Analytics","path":"/guides/analytics/","metadataChildren":[{"title":"Introduction","path":"/guides/analytics/introduction/","order":1,"label":null},{"title":"Core Analytics (Beta)","path":"/guides/analytics/core-analytics/","order":1.1,"label":null},{"title":"Analytics Free Trial","path":"/guides/analytics/analytics-free-trial/","order":2,"label":null},{"title":"Setup","path":"/guides/analytics/setup/","order":3,"label":null},{"title":"Settings","path":"/guides/analytics/settings/","order":4,"label":null},{"title":"Analytics Data Manager","path":"/guides/analytics/data-manager/","order":5,"label":null},{"title":"Analyses","path":"/guides/analytics/analyses/","order":6,"label":null},{"title":"Query Builder","path":"/guides/analytics/query-builder/","order":6,"label":null},{"title":"User Aliasing","path":"/guides/analytics/user-aliasing/","order":7,"label":null},{"title":"Dashboards","path":"/guides/analytics/dashboards/","order":8,"label":null},{"title":"Analytics Resources","path":"/guides/analytics/resources/","order":10,"label":null},{"title":"User Segments","path":"/guides/analytics/user-segments/","order":10,"label":null},{"title":"Tutorials","path":"/guides/analytics/tutorials/","order":11,"label":null},{"title":"APIs","path":"/guides/analytics/apis/","order":12,"label":null},{"title":"Analytics (Deprecated)","path":"/guides/analytics/analytics-deprecated/","order":13,"label":null}],"metadataParent":{"title":"Guides","path":"/guides/","metadataChildren":[{"title":"Platform Overview","path":"/guides/platform-overview/","order":0.5,"label":null},{"title":"Getting Started","path":"/guides/getting-started/","order":1,"label":null},{"title":"mParticle Agent","path":"/guides/mparticle-agent/","order":1.5,"label":null},{"title":"Oversight","path":"/guides/oversight/","order":2,"label":null},{"title":"Data Platform","path":"/guides/data-platform/","order":3,"label":null},{"title":"Data Warehouse","path":"/guides/data-warehouse/","order":4,"label":null},{"title":"Customer 360","path":"/guides/customer-360/","order":5,"label":null},{"title":"Analytics","path":"/guides/analytics/","order":6,"label":null},{"title":"Audiences","path":"/guides/audiences/","order":7,"label":null},{"title":"Settings","path":"/guides/settings/","order":8,"label":null},{"title":"Troubleshooting mParticle","path":"/guides/troubleshooting/","order":10,"label":null},{"title":"Default Service Limits","path":"/guides/default-service-limits/","order":11,"label":null},{"title":"Glossary","path":"/guides/glossary/","order":12,"label":null}]}}}}}},"markdownRemark":{"html":"\n\n

\n \n \n \n \n \n \n \n Prerequisites

\n

The Snowplow Unified Log is stored in an S3 bucket and you are required to write an IAM policy to grant Analytics programmatic access to the respective S3 bucket.

\n

If there are additional enrichments required, such as joining with user property tables or deriving custom user_ids, please contact us.

\n

\n \n \n \n \n \n \n \n Instructions

\n

To connect your real time Snowplow data to Analytics, follow the instructions below:

\n
    \n
  1. In Analytics, click on the gear icon and select Project Settings.\n\"Project
    2. \n
  2. Select the Data Sources tab.\n\"Data
    3. \n
  3. Select New Data Source.\n\"New
    4. \n
  4. Select Snowplow Kinesis.\n\"Select
    5. \n
  5. Click Next. You will need to use this API Key in step 4 of Create the Lambda Function.\n\"API
    6. \n
\n

\n \n \n \n \n \n \n \n Create an IAM Role for the Lambda

\n

Your AWS Lambda needs to have an Execution Role that allows it to use the Kinesis Stream and CloudWatch. (For more information on setting up IAM Roles, please see the official AWS tutorial.)

\n
    \n
  1. Go to IAM Management in the Console and choose Roles from the sidebar.\n\"IAM
    2. \n
  2. Click Create role.\n\"Create
    3. \n
  3. For the type of trusted entity select AWS Service and for the service that will use this role choose Lambda. Click Next: Permissions at the bottom of the screen.\n\"Trusted
    4. \n
  4. Now you need to choose a permission policy for the role. The Lambda needs to have read access to Kinesis and write access to CloudWatch logs - for that we will choose AWSLambdaKinesisExecutionRole. Search for AWSLambdaKinesisExecutionRole in the search and mark the checkbox as shown below.\n\"Permission
    5. \n
  5. Click Next: Review at the bottom of the screen.
    6. \n
  6. On the next screen provide a name for the newly created role under Role Name, then click Create role to finish the process.
    7. \n
\n

\n \n \n \n \n \n \n \n Create the Lambda Function

\n

The Lambda function can be created either directly through AWS Console or through other tools like the AWS CLI. For this integration, the recommended memory setting is 256 MB and because the JVM has to cold start when the function is called for the first time on a new instance, you should set a high timeout value; 90 seconds should be safe.

\n

As with the IAM Role, we will be using the AWS Console to get our Lambda function up and running. Make sure you are in the same region as where your Kinesis streams are defined.

\n
    \n
  1. On the Console navigate to the Lambda section and click Create a function (runtime should be Java 8).\n\"Function
    2. \n
  2. Write a name for your function in Name. In the Role dropdown pick Choose an existing role; then in the dropdown below choose the name of the role you created in the previous step. Click Create function.\n\"Create
    3. \n
  3. \n

    The Lambda has been created, although it does not do anything yet. We need to provide the code and configure the function:

    \n

    a. Take a look at the Function code box. In the Handler textbox paste: com.snowplowanalytics.indicative.LambdaHandler::recordHandler

    \n

    b. From the Code entry type dropdown pick Upload a file from Amazon S3. A textbox labeled S3 Link URL will appear. We are hosting the code through our hosted assets. You will need to choose the S3 bucket in the same region as your AWS Lambda function: for example if your Lambda is us-east-1 region, paste the following URL: s3://snowplow-hosted-assets-us-east-1/relays/indicative/indicative-relay-0.4.0.jar in the textbox. Take a look at this table to pick the right bucket name for your region. Make sure Runtime is Java 8.\n\"Lambda

    \n
    4. \n
  4. Get the API Key from step 4 from the Analytics UI.
    5. \n
  5. \n

    Below Function code settings you will find a section called Environment variables.

    \n

    a. In the first row, first column (the key), type INDICATIVE_API_KEY. In the second column (the value), paste your API Key.\n\"Environment

    \n

    b. The relay lets you configure the following filters:

    \n
      - UNUSED_EVENTS: events that will not be relayed to Analytics;\n  - UNUSED_ATOMIC_FIELDS: fields of the [canonical](https://github.com/snowplow/snowplow/wiki/canonical-event-model) Snowplow event that will not be relayed to Analytics;\n  - UNUSED_CONTEXTS: contexts whose fields will not be relayed to Analytics.
    \n
    6. \n
\n

Out of the box, the relay is configured to use the following defaults:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Unused eventsUnused atomic fieldsUnused contexts
app_heartbeatetl_tstampapplication_context
app_initializedcollector_tstampapplication_error
app_shutdowndvce_created_tstampduplicate
app_warningeventgeolocation_context
create_eventtxn_idinstance_identity_document
emr_job_failedname_trackerjava_context
emr_job_startedv_trackerjobflow_step_status
emr_job_statusv_collectorparent_event
emr_job_succeededv_etlperformance_timing
incidentuser_fingerprinttiming
incident_assigngeo_latitude
incident_notify_of_closegeo_longitude
incident_notify_userip_isp
job_updateip_organization
load_failedip_domain
load_succeededip_netspeed
page_pingpage_urlscheme
s3_notification_eventpage_urlport
send_emailpage_urlquery
send_messagepage_urlfragment
storage_write_failedrefr_urlscheme
stream_write_failedrefr_urlport
task_updaterefr_urlquery
wd_access_logrefr_urlfragment
pp_xoffset_min
pp_xoffset_max
pp_yoffset_min
pp_yoffset_max
br_features_pdf
br_features_flash
br_features_java
br_features_director
br_features_quicktime
br_features_realplayer
br_features_windowsmedia
br_features_gears
br_features_silverlight
br_cookies
br_colordepth
br_viewwidth
br_viewheight
dvce_ismobile
dvce_screenwidth
dvce_screenheight
doc_charset
doc_width
doc_height
tr_currency
mkt_clickid
etl_tags
dvce_sent_tstamp
refr_domain_userid
refr_device_tstamp
derived_tstamp
event_vendor
event_name
event_format
event_version
event_fingerprint
true_tstamp
\n

To change the defaults, you can pass in your own lists of events, atomic fields or contexts to be filtered out. For example:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Environment variable keyEnvironment variable value
UNUSED_EVENTSpage_ping,file_download
UNUSED_ATOMIC_FIELDSname_tracker,event_vendor
UNUSED_CONTEXTSperformance_timing,client_context
\n

Similarly to setting up the API key, the first column (key) needs to be set to the specified environment variable name in ALLCAPS. The second column (value) is your own list as a comma-separated string with no spaces.

\n

If you only specify the environment variable name but do not provide a list of values, then nothing will be filtered out.

\n

If you do not set any of the environment variables, the defaults will be used.

\n
    \n
  1. Scroll down a bit and take a look at the Basic settings box. There you can set memory and timeout limits for the Lambda. As mentioned earlier, we recommend setting 256 MB of memory or higher (on AWS Lambda the CPU performance scales linearly with the amount of memory) and a high timeout time of 1 minute 30 seconds.
    2. \n
\n

\"Basic\nkinesis

\n
    \n
  1. As a final step, add your Snowplow enriched Kinesis stream as an event source for the Lambda function. You can follow the official AWS tutorial if you are using AWS CLI or do it directly from the AWS Console using the following instructions. Scroll to the top of the page and from the list of triggers in the Designer configuration up top, choose Kinesis.\n\"Kinesis
    2. \n
\n

Take a look at the Configure triggers section which just appeared below. Choose your Kinesis stream that contains Snowplow enriched events. Set the batch size to your liking - 100 is a reasonable setting. Note that this is a maximum batch size, the function can be triggered with fewer records. For the starting position we recommend Trim horizon, which starts processing the stream from an observable start (Alternatively, you can select At timestamp to start sending data from a particular date). Click the Add button to finish the trigger configuration. Make sure Enable trigger is selected.

\n

\"Basic

\n
    \n
  1. Save the changes by clicking the Save button in the top-right part of the page.
    2. \n
\n

\n \n \n \n \n \n \n \n Validate Your Data

\n

Go to your Indicative project to check if you are receiving data. You can also go to the debug console to troubleshoot the relay in real time.

","frontmatter":{"title":"AWS Kinesis (Snowplow)","order":5.1,"seoTitle":null,"seoDescription":null},"parent":{"__typename":"File","modifiedTime":"October 12, 2026"}}},"pageContext":{"slug":"/guides/analytics/analytics-deprecated/developer-docs/data-warehouses-lakes/aws-kinesis/"}},"staticQueryHashes":["2950460867"]}