Firehose Java SDK

What is mParticle Firehose?

mParticle aims to allow our customers to send their data wherever they want it to go, and we integrate with many services across the mobile-app ecosystem in order to achieve that. mParticle Firehose is an API that lets anybody become one of our integration partners. We support two basic types of Firehose integration:

  • Leveraging Amazon AWS’s Lambda platform, mParticle can send your “lambda function” data as it comes into our system so that your function can then forward it along to your own API.

  • mParticle can forward data to an HTTP endpoint you provide. This gives you the freedom to set up your integration however you like, leveraging your existing architecture.

The Firehose Java SDK can be used for either integration type.

Java SDK Javadocs

Check out the javadocs here

Including this Library

Java Version

Releases of this library are compiled using JDK 1.8 - any project using this library must also be compiled using JDK 1.8.

Maven/Gradle

mParticle deploys the Java SDK to Maven Central as the com.mparticle:java-sdk artifact:

dependencies {
    compile (
            'com.amazonaws:aws-lambda-java-core:1.1.0',
            'com.amazonaws:aws-lambda-java-events:1.1.0',
            'com.mparticle:java-sdk:1.1.+'
    )
}

Publish to a local Maven repo

Rather than using Maven Central, you can also publish the library to your local Maven server by cloning this repo and invoking the following:

./gradlew publishToMavenLocal

Building

You can clone this project and build the jar manually to include in your project’s classpath:

./gradlew assemble

We also make the latest jar binaries available in the releases section of this repository.

Getting Started

In order to create a Lambda function compatible with mParticle Firehose you’ll be creating a Lambda function just like any other, and then using the Java library in this repository to parse and facilitate communication in between mParticle and your function.

mParticle has created a sample project to get you started quicker, but it may also be useful to read an overview on AWS Lambda here: http://docs.aws.amazon.com/lambda/latest/dg/java-gs.html

Cloning the Sample Project

In order to get started quicker, you can clone the sample project located here:

https://github.com/mParticle/lambda-extension-sample

How the sample project works

The sample project takes care of the basics of creating a lambda function for you. Most importantly it includes a class that implements the RequestStreamHandler interface of the Amazon AWS lambda SDK. When data is received by the RequestStreamHandler, it’s deserialized by the mParticle Java SDK into POJOs to be used by you, as in the following snippet:

/**
 * Sample AWS Lambda Endpoint
 */
public class SampleLambdaEndpoint implements RequestStreamHandler {
    
    MessageSerializer serializer = new MessageSerializer(); //mParticle class for deserialization
    
    @Override
    public void handleRequest(InputStream input, OutputStream output, Context context) throws IOException {
        SampleExtension processor = new SampleExtension(); //you will implement this class in a later step
        Message request = serializer.deserialize(input, Message.class);
        Message response = processor.processMessage(request); //request is processed by you here
        serializer.serialize(output, response);
    }
}

Implementing MessageProcessor

In the example above, incoming data is deserialized to a Message object, and passed in to SampleExtension. SampleExtension is any class that extends the mParticle Java SDK’s MessageProcessor - which you will implement. This will serve as the entry point to all of your logic and ultimately, the 3rd-party service to which you’re integrating.

Message Processor

Overview

The base MessageProcessor implementation is responsible for parsing incoming Message objects. MessageProcessor contains several abstract and empty method implementations responsible for processing Message objects which you will override and implement. Currently the types of messages are:

  1. Module Registration Request and Response
  2. Event Processing Request and Response
  3. Audience Membership Subscription Request and Response
  4. Audience Subscription Request and Response

To get started creating your MessageProcessor, if you’re using the sample project, open and edit SampleExtension.java. Otherwise, create a class that extends com.mparticle.sdk.MessageProcessor.

Module Registration

To process the module registration message, override the processRegistrationRequest method in your MessageProcessor. From this method you must return a ModuleRegistrationResponse describing what your integration can do and what it requires to contact your service, such as an API key.

The details of the registration response are outlined below. For a full example of a response object, see here.

1. Creating your ModuleRegistrationResponse and Describing your Service

Users of the mParticle platform will see the result of your module registration within our website’s user interface. You should set a human-readable title for your service (such as your company’s name), as well as a short description of your company including a link to your company’s website, and it should be less than 230 characters.

Override the processRegistrationRequest method in your MessageProcessor and add the following code, customized for your service:

ModuleRegistrationResponse response = new ModuleRegistrationResponse("My Company Name", "1.0");
response.setDescription("<a href=\"http://www.yourcompany.com\" target=\"_blank\">Company Name</a> A short marketing description of what your company does.\"");

2. Permissions

Device IDs and User IDs

All users and data in the mParticle platform are associated with a set of device IDs such as Apple IDFA, and a set of identities such as Email. Event data and audience data received by your integration will always be associated with a user, but your integration must register for permission to receive each ID.

If you mark a device id or user identity as required, only users and their associated data that contain at least 1 of these IDs will be sent to your integration.

Location and IP Address

All mParticle event data can also be associated with location information, and your Firehose integration requires special permissions to access it. Depending on the app and how it has instrumented the mParticle SDK, location information may originate from the device sensors or may be determine to varying degrees of accuracy using reverse IP lookup. If your service needs access to location, you must request access by setting the following:

permissions.setAllowAccessLocation(true);

Data can be sent into the mParticle platform via one of our native or web SDKs, as well as our Events APIs, and will have an associated IP address of the originating client. If your service needs access to the IP address, you must request access by setting the following:

permissions.setAllowAccessIpAddress(true);

See the example below or the javadocs for more information on populating the Permissions object in your ModuleRegistrationResponse response.

Device Information

mParticle data can contain information about the device being used, including device name, OS Version, and carrier information. By default this information will not be forwarded to your integration. You can request access to device information with the following:

permissions.setAllowDeviceInformation(true);

For a full list of possible device information, see our JSON reference. Note that even if you don’t set this permission, the Device Identites you registered for above will still be forwarded.

User Attributes

mParticle data can include a dictionary of custom user attributes containing information about the user. By default this information will not be forwarded to your integration. You can request access to user attributes with the following:

permissions.setAllowUserAttributes(true);

This permission is also required to receive User Attribute Change events.

Audience User Atributes

mParticle can include user attributes with audience membership change messages as well as event data. You can request access to audience user attributes with the following:

permissions.setAllowAudienceUserAttributeSharing(true);

mParticle allows clients to record consent for data collection from their users. By default, this information will not be forwarded to your integration. You can request access to the user’s consent state by setting

permissions.setAllowConsentState(true);

See our GDPR Consent Management guide to learn more about the consent state object.

3. Supported Runtimes

You need to select which platforms you want your integration to receive data from. mParticle’s Firehose forwarder allows you to receive data from the following platforms:

  • ALEXA
  • ANDROID
  • FIRETV
  • IOS
  • MOBILEWEB
  • ROKU
  • SMARTTV
  • TVOS
  • UNKNOWN
  • XBOX

See the example below or the javadocs for more information on populating the Permissions object in your ModuleRegistrationResponse response.

4. Account Settings

Once we enable your Firehose integration, all mParticle customers will have access to enable it for their account. To allow your integration to correctly identify which customer your data belongs to, and how it should be processed, mParticle enables you to create custom settings for the customer to provide when they set up the account. For example, if your service uses an API key for authentication:

  1. Your integration must include an account setting representing the API key (marking it as required and confidential) in the ModuleRegistrationResponse.
  2. When customers configure your integration via mParticle, they will see the API Key setting in a dialogue, and will need to provide a key before they can enable the integration. medium
  3. Once enabled, mParticle will send data to your integration, including the value of the API key associated with that customer.
  4. Your integration can then use that API key to make authenticated calls to your API or service.

Settings need to be registered individually for Event and Audience-based integrations. Each setting will appear in the mParticle UI (unless marked as not-visible), and must contain the following:

  1. An ID - you will use this in step 4 above to extract the setting value.
  2. A short, human-readable name.
  3. A description of what the setting means in the context of your service. This is used as the tooltip in the mParticle platform.
  4. The data type of the setting
  5. Required flag - if true, mutual customers will be required to enter a value for this setting to configure the integration
  6. Confidential flag - if true, the value entered will be masked in the mParticle UI once set

See the following code snippet, the full example below, or the Javadocs for more on the various types of settings and properties of settings.

List<Setting> eventSettings = new ArrayList<>();
Setting apiKey = new TextSetting("apiKey", "API Key")
    .setIsRequired(true)
    .setIsConfidential(true)
    .setDescription("Your API key issued by <insert company here>.");
eventSettings.add(apiKey);
eventProcessingRegistration.setAccountSettings(eventSettings);

List<Setting> audienceSettings = new ArrayList<>();
audienceSettings.add(apiKey);
audienceProcessingRegistration.setAccountSettings(audienceSettings);

5. Event Registration

If you’re writing an integration that can handle event and analytics data, you’ll specify that your service supports certain types of events in your ModuleRegistrationResponse. This means you will only receive data when those events occur. The possible event types are:

  1. Application State Transition
  2. Custom Event
  3. Error
  4. Privacy Setting Change
  5. Product Action
  6. Promotion Action
  7. Impression
  8. Push Message Receipt
  9. Push Subscription
  10. Screen View
  11. Session Start
  12. Session End
  13. User Attribute Change
  14. User Identity Change
  15. Attribution

Note that to receive User Attribute Change events, you must also register the Allow User Attributes permission. Similarly, for User Identity Change events, you will only receive events for the identities you have set permissions to receive.

See here for brief descriptions of each event type.

System Notification Types

System Notifications are used to communicate changes to a user. Currently The only available System Notifcation is ‘GDPR Consent State’. Note the difference between this notification and the allowConsentState permission. Consent State always gives the current consent state for the user at the time of an event. The System Notification for GDPR Consent State describes changes to the consent state. Each System Notification contains the full current and old consent states.

See here for more on the GDPR Consent State system notification.

6. Push Messaging Provider ID

If you have registered for Push Message Receipt events, you must provide the Provider ID key used in the payload of your platform’s Push messages. This ensures that you only receive events related to Push messages from your platform. Work with mParticle to ensure the you are providing the correct Provider ID.

eventProcessingRegistration.setPushMessagingProviderId("your-push-messaging-provider-id");

7. Audience Registration

Registering for Audience will allow your integration to receive Audience Subscription and Membership messages. During registration your integration needs to specify the settings required to map and send audience data from mParticle to your service:

  • (required) user modifiable account settings that your integration requires, such as an API key
  • (optional) user modifiable audience-specific settings such as an audience ID
  • (optional) internal audience-specific settings. These should be marked as not visible, so that users of the mParticle console cannot modify them. You can modify the values of these settings in your response to Audience Membership messages.

See here for more info on AudienceProcessingRegistration

Sample Registration

Refer to the sample project or the simple example below, which shows how to subscribe for both Event Processing and Audience:

 @Override
    public ModuleRegistrationResponse processRegistrationRequest(ModuleRegistrationRequest request) {
        ModuleRegistrationResponse response = new ModuleRegistrationResponse("My Company Name", "1.0");
        response.setDescription("A short marketing description of what your company does with a <a href=\"http://www.yourcompany.com\" target=\"_blank\">link to your website</a>.");

        //Set the permissions - the device and user identities that this service can have access to
        Permissions permissions = new Permissions();
        permissions.setUserIdentities(
                Arrays.asList(
                        new UserIdentityPermission(UserIdentity.Type.EMAIL, Identity.Encoding.RAW),
                        new UserIdentityPermission(UserIdentity.Type.CUSTOMER, Identity.Encoding.RAW)
                )
        );
        response.setPermissions(permissions);

        //the extension needs to define the settings it needs in order to connect to its respective service(s).
        //you can using different settings for Event Processing vs. Audience Processing, but in this case
        //we'll just use the same object, specifying that only an API key is required for each.
        List<Setting> processorSettings = Arrays.asList(
               new TextSetting("apiKey", "API Key")
                  .setIsRequired(true)
                  .setIsConfidential(true)
                  .setDescription("Your API key issued by <insert company here>.");
        );

        //specify the supported event types. you should override the parent MessageProcessor methods
        //that correlate to each of these event types.
        List<Event.Type> supportedEventTypes = Arrays.asList(
                Event.Type.CUSTOM_EVENT,
                Event.Type.USER_ATTRIBUTE_CHANGE,
                Event.Type.USER_IDENTITY_CHANGE);

        //this extension only supports event data coming from Android and iOS devices
        List<RuntimeEnvironment.Type> environments = Arrays.asList(
                RuntimeEnvironment.Type.ANDROID,
                RuntimeEnvironment.Type.IOS);

        //finally use all of the above to assemble the EventProcessingRegistration object and set it in the response
        EventProcessingRegistration eventProcessingRegistration = new EventProcessingRegistration()
                .setSupportedRuntimeEnvironments(environments)
                .setAccountSettings(processorSettings)
                .setSupportedEventTypes(supportedEventTypes);
        response.setEventProcessingRegistration(eventProcessingRegistration);

        //Segmentation/Audience registration and processing is treated separately from Event processing
        //Audience integrations are configured separately in the mParticle UI
        //Customers can configure a different set of account-level settings (such as API key here), and
        //Segment-level settings (Mailing List ID here).
        List<Setting> subscriptionSettings = new LinkedList<>();
        subscriptionSettings.add(new IntegerSetting(SETTING_MAILING_LIST_ID, "Mailing List ID"));
        Setting invisibleSetting = new IntegerSetting(SETTING_INTERNAL_LIST_ID, "Internal ID");
        invisibleSetting.setIsVisible(false);
        subscriptionSettings.add(invisibleSetting);

        AudienceProcessingRegistration audienceRegistration = new AudienceProcessingRegistration()
                .setAccountSettings(processorSettings)
                .setAudienceSubscriptionSettings(subscriptionSettings);
        response.setAudienceProcessingRegistration(audienceRegistration);
        return response;
    }

Verifying

Once you have completed the implementation of your registration, you must send the result to mParticle for verification. See testing for more information.

Event Processing

mParticle customers send batches of events to the mParticle platform via the mParticle SDKs (iOS, Android, Javascript, etc), direct Events data or feeds. mParticle can forward these event batches to your integration. The base MessageProcessor implementation is designed to handle these incoming batches of events, and appropriately iterate over each different event, providing you with convenient hooks in which to handle each distinct event type.

In order to process these events, you’ll override (in your MessageProcessor subclass) the appropriate method given the event types that you registered for during module registration.

Event Object

Each event hook method is given an object that contains the event itself as well as information about the user, device, and application that fired the event.

See here for more on the Event object from which all other events inherit.

Event Methods

The following table shows the correlation between each type and overridable method:

Event Type Method
APPLICATION_STATE_TRANSITION MessageProcessor#processApplicationStateTransitionEvent
ATTRIBUTION MessageProcessor#processAttributionEvent
CUSTOM_EVENT MessageProcessor#processCustomEvent
ERROR MessageProcessor#processErrorEvent
IMPRESSION MessageProcessor#processImpressionEvent
PRIVACY_SETTING_CHANGE MessageProcessor#processPrivacySettingChangeEvent
PRODUCT_ACTION MessageProcessor#processProductActionEvent
PROMOTION_ACTION MessageProcessor#processPromotionActionEvent
PUSH_MESSAGE_RECEIPT MessageProcessor#processPushMessageReceiptEvent
PUSH_SUBSCRIPTION MessageProcessor#processPushSubscriptionEvent
SCREEN_VIEW MessageProcessor#processScreenViewEvent
SESSION_END MessageProcessor#processSessionEndEvent
SESSION_START MessageProcessor#processSessionStartEvent
USER_ATTRIBUTE_CHANGE MessageProcessor#processUserAttributeChangeEvent
USER_IDENTITY_CHANGE MessageProcessor#processUserIdentityChangeEvent

Verification Requests

To ensure a continuous connection with your integration, mParticle will periodically send event processing requests, even if there are no new events to send. Your integration must be able to accept an empty events array and return a a standard event processing response.

Audience Processing

Audience Subscription Event

The audience subscription event tells your integration when a new audience has been created, removed, and/or updated in the mParticle platform. The response allows you to set additional subscription settings that will be stored together with the subscription and sent back with audience membership changes. All subscription settings must be declared during module registration.

See here for more info on the AudienceSubscriptionRequest object that you’ll be given to parse the event.

Audience Membership Change Event

The audience membership change event tells your integration whenever users are added or removed from a given audience. If you have set permissions.setAllowAudienceUserAttributeSharing(true), each audience will also include any user attributes the customer has shared.

See here for more info on the AudienceMembershipChangeRequest object that you’ll be given to parse the event.

Verification Requests

To ensure a continuous connection with your integration, mParticle will periodically send membership change requests, even if there are no new events to send. Your integration must be able to accept an empty events array and return a a standard audience membership change response.

Testing

Registration Testing

Prior to deployment, please send mParticle the JSON result of your registration response. There are several ways to easily acquire the JSON:

Unit tests in the sample project

The latest Firehose Sample integration includes a unit test which will output the result of your registration function. See the Sample Firehose Integration README for more information. This is the quickest way to verify the contents of your module registration.

Manually send in a registration request via the AWS console

If you’re unable to run the unit tests, you may also deploy your function to AWS and then use this sample JSON (also available here) as a test event within the AWS console:

{
   "type":"module_registration_request",
   "id":"76e558fa-f456-437c-bb81-e658f75a7bd9",
   "timestamp_ms":1446250094767
}

Additional Unit Testing

In order to ensure that your lambda function works correctly, we recommend unit testing prior to deployment. Unit testing is quicker and easier than manually sending in JSON to your lambda function via AWS.

Integration Testing

You can send JSON messages manually into your lambda function via the AWS-lambda console.

See here for sample JSON that you can use.

Deployment

Once you’ve completed development and feel you’ve sufficiently tested your integration, the next step is to register and add it to the mParticle platform. This is a multistep process:

Step 1 (AWS Lambda) - Grant Permissions

You will need to grant mParticle permissions to call your lambda function. Using the Amazon CLI tool, execute the commands below, altering them for your environment:

  • change the --function-name argument (arn:aws:lambda:us-east-1:123456789:function:myLambdaFunction:production) to the full ARN of your lambda function. This can be found by navigating to your AWS Lambda console and selecting your lambda function.
  • you should be using an alias for your lambda function to allow for additional development and testing while your integration is live. In the commands below, be sure to append your production alias name to your lambda-function’s ARN. The examples below use an alias named “production”. See here for more information on Aliases.
  • statement-id must be unique - if you receive an error stating that the provided statement-id already exists, increment the statement-id(s) to a higher value.
aws lambda add-permission \
--region us-east-1 \
--function-name arn:aws:lambda:us-east-1:123456789:function:myLambdaFunction:production \
--statement-id 1 \
--principal 338661164609 \
--action lambda:InvokeFunction
aws lambda add-permission \
--region us-east-1 \
--function-name arn:aws:lambda:us-east-1:123456789:function:myLambdaFunction:production \
--statement-id 2 \
--principal 457804467337 \
--action lambda:InvokeFunction

Step 1 (HTTP) - Expose endpoint

For HTTP integrations, your endpoint must be able to receive HTTP requests.

Once permissions have been granted, mParticle will use an internal tool to send a registration request to your integration. We will work with you to ensure that your registration response looks correct and that it includes what’s necessary for your service.

mParticle requires a high-resolution logo to represent your service in our dashboard. The logo must:

  • be provided in SVG format
  • have a transparent background
  • be easily readable when set on top of a white or light-colored page

Provide mParticle with credentials for the required fields from your registration response for testing configuration.

Step 3 - Final testing and release

Once your integration has been registered and added to the mParticle platform, we will work with you to perform final testing prior to making your service available to all mParticle customers.

Data Rate and Volume

mParticle servers will stream data to your function at the same rate at which it is received. By default, AWS lambda allows for 100 concurrent executions of your function. If your lambda function reaches this limit, mParticle will continually retry requests with a back-off until they have succeeded. If you are expecting a large customer integration, it is recommended that you request a higher limit by opening a case with AWS Support.

See here for more information on AWS rate limiting and how to increase your limit.

Keep this in mind as you’re developing your lambda function as it may determine how it should process and to which of your APIs you should send data. Data rate and volume will depend on:

  1. The size of the customer that has enabled your integration.
  2. The amount of/types of data for which you register.
  3. If you’ve registered to receive Audience data, the volatility and size of a given customer’s audiences.

For HTTP Integrations, you will need to have infrastructure in place to support a high volume of data.

Was this page helpful?