Rules

The powerful Rules feature allows you to cleanse, enrich and transform your incoming data before it is forwarded to downstream services. A Rule is a JavaScript function which takes an incoming batch object and modifies it according to user-defined criteria. Any rule will cover one or more of five basic use cases:

  • Modify a batch’s data
  • Drop a batch
  • Modify an event’s data
  • Drop an event from the events array
  • Add events to the events array

Rules can be applied to an input in two ways:

  • All Outputs Rules are applied first and affect the data forwarded to all Outputs. For example, if you forward iOS data to Mixpanel and Google Analytics and you apply an All Outputs Rule that drops all batches from outside the United States, neither Mixpanel or Google Analytics will receive the dropped batches.

  • Specific Output Rules are applied only to data passed to selected Outputs. For example, if you forward iOS data to Mixpanel and Google Analytics, and you apply a Specific Outputs rule to the Google Analytics output that drops all batches from outside the United States, Mixpanel will receive events from outside the United States, but Google Analytics will not.

These two rule types operate almost identically. Both take in an mParticle batch object and return either null or a modified batch object in the same format. There are some differences in error handling and available fields. See the Developer documentation for more info.

All Outputs Rules are applied to data as soon as it arrives in mParticle. For example, if a batch received from an Input has 100 events and a rule is applied which drops half of them, the Activity Overview will show 50 events. The original input feed details are not available for reporting. A 200ms timeout applies to all rules. If a result is not returned within the timeout, the entire batch is dropped.

Create a Function in AWS

mParticle Rules are hosted in your AWS account as Lambda functions. To do this, you need to be able to provide an Amazon Resource Number (ARN) for your rule. See the AWS Lambda documentation for help creating a function.

  • The name of the function must begin with ‘mpr’
  • Your Development rule must have an alias of ‘$LATEST’
  • Your Production rule must have an alias of ‘PROD’

Your ARNs should look something like this:

arn:aws:lambda:us-east-1:999999999999:function:mprmylambdafunction:PROD

arn:aws:lambda:us-east-1:999999999999:function:mprmylambdafunction:$LATEST

IAM User

To connect to your AWS Lambda function, you must provide the AWS Access Key ID and Secret Access Key for an IAM User.

In the IAM Dashboard, add the following permissions policy for the User:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "mpRulesLogs",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:FilterLogEvents",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:us-east-1:*:log-group:/aws/lambda/mpr*"
            ]
        },
        {
            "Sid": "mpRulesMetrics",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:GetMetricStatistics"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "mpRulesLambda",
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeFunction",
                "lambda:GetAlias"
            ],
            "Resource": [
                "arn:aws:lambda:us-east-1:*:function:mpr*"
            ]
        }
    ]
}

IAM Role

You will also need to create a Role in IAM. Assign this role the same policy document created above.

Assign this role to each Lambda function you plan to deploy as an mParticle Rule.

Creating a Rule in the Dashboard

  1. Create a rule by navigating to Setup > Rules
  2. Click New Rule.

Rules

Enter your Development and Production ARNs and click Test.

Rules

Error Handling

When you first test a rule, you must select a Failure Action. This determines what will happen if your rule throws an unhandled exception. There is no default action, you must select one of the following:

  • If you choose Discard, an unhandled exception will cause your rule return null, effectively dropping the batch.
  • If you choose Proceed, an unhandled exception will cause your rule to return the unaltered batch object, proceeding as if the rule had not been applied.

Regardless of which option you choose, it’s best practice to handle all exceptions in your code, rather than falling back on the above defaults. This is especially true if your rule deals with events, where an unhandled exception from just one event could lead to all events in the batch being dropped.

Javascript Syntax

exports.handler=(batch,context,callback)=>{
    //do something with batch
    callback(null, batch)
}

Your code must be a valid Lambda function.

  • batch is the complete incoming batch object.
  • context is a required argument for Lambda functions, but is effectively null for mParticle rules.

Testing Rules

The first time you test a rule, you will be asked to provide a name, description and failure action. After naming a rule, you can test it by using one of the sample templates provided in the Test Rule dialog. You can also copy and paste batch JSON from your Live Stream. If you do this, be sure to pick a full batch to copy, not a single event. Click Test to run. Optionally, check a box to save your JSON template in local storage for future testing.

You must enter valid batch JSON in the code editor.

If there are any syntactical errors in your code, warning or error icons will display next to the line number with details of the problem so you can correct.

After clicking Test, you can examine the JSON output from your function to see that the input has been modified as expected.

After a successful test you can click Save to save the rule.

If your test fails, try examining the logs for any console output.

Versioning

When you first create a rule, by default it will only be applied to DEV data. As well as testing a rule with sample JSON you should test the rule in your dev environment to make sure data reaching your output services is as expected. When you are ready to apply a rule to your production data, click Promote to Prod on the rule page. This will create a ‘v1’ production rule.

Note that before a rule can be promoted to Prod, you must remove all console.log() statements.

If you need to make changes, choose $LATEST from the Version dropdown. All other versions are read only. Test your changes with your dev environment and, when you are ready, click Promote to Prod to create ‘v2’ of your production rule.

Note that you can have a maximum of 50 versions per rule. If you have too many versions, select a version and click the trash can icon to the right of the version number to delete it.

Status

Each rule has a master switch in the Settings panel. If there is a problem with your rule, you can switch it off and it will be disabled for all connections until you enable it again. To disable, click Edit in the right sidebar and set the Status slider to inactive.

Metrics

The following metrics are available:

  • Invocations - how many times the rule was invoked
  • Throttles - how many times a 429 throttling response was returned when calling the rule
  • Errors - how many errors have occurred when calling the rule

These metrics are for the last 24 hours and apply to all connections. Summaries for each rule can be seen on the main rules page. Detailed graph of the previous 24 hours is available on the Monitoring tab of the individual rule page.

Logs

To help you with troubleshooting rules, mParticle maintains logs for each rule where you can view all console output. From an individual rule page, select the Logs tab. You can filter messages by date range or search for keywords.

Deleting Rules

From the Rules listing, select the Delete action to delete the Rule. If the rule is applied to any connections, it will be removed.

Was this page helpful?