Observability Troubleshooting Examples

Below are three example scenarios for when Observability help you troubleshoot common issues encountered when ingesting and forwarding data.

• An event is blocked by a forwarding rule
• An event is blocked by a feed filter
• An event is blocked by a data plan

The following scenarios demonstrate the use of tracing to troubleshoot development environment data. To learn more about the differences between tracing your development data and your production data throughout the process of setting up your mParticle implementation, see Tracing best practices.

An event is blocked by a forwarding rule

Forwarding rules allow you to prevent events from being forwarded to an output according to criteria like the event’s attributes, user consent preferences, or whether the user who triggered the event was logged in.

Sometimes, a forwarding rule might block an event that you actually want to send to your output. Using Observability, you can more easily identify and troubleshoot these instances.

For example, consider a scenario where we send the following event to mParticle when a user subscribes to a premium service, but we don’t see the event appear downstream in our output.

Example event batch

'events': [
    {
        'data': {
            'custom_event_type': 'other',
            'event_name': 'subscription',
            'custom_attributes': {
              'plan_type': 'premium'
            }
        },
        'event_type': 'custom_event'
    }
]

1. First, let’s check to see if Observability flagged any recent traces by logging into mParticle and navigating to Observability > Trace Activity.
2. We click Sort & Filters and select our input feed under Inputs. Under Results we select “Warning”, “Insight”, and “Needs Attention” and click Apply.

3. We see a trace matching our input and output with “Insight” as the result, which indicates some detected behavior (either intended or not) that we should investigate. We click on the Trace ID to open the details page to learn more.

4. In the Trace Details page, we scroll through the Timeline View until we see a highlighted span under the section called Transformations. This includes the error message: “Event Batch Filtering: The batch was dropped because all events contained within it were filtered out by event attribute forwarding rules. If this is not expected, check the event attribute rules for this connection.” Now we know our event was dropped due to a forwarding rule.

5. We navigate to the connection between our custom feed input and our webhook output to view our forwarding rules.
6. We go to the Forwarding tab and see that we have a Forwarding Rule set to “Only Forward” events if they contain the event attribute plan_type set to basic.

7. After re-examining our original event batch, our event’s plan_type was set to premium, which explains why it wasn’t forwarded to our output.
8. We can now either remove our feed filter or adjust it to forward subscription events with either basic or premium plan types.

An event is blocked by a feed filter

Filters provide granular control over exactly which data points are forwarded to a specific output. You can switch OFF certain data points to exclude them from the data you forward, or you can switch ON data points to forward them.

Sometimes, you might mistakenly toggle a filter for a data point that you actually want to send to an output.

For example, consider a scenario where we send the following event to mParticle when a user subscribes to a premium service, but we don’t see the plan_type event attribute appear downstream in our output.

Example event batch

'events': [
    {
        'data': {
            'custom_event_type': 'other',
            'event_name': 'subscription',
            'custom_attributes': {
              'plan_type': 'premium'
            }
        },
        'event_type': 'custom_event'
    }
]

1. First, let’s check to see if Observability flagged any recent traces by logging into mParticle and navigating to Observability > Trace Activity.
2. We click Sort & Filters and select our input feed under Inputs. Under Results we select “Warning”, “Insight”, and “Needs Attention” and click Apply.

3. We see a trace matching our input and output with “Insight” as the result, which indicates some detected behavior (either intended or not) that we should investigate. We click on the Trace ID to open the details page to learn more.

4. In the Trace Details page, we scroll through the trace timeline until we see a highlighted span under Transformations. If we click on this, we see the error message: “Event Batch Filtering: The batch was dropped because all events contained within it were filtered out by event attribute forwarding rules. If this is not expected, check the event attribute rules for this connection.” Now we know our event was dropped due to a feed filter. Let’s check our filter settings to see if we’re unintentionally removing any event attributes.

5. To find our feed filter settings, we navigate to Connections > Feed Filters using the left hand navigation, and then we select the Events tab. The column for Webhook (our output that didn’t receive the event) lists each event type that will be forwarded (or filtered).
6. If we expand the Other event type section, we see our subscription event. If we expand this, we see that the plan_type event attribute is switched OFF. As long as this filter is enabled, mParticle won’t forward the plan_type attribute with subscription events.

7. Now we know we need to remove the feed filter so the plan_type attribute is always forwarded.

An event is blocked by a data plan

Data plans allow you to describe the extent and the shape of the data you collect using mParticle. Each data plan contains a list of data points which describe the event names, event attributes, user attributes, and user identities that you want mParticle to ingest. If your input sends data that doesn’t match your data plan, it can be either marked as invalid or blocked entirely, depending on your data plan settings.

Sometimes, data plans will block data that you actually want to ingest. We can use Observability to help identify and troubleshoot these instances.

For example, consider a scenario where we send the following event to mParticle when a user subscribes to a service, but we don’t see the language_preference user attribute appear downstream in our output.

Example event batch

{
  'user_attributes' : {
    'language_preference': 'EN'
  },
  'events': [
    {
      'data': {
        'custom_event_type': 'other',
        'event_name': 'subscription',
      },
      'event_type': 'custom_event'
    }
  ],
  'context': {
    'data_plan': {
      'plan_id': 'higgs_data_plan',
      'plan_version': 1
    }
  }
}

1. First, let’s check to see if Observability flagged any recent traces by logging into mParticle and navigating to Observability > Trace Activity.
2. We click Sort & Filters and select our input feed under Inputs. Under Results we select “Warning”, “Insight”, and “Needs Attention” and click Apply.

3. We see a trace matching our input and output with “Insight” as the result, which indicates some detected behavior (either intended or not) that we should investigate. We click on the Trace ID to open the details page to learn more.

4. In the Trace Details page, we scroll through the trace timeline until we see the highlighted span called “Evaluating Data Plan ‘higgs_data_plan’” under Transformations. If we click on this, we see the error message: “Data Planning: All user attributes in the batch were dropped by the data plan. Not all integrations and use cases support batches with no user attributes.” Now we know our user attribute was blocked by the Higgs Data Plan.

5. The Setup Details section on the left side of the trace details page gives us a direct link we can click to take us to our Higgs Data Plan.
6. Once on the Higgs Data Plan settings page, we click Block Volume to view the data plan report.

7. From the Report tab, make sure to select “Blocked”. Under the list of blocked data points, we see our language preference user attribute.

8. At this point, we can modify our data plan’s block settings to allow unplanned user attributes, or we can add the language preference attribute to our list of expected data points. Note: if you have a quarantine connection configured for your data plan, you can backfill any blocked data.