When ingesting historical event data into mParticle with Warehouse Sync, it is important to consider historical data handling, data quality, data retention, and platform limits. Historical data is processed differently by mParticle, so pipelines that ingest it have special considerations and requirements.
\nThis guide outlines key points and best practices for handling historical and large-volume data ingestion.
\nThere are two approaches to ingesting data with Warehouse Sync:
\nWhen ingesting historical data, you can use a combination of these two pipeline types to backfill old data and then switch to an ongoing incremental sync.
\nAs a general rule, any event with a timestamp older than 30 days is considered historical. This data requires special handling to ensure it is processed correctly and made available for long-term use cases like audience segmentation with extended lookback windows.
\nData that you flag as historical is handled differently from real-time data:
\nAll other processing rules for identity resolution and storage remain the same for historical data.
\nA core concept in mParticle is the relationship between user data (attributes and identities) and event data. Understanding this is critical for a successful historical data ingest.
\nData is sent to mParticle in batches (multiple events batched together) to optimize throughput. Each batch contains event data, and provides context about the user in the form of user attributes and identities.
\nproduct_view event captures that a user looked at a product. Event data is captured and stored as events using mParticle’s event data format.membership_tier: gold.User attributes and identities are stored in a persistent User Profile, which creates a complete, 360-degree view of your user.
\nWhen ingesting historical data, you can include user attributes and identities in the same batch as your events. However, in many backfill scenarios, it’s more effective to ingest them separately. This is particularly true when your goal is to have the final user profile reflect the most recent information about a user, rather than the state they were in when a historical event occurred. For example, you can send a batch containing only user_attributes and user_identities to update a user’s profile to ensure that the profile is accurate and up-to-date, then ingest the users’ events when the profile reflects the most recent information.
mParticle enforces service limits specific to Warehouse Sync, including limits on:
\nWarehouse Sync ingests data at a rate according to your account’s configured limits and expected data volumes. When the Warehouse Sync API or UI shows a “success” status, it means your data has been ingested can be used by downstream features and sent to output integrations. Keep in mind that while recent non-historical data may be available quickly, some downstream features and integrations may require additional processing time before all ingested data is available. This is especially true for large or historical data loads that are processed differently from real-time data and connected outputs which have their own processing times.
\nBased on your estimates, coordinate with your mParticle Customer Success team if you anticipate exceeding any limits.
\nWhen ingesting historical data, it’s essential to distinguish between batch timestamps and event timestamps:
\nsource_info.is_historical field.This approach ensures that both event and profile data are processed accurately and in the correct order, while avoiding issues with data retention, forwarding, and audience availability.
\nThere are two common strategies for ingesting historical data efficiently:
\nfrom date – Set the from value to the start of the period you want to ingest, and leave until blank. The initial run will backfill the entire range, after which the pipeline will automatically switch to incremental updates on its schedule.full On Demand pipeline and use a WHERE clause in your data model to limit the dataset (for example, WHERE event_timestamp BETWEEN '2024-01-01' AND '2024-06-30'). Trigger the pipeline whenever you need to ingest another slice, updating the clause between runs as needed. Once you have finished ingesting the historical data, disable the full pipeline and create a new incremental pipeline to sync ongoing new data on a regular schedule, setting the from value to the last date you ingested with the full pipeline.Warehouse Sync deduplicates inbound data using the event batch source_request_id to prevent identical batches from being processed multiple times (for example, retries, replays, or repeated full sync runs).
This applies even when you are ingesting historical data (including batches flagged with source_info.is_historical). If you re-run a backfill and the resulting batch produces the same source_request_id as a previously processed batch within the deduplication retention window (approximately 30–60 days, scoped by the input/feed API key), the replayed batch may be treated as a duplicate and discarded.
source_request_id in your field transformation and ensure it changes for every replay you intend to process as new.timestamp_unixtime_ms for your events and ensure it changes when the event should be reprocessed. When source_request_id is not mapped, timestamp_unixtime_ms can influence the computed source_request_id.WHERE event_timestamp BETWEEN ...) to avoid overlapping slices between backfill runs.For details, see Deduplication and replays.
\nConsent on the profile is not last-write-wins. Incoming consent_state is applied only if its timestamp_unixtime_ms is newer than the timestamp already stored for that regulation and purpose. Older or equal timestamps leave the existing consent in place, even when the pipeline reports success.
Example of a consent backfill that would no-op: the profile already has GDPR purpose marketing with consented: false at 1710000000000. A historical load maps consented: true with timestamp 1700000000000. The profile keeps consented: false.
Use a unique source_request_id for each backfill batch you intend to process. Deduplication happens before consent timestamps are evaluated. For the full timestamp rule and links to existing source_request_id guidance, see Consent timestamp precedence.
This section provides an overview of creating a Warehouse Sync pipeline specifically for ingesting historical data. For a complete, detailed walkthrough of how to create a Warehouse Sync pipeline, refer to the main Warehouse Sync setup guide.
\nThe first step is to connect mParticle to your data warehouse. This process is the same for all Warehouse Sync pipelines.
\nYour data model is the SQL query that defines what data to pull from your warehouse. When ingesting historical data, your query needs special consideration.
\nTo ensure mParticle correctly identifies and processes your historical data, you must explicitly flag it when creating your data model. The “Sync Historical Data” setting in your Warehouse Sync pipeline only controls how far back the pipeline reads from your warehouse; it does not automatically mark old data as historical.
\n\nExample SQL:
\nAdd a CASE statement to your SQL query to create an is_historical column.
SELECT\n *,\n CASE\n WHEN event_timestamp < CURRENT_DATE - INTERVAL '30 days' THEN TRUE\n ELSE FALSE\n END AS is_historical\nFROM your_tableWarehouse Sync provides two complementary ways to control which rows are pulled from your warehouse:
\nWHERE clausesAdd predicates such as WHERE event_timestamp >= DATE '2023-01-01' or a bounded BETWEEN statement inside the query you save with the data model. The filter executes in your warehouse on every run. This is recommended for filtering on fields other than the iterator field, such as event timestamp, event type, or user region, and is useful for validating your pipeline by limiting the dataset to a small subset of data.
from / until)Use the Sync Historical Data setting in the UI or use the from and until fields on sync_mode when calling the API to set the minimum and maximum iterator values a pipeline will ever request. This is recommended for a seamless transition between your initial backfill and ongoing incremental syncs, and makes your pipeline runs more easily auditable via the pipeline status APIs.
A row must satisfy both the iterator window and your SQL to be ingested, giving you granular control over the time span and the business logic applied to your data.
\nWhen using incremental pipelines in Warehouse Sync, you must specify an iterator column—a timestamp field (such as datetime, date, or Unix timestamp) that tracks which rows have already been processed. This iterator column is essential for reliable incremental updates and should be distinct from your event timestamp field whenever possible.
\nBest practices for iterator columns:
\nWHERE clause) and for mapping to the appropriate mParticle field.delay field to 1d to account for this upstream processing time.In the “Create data mapping” step of your Warehouse Sync setup, map the is_historical column from your SQL query to the source_info.is_historical field. You should also set a channel for the data. For more details on this process, see Field Transformations.
Example Field Transformation Mapping:
\n[\n {\n \"mapping_type\": \"column\",\n \"source\": \"is_historical\",\n \"destination\": \"source_info.is_historical\"\n },\n {\n \"mapping_type\": \"static\",\n \"destination\": \"source_info.channel\",\n \"value\": \"server_to_server\"\n }\n]Your sync settings determine whether your pipeline runs on a schedule (incremental) or on-demand (full), and over what time period. These settings work together with the WHERE clause in your data model to control what data is ingested.
from date to the beginning of your historical period. Subsequent runs will automatically pick up where the last run left off.Before activating your pipeline, review all settings to ensure they match your intended historical ingestion strategy. It’s a best practice to start with a small test batch to validate your configuration before running a large backfill.
\nSuccessfully ingesting large volumes of historical data requires thoughtful preparation. In addition to the specific practices embedded in the steps above, keep the following general guidelines in mind:
\nEvaluate connected systems and features:
\nsync_mode can’t be changed after creation.from/until iterator windows to precisely control which rows are ingested and avoid overlapping backfill runs.Your mParticle customer success team can:
\nEngage your Customer Success team early in the planning process for large or unusual data ingests.
\n