Overview
Routing enables publishers to export all of their first-party event data from web and mobile applications directly into data warehouses like Google BigQuery, Snowflake, or Amazon S3. Publishers can route Permutive data into their warehouses to build a single view of their audience, create custom reporting and analytics, and integrate with other data sources using Permutive’s Identity Graph.Why Use Routing?
Build a single view of your audience — Routing allows publishers to consolidate all their first-party event data from Permutive into their own data warehouse, where they can link it with other data sources using Permutive’s Identity Graph. Create custom reporting and analytics — With full access to raw event data in your warehouse, you can build custom reports, dashboards, and analytics tailored to your specific business needs. Run complex queries, generate insights, and integrate with your existing BI tools without limitations.Concepts
Definitions
- Events: Actions performed by users on your website or app that are tracked by the Permutive SDK. Events include standard events like Pageview, SlotClicked, and custom events you define. Each event captures contextual properties about the user’s action, session, and identity.
- Routing Integration: A configuration that specifies the destination for event data. Each routing integration defines where events should be sent, authentication credentials, and data format preferences. BigQuery integrations can be configured via the Dashboard self-service; Snowflake and S3 integrations require assistance from Permutive support.
- Streaming Routing: Event data is streamed to the destination in near real-time as events are collected. Available for BigQuery, Snowflake, and S3 Streaming destinations.
- Batch Routing: Event data is exported on a scheduled basis (typically 24-hour cycles). Available for S3 destinations. Batch routing operates at the Organization level rather than individual workspaces.
- Schema Generation: Routing automatically generates and updates table schemas based on the events and properties being tracked. When new event types or properties are added, the schema is automatically updated without manual intervention.
- Data Partitioning: Tables are organized by date to optimize query performance and data management. Partitioning strategy varies by destination—see How is data partitioned in my destination? for details on each platform.
Data Types
Routing supports exporting different types of data to your destination. The availability of each data type depends on the destination:| Data Type | Description | BigQuery | Snowflake | S3 Streaming | S3 Batch |
|---|---|---|---|---|---|
events | User events tracked by the Permutive SDK (Pageview, custom events, etc.) | Yes | Yes | Yes | Yes |
aliases | Identity data linking user identifiers (sync_aliases) | Yes | Yes | Yes | Yes |
domains | Domain-level metadata and configuration | No | No | No | Yes |
segment_metadata | Segment definitions and metadata | Yes | Yes | Yes | Yes |
Workflows
Setting Up a Routing Integration
For BigQuery: Publishers navigate to the Routing section in the Permutive Dashboard and click “Add Integration” to begin. They select BigQuery as the destination and provide configuration details such as project ID, dataset name, and region. Permutive generates a unique service account that the publisher must grant access to in their GCP project. For Snowflake and S3: Contact Technical Services to set up routing integrations. The support team will work with you to configure the destination and provide the necessary authentication details.Monitoring Routing Status
Once configured, routing integrations display their current status in the Dashboard. Status values include “Running” (active and operational), “Created” or “Updating” (in progress), and “Failed” (setup error occurred). Publishers can monitor the health of their routing integrations and troubleshoot issues using the status information and error messages provided.Querying Routed Data
After routing is enabled, event data begins flowing to the destination. Publishers can query their routed data using standard SQL tools for BigQuery and Snowflake, or process files from S3 using their preferred data processing framework.Troubleshooting
Integration fails during setup with permission errors
Integration fails during setup with permission errors
This error typically occurs when the service account does not have the correct permissions in the destination platform.Solution:For BigQuery: Ensure you have granted the Permutive service account the BigQuery User role at the PROJECT level (not just the dataset level). If you are manually creating the dataset, you must grant BigQuery Data Owner role instead.For Snowflake: Verify that key-pair authentication is configured correctly and that the Permutive user has appropriate permissions on the target database and schema.For S3: Confirm that the AWS IAM user has been granted write permissions to the specified bucket and path prefix.Review the permissions in your cloud platform’s IAM console and grant the required roles to the Permutive service account or user. After granting permissions, click “Confirm account access granted” in the Permutive Dashboard.
Integration status shows 'Failed' after setup
Integration status shows 'Failed' after setup
Integration failures can occur due to:
- Invalid project ID, dataset name, or bucket name
- Permission issues (see above)
- IAM policy restrictions (e.g., domain restrictions in GCP)
- Network or firewall rules blocking access
Events not appearing in destination or significant delay
Events not appearing in destination or significant delay
If events are not appearing or are delayed, first verify:
- The integration status is “Running” in the Permutive Dashboard
- Events are being tracked successfully - check Event Inspector in the Dashboard
- The Permutive SDK is properly deployed on your site/app
{event_name}_events exist (e.g., Pageview_events). Verify you can query recent data.For Snowflake: Events are batched before delivery. Check the EVENTS table for recent records. Note that there may be reporting lag due to Snowflake COPY_HISTORY refresh.For S3 Streaming: Check bucket permissions and verify files are being written to the expected path.For S3 Batch: Exports run on 24-hour cycles. Verify the last successful export completed.Solution: If the integration shows “Running” but data is not appearing, contact Technical Services with your integration details and the time range of missing data.Schema changes not being applied automatically
Schema changes not being applied automatically
Routing automatically updates schemas when new event types or properties are added. If schema changes are not appearing:
- There may be a delay in schema propagation (typically a few minutes)
- The event or property may not be firing correctly
- For customer-created datasets in BigQuery, permissions may prevent schema updates
Modified routing tables and data is no longer appearing
Modified routing tables and data is no longer appearing
Permutive generates and manages routing table schemas automatically. Do not modify the tables created by routing, as this can break the routing pipeline.Solution: If tables have been modified, contact Permutive support at [email protected] to restore the correct schema and resume routing. In the future, avoid making any changes to routing-created tables. If you need custom table structures, create views or separate tables based on the routing data instead of modifying the source tables.
IAM policy error: 'Users do not belong to a permitted customer'
IAM policy error: 'Users do not belong to a permitted customer'
This error occurs when your GCP organization has domain restriction policies that prevent external service accounts from being granted access.Solution: Review your organization’s IAM domain restriction policies. You may need to add Permutive’s domain to the allowlist or adjust the policy to permit the specific service account. Contact your GCP organization administrator for assistance.
Environment Compatibility
Core Product
Routing supports event collection from all Permutive SDK platforms:| Functionality | Web | iOS | Android | CTV | API Direct |
|---|---|---|---|---|---|
| Event routing | Yes | Yes | Yes | Yes | Yes |
| Identity routing | Yes | Yes | Yes | Yes | Yes |
| Segment routing | Yes | Yes | Yes | Yes | Yes |
| Schema auto-update | Yes | Yes | Yes | Yes | Yes |
Destinations
Routing supports the following destination platforms: Streaming Routing:- Google BigQuery
- Snowflake (via Snowpipe)
- Amazon S3 Streaming (GZIP compressed JSONL files)
- Amazon S3 Batch (scheduled exports, JSON or Parquet format)
Guides
Step-by-step instructions for working with Routing.Setting Up BigQuery Routing
Configure near real-time event streaming to Google BigQuery
Setting Up Snowflake Routing
Set up automated data loading to Snowflake using Snowpipe
Setting Up S3 Streaming Routing
Stream events to Amazon S3 in near real-time
Setting Up S3 Batch Routing
Configure scheduled daily exports to S3 in JSON or Parquet format
Dependencies
Routing requires the following products and infrastructure:| Dependency | Required | Description |
|---|---|---|
| Permutive SDK | ✓ | The Permutive SDK (Web, iOS, Android, CTV, or API Direct) must be deployed to track events that will be routed to the destination. |
| Cloud Platform Account | ✓ | A Google Cloud Platform project (for BigQuery), Snowflake account, or AWS account (for S3) is required to receive routed data. |
| Event Tracking | ✓ | Events must be properly configured and firing. |
Limits
Routing adheres to the following product specifications and limits.Feature Limits
| Feature | Description | Limit |
|---|---|---|
| Routing integrations | The number of active routing integrations. | 1 per destination type |
| Historical data | Routing is point-in-time; no historical backfill is available when enabling routing. | No backfill |
| Table modification | Do not modify routing-created tables, as this can break the pipeline. | No manual modifications are supported |
Performance Limits
| Metric | Description | Expected Latency |
|---|---|---|
| BigQuery latency | Expected time for events to appear in BigQuery after being tracked. | ~5 minutes |
| Snowflake latency | Expected time for events to appear in Snowflake after being tracked. | ~5 minutes |
| S3 Streaming latency | Expected time for events to appear in S3 Streaming after being tracked. | ~5 minutes |
| S3 Batch frequency | Frequency of scheduled batch exports to S3. | 24-hour cycles |
FAQ
What events are routed to my destination?
What events are routed to my destination?
Only User Events (events managed via the Events UI) are routed to your destination. These are the custom events you configure and include events like Pageview, SlotClicked, FormSubmission, LinkClick, VideoView, and any custom events you create.
Will enabling Routing impact other Permutive features?
Will enabling Routing impact other Permutive features?
No, Routing is a completely separate system and does not impact any other Permutive features. Enabling or disabling Routing does not affect cohort creation, activation, reporting, or any other functionality. Routing simply creates a copy of your event data in your own data warehouse.
Do Permutive employees have access to my routed data?
Do Permutive employees have access to my routed data?
No, Permutive employees do NOT have access to your routed data. The data is written to your own cloud infrastructure (your BigQuery project, Snowflake account, or S3 bucket), and Permutive employees would need explicit permission from you to access it. You have full control over who can access your routed data.
What data is included in routed exports?
What data is included in routed exports?
Routed data includes:
- All User Events with full properties
- User identities and aliases
- Segment IDs that users belong to
- Cohort memberships
- Domain information
- Segment metadata
Can I get historical data when I enable Routing?
Can I get historical data when I enable Routing?
No, Routing is point-in-time only. When you enable Routing, only new events going forward will be routed to your destination. There is no historical backfill of past data. If you need historical data, contact Support to discuss options.
How much will Routing cost?
How much will Routing cost?
Cloud platforms charge for data ingestion and storage. Costs depend on your event volume and the destination you choose:
- BigQuery: Google Cloud charges for streaming inserts and storage separately
- Snowflake: Snowflake charges for compute (data loading) and storage
- S3: AWS charges for storage and data transfer
Can I route data to multiple destinations?
Can I route data to multiple destinations?
Yes, you can enable routing to one destination of each type (one BigQuery integration, one Snowflake integration, and one S3 integration). Each integration operates independently.
What format is the routed data in?
What format is the routed data in?
Data format depends on the destination:
- BigQuery: Native BigQuery tables with automatically managed schemas
- Snowflake: Three tables (EVENTS, SYNC_ALIASES, SEGMENTS) with structured schemas
- S3 Streaming: GZIP compressed newline-delimited JSON (.jsonl.gz) with Hive-style partitioning
- S3 Batch: JSON (GZIP compressed) or Parquet (Snappy compressed) with Hive-style partitioning
What if I need to disable or delete a Routing integration?
What if I need to disable or delete a Routing integration?
Disabling or deleting Routing integrations requires assistance from Permutive. Contact support at [email protected] to request disabling routing.
How is data partitioned in my destination?
How is data partitioned in my destination?
Data partitioning varies by destination:This structure enables efficient querying by allowing you to scan only the relevant time partitions.
- BigQuery: Tables named
{event_name}_events(e.g.,Pageview_events) containing daily partitions based on event date. - Snowflake: Tables organized by data type (EVENTS, SYNC_ALIASES, SEGMENTS)
- S3 Streaming: Hive-style partitioning for events, sync_aliases, and segments with structure
type={data_type}/year=YYYY/month=MM/day=DD/hour=HH/ - S3 Batch: Hive-style partitioning for events, sync_aliases, and segments with structure
{data_type}/year=YYYY/month=MM/day=DD/hour=HH/
How do I set up routing for Snowflake or S3?
How do I set up routing for Snowflake or S3?
BigQuery routing can be configured via the Dashboard self-service. For Snowflake and S3 integrations, contact Permutive support at [email protected]. The support team will work with you to configure the destination and provide the necessary authentication details.
Changelog
For detailed changelog information, visit our
Changelog.