Setup
Troubleshooting
Overview
The AWS S3 integration enables publishers to leverage Permutive’s bi-directional data capabilities with their S3 storage. This integration operates in two modes: Routing (Destination): Export first-party event data from Permutive to S3 buckets. Permutive offers two distinct routing modes:- S3 Streaming (recommended): Near real-time streaming, scoped to the workspace you configure, ideal for low-latency data pipelines and analytics
- S3 Batch: Daily scheduled exports covering every workspace in your organization, with the option of Parquet output
Routing capability requires the Routing package in addition to Core Platform. Contact your Customer Success Manager to enable Routing.
Environment Compatibility
Prerequisites
For Routing (exporting data to S3):- AWS account with permissions to create S3 buckets
- S3 bucket created in the appropriate AWS region
- Ability to configure S3 bucket policies with specific permissions
Setup
- Routing Streaming Setup
- Routing Batch Setup
- Connectivity Setup
S3 Streaming routing exports your first-party event data to an S3 bucket in near real-time as GZIP-compressed JSONL files with Hive-style partitioning. Data arrives with approximately 5-minute latency, making it ideal for low-latency data pipelines and ingestion into AWS services such as Athena, Redshift, or EMR.Setup requires coordination with Permutive support. You will need to create an S3 bucket, attach a bucket policy granting a Permutive-owned IAM role write access, and then share your bucket details with the Permutive team.
Prerequisites
- AWS account with permissions to create S3 buckets
- An S3 bucket in a region-specific location (e.g.,
us-east-1,eu-west-1) with public access blocked and ACLs disabled - A bucket policy granting
arn:aws:iam::941252478151:role/permutive-s3-routingthes3:PutObject,s3:GetObjectands3:DeleteObjectactions on the target path, pluss3:ListBucketon the bucket
What Happens After Setup
Once routing is active:- Files stream to S3 in near real-time with approximately 5-minute latency
- Hive-style partitions are created automatically by hour
- Event data is written as GZIP-compressed JSONL files
- File naming follows the pattern
{timestamp}-{hash}-{worker_id}.jsonl.gz
Data Types
Streaming Schema
Event Data (events)
Event Data (events)
Identity Sync Data (sync_aliases)
Identity Sync Data (sync_aliases)
Batch Schema
Event Tables (e.g., pageview_events)
Event Tables (e.g., pageview_events)
Domains Table (domains)
Domains Table (domains)
File Formats and Compression
Streaming Format
Streaming Format
- Format: Newline-delimited JSON (
.jsonl) - Compression: GZIP (
.gz) - File Extension:
.jsonl.gz - Character Encoding: UTF-8
Batch Formats
Batch Formats
JSON Format
- Format: Newline-delimited JSON
- Compression: GZIP
- File Extension:
.json.gz - Character Encoding: UTF-8
Parquet Format
- Format: Apache Parquet columnar format
- Compression: Snappy
- File Extension:
.snappy.parquet - Schema: Derived from BigQuery table structure
Parquet format is recommended for data warehouse ingestion and analytics workloads due to better compression and query performance.
Troubleshooting
Permission Denied Errors
Permission Denied Errors
Symptom: Files are not appearing in S3 bucket, or logs show permission errors.Solution:
-
Verify the bucket policy grants
arn:aws:iam::941252478151:role/permutive-s3-routingall required permissions:s3:PutObjects3:GetObjects3:DeleteObjects3:ListBucket
-
Check that the bucket policy includes the correct bucket ARN:
-
If the policy is scoped to a prefix, confirm the
Resourceends with/*after the prefix (.../permutive/*, not.../permutive*) — a wildcard does not match across/ - Confirm the policy is attached to the bucket, not to an IAM user or role in your own account
- If your bucket’s default encryption uses a customer-managed KMS key, confirm the Permutive role also has permissions on that key (see KMS Encryption Issues below)
If your integration was set up using AWS access keys rather than a bucket policy, contact Technical Services — we can migrate it to role-based access so you no longer need to rotate credentials.
Data Not Appearing in S3 (Streaming)
Data Not Appearing in S3 (Streaming)
Symptom: No files appearing in S3 bucket after setup, or files stopped appearing.Solution:
- Verify the Permutive SDK is properly deployed and events are being collected (check Event Inspector in the Dashboard)
- Low-traffic sites may see longer delays between files due to batch size thresholds
-
Verify the bucket region matches the configured region:
- Region must be specific (e.g.,
eu-central-1, not justEU)
- Region must be specific (e.g.,
-
Verify bucket path structure is correct:
- If issues persist, contact Permutive support at technical-services@permutive.com with your integration details
Data Not Appearing in S3 (Batch)
Data Not Appearing in S3 (Batch)
Symptom: Daily batch exports are missing or delayed.Solution:
- Batch exports run on 24-hour cycles. Check if sufficient time has passed since the last export window.
- Verify the Permutive SDK is properly deployed and events are being collected
- Contact Permutive support at technical-services@permutive.com to check batch export job logs and status
Incorrect Bucket Path Structure
Incorrect Bucket Path Structure
Symptom: Files appearing in unexpected locations or wrong folder structure.Solution:
-
Verify the
bucketPrefixconfiguration:- Should NOT include leading
/unless intentional - Should NOT include bucket name
- Example:
permutive/not/permutive/ors3://bucket/permutive/
- Should NOT include leading
-
For Streaming, data uses Hive-style partitioning:
type=events/year=2026/month=01/day=15/hour=14/- This is expected behavior and cannot be customized
-
For Batch, data is organized by table name:
data/{table_name}/year=2026/month=1/day=15/- This is expected behavior and cannot be customized
Bucket Policy Validation Errors
Bucket Policy Validation Errors
Symptom: AWS returns validation errors when applying bucket policy.Solution:
-
Ensure the bucket policy JSON is valid:
- Check for missing commas, brackets, or quotes
- Use AWS Policy Generator or an online JSON validator
-
Verify ARN format is correct:
- Bucket ARN:
arn:aws:s3:::BUCKET_NAME - Object ARN:
arn:aws:s3:::BUCKET_NAME/* - Note the three colons
:::before bucket name
- Bucket ARN:
-
Confirm the
StringEqualscondition is correctly formatted:
Missing Event Types or Fields
Missing Event Types or Fields
Symptom: Some event types or fields are not appearing in exported data.Solution:
-
Verify events are being collected in Permutive:
- Check Event Inspector in the Dashboard to confirm events are tracked
- Use browser developer console to verify SDK is firing events
-
Check event schema matches expected structure:
- Events must include required fields:
event_id,user_id,event_name, etc. - Custom properties are in the
propertiesobject
- Events must include required fields:
-
Schema changes may require integration reconfiguration:
- Contact Permutive support if you’ve made significant schema changes
KMS Encryption Issues
KMS Encryption Issues
Symptom: Errors related to KMS encryption when writing to S3.Solution:
-
If using customer-managed KMS keys, verify the Permutive role (
arn:aws:iam::941252478151:role/permutive-s3-routing) has KMS permissions: - Confirm the KMS key policy allows the Permutive role to use the key
- Verify the S3 bucket’s default encryption settings are compatible
SSE-S3 (
AES256) works with no additional configuration. Customer-managed KMS keys require permissions on both sides — contact Permutive support. Note that an AWS-managed KMS key (aws/s3) cannot be used, as AWS does not permit resources encrypted under an AWS managed key to be shared with other accounts.Changelog
No changes listed yet. For detailed changelog information, visit our Changelog.