Setting Up AWS S3 Streaming Routing

Overview

Set up S3 Streaming routing to export your first-party event data to your AWS S3 bucket in near real-time as GZIP-compressed JSONL files. This guide provides context for choosing S3 Streaming and what to expect after setup.

Prerequisites:

AWS account with S3 bucket creation permissions
Permission to attach a bucket policy to that bucket
S3 bucket in the appropriate AWS region

You do not need to create an IAM user or share AWS access keys with Permutive. Access is granted by attaching a bucket policy that allows a Permutive-owned IAM role to write to your bucket.

When to Choose S3 Streaming

Best for:

Organizations using AWS as their primary cloud provider
Teams needing raw event files for custom processing pipelines
Publishers requiring data in S3 for ingestion into other AWS services (Athena, Redshift, EMR)
Organizations preferring file-based data over database connections

Consider alternatives if:

You prefer automatic schema management in a database (consider BigQuery or Snowflake)
You need immediate SQL query access without additional setup

Setup Steps

S3 Streaming routing requires coordination with Permutive support.

Create Your S3 Bucket

Create a bucket with the following settings:

Bucket name: must be globally unique across all of Amazon S3
Object Ownership: ACLs disabled (Bucket owner enforced)
Block Public Access: Block all public access
Default encryption: SSE-S3 (AES256), the default for new buckets, which requires no additional configuration — see AWS default bucket encryption

Use a region-specific location (e.g., us-east-1, eu-west-1) rather than generic regions.

Attach a Bucket Policy

Attach a policy to the bucket granting Permutive’s IAM role arn:aws:iam::941252478151:role/permutive-s3-routing permission to write objects.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PermutiveRoutingWrite",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::941252478151:role/permutive-s3-routing"
      },
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/YOUR_PREFIX/*"
    },
    {
      "Sid": "PermutiveRoutingList",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::941252478151:role/permutive-s3-routing"
      },
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
    }
  ]
}

If you scope the policy to a prefix, the Resource must end with /* after the prefix — for example arn:aws:s3:::my-bucket/permutive/*. A wildcard does not match across /, so .../permutive* will reject every write.

Attach this at the bucket level (S3 console → your bucket → Permissions → Bucket policy), not to an IAM user or role in your own account.

Contact Permutive Support

Email technical-services@permutive.com with:

Bucket Name
Bucket Region (e.g., us-east-1)
Bucket Prefix (optional, e.g., permutive/ — include the trailing /)
Routing Mode: Streaming

Confirm that the bucket policy above has been attached.

Setup Completion

Permutive will configure your routing instance and notify you when the integration is live.

Understanding S3 Streaming Data Structure

S3 Streaming uses Hive-style partitioning to organize data efficiently:

Folder Structure

s3://bucket/prefix/
├── type=events/
│   ├── year=2026/
│   │   ├── month=01/
│   │   │   ├── day=15/
│   │   │   │   ├── hour=14/
│   │   │   │   │   └── 2026-01-15T14:00:00.000000Z-abc123-worker1.jsonl.gz
│   │   │   │   └── hour=15/
├── type=sync_aliases/
│   └── year=2026/month=01/day=15/hour=14/...
└── type=segment/
    └── timestamp-hash-worker.jsonl.gz

File Format

Format: Newline-delimited JSON (JSONL)
Compression: GZIP (.gz)
Extension: .jsonl.gz
Encoding: UTF-8

Data Types Exported

Data Type	Description	Partitioned
`events`	User behavioral events	Yes (hourly)
`sync_aliases`	Identity synchronization data	Yes (hourly)
`segment`	Segment metadata snapshots	No

See the S3 integration documentation for detailed schema information.

Common Considerations

Latency: S3 Streaming has approximately 5-minute latency from event collection to file availability in S3.

Bucket Prefix: Use a bucket prefix (e.g., permutive/) to organize Permutive data separately from other data in your bucket. The prefix should NOT include a leading / or the bucket name, and should end with a trailing /.

Encryption: Permutive does not specify an encryption algorithm when writing, so objects are encrypted using your bucket’s default encryption setting. SSE-S3 requires nothing further. If your bucket’s default is a customer-managed KMS key, contact Technical Services before setup — our role (arn:aws:iam::941252478151:role/permutive-s3-routing) will need kms:Encrypt and kms:GenerateDataKey on the key, and your own users will need kms:Decrypt to read the files. Buckets encrypted with an AWS-managed key (aws/s3) are not supported: AWS does not permit resources encrypted under an AWS managed key to be shared with other accounts.

What Happens After Setup

Once routing is active:

Files stream to S3 in near real-time with approximately 5-minute latency
Hive-style partitions are created automatically by hour
Event data is written as GZIP-compressed JSONL files
File naming follows the pattern {timestamp}-{hash}-{worker_id}.jsonl.gz

Next Steps

S3 Integration

View full integration documentation

Back to Routing

Return to Routing overview

​Overview

​When to Choose S3 Streaming

​Setup Steps

​Understanding S3 Streaming Data Structure

​Folder Structure

​File Format

​Data Types Exported

​Common Considerations

​What Happens After Setup

​Next Steps

S3 Integration

Back to Routing

Overview

When to Choose S3 Streaming

Setup Steps

Understanding S3 Streaming Data Structure

Folder Structure

File Format

Data Types Exported

Common Considerations

What Happens After Setup

Next Steps