Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.permutive.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Overview

When your source data changes, Permutive can detect and apply certain types of schema updates to an existing import. This guide explains how to action those updates — including adding new columns — and how to resolve common schema-related errors.
Prerequisites:
  • An active Connectivity source connection
  • An existing import configured against that source
  • Access to Sources > Imports in the Permutive Dashboard

​
Supported Schema Changes

What you can change depends on your source type. Select your source below for details.
Support depends on the file format. We recommend Parquet where possible — it carries its schema per file, supports native typed columns, and avoids CSV parsing edge cases.Supported
  • Adding new nullable columns (at any position)
  • Adding fields to nested struct columns — for example, if address contains (city, street) and a new zip field is added, you can accept the change to import all sub-fields. Nested struct changes are all-or-nothing: you must accept all sub-fields or none.
Not supported
  • Widening or narrowing a column’s data type (e.g. INT64 → NUMERIC, DATE → TIMESTAMP)
  • Removing, renaming, or reordering columns
Why Parquet
  • Type safety — native typed columns (int, float, string, timestamp) avoid issues like integers silently becoming floats
  • Better compression — typically 5–10× smaller than CSV
  • No parsing edge cases — no issues with escaping, quoting, delimiters, or encoding
If you expect schema changes over time, consider migrating the table to Parquet. Parquet embeds the schema per file, so new and old files with different column layouts can co-exist under the same prefix — provided the files are named so that the alphabetically last file reflects the latest schema (Permutive infers the schema from the alphabetically last file in the prefix).

​
CSV

Supported
  • Adding new columns, appended to the end of the row only
Not supported
  • Inserting a new column in the middle of the row
  • Removing, renaming, or reordering columns
  • Any change to an existing column’s data type — CSV has no reliable way to enforce types
  • Nested struct columns (not supported in CSV)
When you add a new column to a CSV source, existing CSV files written with the old schema must be cleared from the table prefix before triggering a resync. Mixing CSV files with different schemas under the same table prefix will cause the import to fail.
Permutive’s underlying infrastructure supports a maximum of 10,000 columns per table. This limit applies to all source types.
If you make an unsupported change for your source type (such as removing or renaming columns, or an unsupported data type change), the source schema must be reverted to its previous state. Unsupported changes can cause cohorts that reference the affected import’s data to stop functioning correctly until the schema is restored. See Troubleshooting.

​
How Schema Resync Works

  1. You change the source schema in your data warehouse or data lake (for example, add a column to a CSV or Parquet table in S3).
  2. You trigger a resync from the Permutive dashboard — detection does not happen automatically on a schedule.
  3. Permutive re-reads the schema of every table under the connection’s prefix. The resync can take up to 30 minutes.
  4. If changes are detected, affected imports are flagged with a banner. You review the changes per import and choose which new columns to bring in.

​
Steps

1

Trigger a resync on the connection

  1. Go to Connectivity > Connections and open the affected connection (or open an affected import and use the header action), then click Resync Connection.
  2. Confirm in the dialog: “You are about to resync your connection to the data source. This can take up to 30 minutes to complete.”
  3. The button shows Resyncing… while Permutive refreshes the schema from source. You can close the page — the resync runs in the background.
Resync runs at the connection level and checks all tables under the connection’s schema prefix. You don’t need to trigger it separately for each import.
2

Review detected changes

Once the resync completes, the Imports page shows a banner — “Schema changes detected in N imports” — and you can use the Pending changes filter to find affected imports. Each affected row is marked with an info-circle tooltip.Open an affected import. You’ll see one of two banners depending on what was detected:
  • New changes detected in your source schema (info) — Supported changes such as new columns. Click Review Changes to open the column-selection dialog.
  • Unsupported changes detected in your source schema (warning) — the detected changes require manual intervention (for example, a column was removed or renamed). Click Learn more for details; you’ll need to revert the change at source or create a new import.
3

Accept the columns you want

In the Review Changes dialog, tick the checkbox next to each new column you want to include in the import, then save. You don’t need to accept every new column — pick only the ones relevant to your use case, or dismiss the banner entirely if none are needed.
The Supported-changes banner is dismissable per schema version. If the source schema changes again later, a new banner appears so you can review the new columns without re-surfacing the previous ones.
4

Verify the import

After accepting, Permutive updates the import’s column mapping and resumes syncing. Allow time for data to be ingested against the new columns, then check the import details page to confirm the update completed successfully.If data isn’t appearing after a reasonable period, see Troubleshooting below.

​
Troubleshooting

  1. Check the resync status — on the import details page, confirm whether the resync shows as Completed, In Progress, or Error. If still in progress, wait before taking further action.
  2. Wait for ingestion — data can take some time to appear after a schema update is applied.
  3. Check your source — confirm the new columns are present and correctly defined in your data warehouse or lake.
  4. Check for unsupported changes — if you’ve also renamed or removed a column, this may be blocking the update.
  1. Check the resync status — look for an error on the import details page. The error message should indicate the type of issue.
  2. Check your source schema — verify the columns are correctly defined in your data warehouse tool.
  3. Revert and re-apply — revert the change at source, re-apply it, then trigger a refresh or resync in Permutive.
  4. Check for unsupported changes — confirm you haven’t also made a change listed under Supported Schema Changes.
Solution: If none of the above resolves the issue, contact Permutive Support with the error message and the name of the affected import.
Any change to an existing column’s data type — whether widening (for example INT64 → NUMERIC) or narrowing — is not supported. The change will be reported as Unsupported and must be reverted at source to restore normal import behaviour. Cohorts referencing the affected data may stop functioning correctly until the schema is reverted.Solution:
  1. Revert the type change at source so the column matches the original type
  2. Trigger Resync Connection to clear the unsupported-change state
  3. If reverting is not possible, apply the updated schema to a new table, delete the existing import (this will impact associated cohorts), and create a new import pointing to the new table
Removing columns is not currently supported on an existing import.Solution:
  1. Delete the existing import in Permutive (if it’s no longer needed, as this will impact associated cohorts)
  2. Create a new import against the updated schema
Permutive has detected a mismatch between the schema it has on record and what it can read from your source. For CSV sources, this most commonly happens when old and new files with different column layouts co-exist under the same table prefix.Solution:
  1. For CSV: remove files that use the old schema from the table prefix so only the new schema remains
  2. Revert any other recent schema changes at source if needed
  3. Trigger Resync Connection from the Connections or Imports page
If the resync still errors, contact Permutive Support.
The source table or schema this import points to can no longer be found.Solution:
  1. Review the change in your warehouse and restore or recreate the table or schema as needed
  2. If needed, delete the import in Permutive and create a new one pointing to the correct location
Solution:
  1. Revert any recent changes at source
  2. Re-apply the changes
  3. Trigger a resync in Permutive from the import details page
If the issue persists, delete the import and create a new one. Contact Permutive Support if the error continues.

​
Next Steps

Sources Overview

Learn more about managing source connections and schemas

Creating an Import

Set up a new import from a source connection