LogoLogo
HomeUser GuidesAPI Reference
  • ๐Ÿ‘Welcome to Whaly ๐Ÿณ
  • Team
    • ๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘งโ€๐Ÿ‘ฆWhat is a team?
    • ๐Ÿ›ก๏ธSingle Sign On
    • ๐ŸฅทImpersonate
  • Organisation
    • ๐ŸซWhat is an organisation?
    • ๐Ÿ“คUpload your Organisation logo
    • ๐Ÿ”‘Manage Access to your organisation
    • โ“Understanding Licences
    • ๐Ÿ‘ฎUnderstanding User Roles
  • User Management
    • ๐Ÿท๏ธUser Attributes
    • ๐Ÿ‘ญUser Groups
    • ๐Ÿค–Service Accounts
  • Workspace
    • โœ๏ธWorkspace
    • ๐Ÿ“‚Report Folders
    • โœจSharing & Collaboration
      • Share a report to the Web
    • ๐Ÿ“—Catalog
    • โš™๏ธSettings
  • Warehouse
    • ๐ŸฆConnect your Warehouse
    • โš”๏ธAmazon Athena
      • Connect your Athena
    • ๐ŸฎAmazon Redshift
      • Connect your Redshift
    • ๐ŸงฑDatabricks
      • Connect your Databricks
    • ๐Ÿ”ทGoogle BigQuery
      • Connect your BigQuery
      • Grant access to BigQuery datasets
      • Enable multi project support
    • ๐Ÿ˜Postgres
      • Connect your Postgres
      • Whitelisting Whaly IPs
    • โ„๏ธSnowflake
      • Connect your Snowflake
      • Giving access to Snowflake data
  • Models
    • ๐Ÿ’žModels sync
      • Where should my models be managed?
      • dbt Cloud
        • Configuration
        • Exposing models into Whaly
    • ๐Ÿ“ฅPersistence Engine
      • Configuration
        • Snowflake
      • Check Materialisation runs status
  • Workbench
    • ๐Ÿš€Navigating the workbench
    • ๐Ÿ› ๏ธModeling
      • Understanding Datasets
        • General Information
        • Drills
        • Relationships
        • Primary Keys
        • Cache
      • Model Data
        • SQL Models
        • Flow Models
          • Create a Flow
          • Update a Flow
          • Flow steps
            • From Model
            • From Raw
            • Hide Column
            • Filter
            • Lookup
            • Rollup
            • Formulas
            • Group
            • Union
      • Import raw data
        • From your warehouse
        • From third party data
    • ๐ŸงญExplorations
      • Configure an exploration
      • Exploration Templates
      • Tables
        • Configure a table
        • Add related data
      • Metrics
        • Create a Metric
        • Create a Calculated Metric
        • Create Drill Downs
        • Using custom formatting
      • Dimensions
        • Create a dimension
      • Check measure usage
      • Row Level Access
  • Data consumption
    • ๐Ÿ’กExploring data
      • How to explore data
      • Drill Down
      • Forecasting
    • ๐Ÿ’นWhat is a Report?
    • ๐Ÿ“ŠDashboards
      • Create a dashboard
      • Manage tiles
        • Add chart tiles
        • Add text tiles
        • Add navigation tiles
        • Arranging tiles
      • Add a description
      • Share a dashboard
      • Filter a dashboard
      • Push dashboard
      • Delete a dashboard
    • ๐Ÿ“ˆQuestions
      • Create a question
      • Add a description
      • Share a question
      • Push question data
      • Delete a question
    • ๐Ÿ”Refreshing a report
  • Data visualisation
    • ๐ŸŽจTheming
    • ๐Ÿ–Œ๏ธChart your data
      • Bar chart
      • Calendar chart
      • Funnel chart
      • Gauge chart
      • Geo map chart
      • Heatmap chart
      • Interactive map chart
      • Line chart
      • Metric chart
      • Pie chart
      • Retention chart
      • Table chart
      • Treemap chart
      • Waterfall chart
      • Custom time format in time series
  • Content management
    • โญExplorations Section
    • โœ‚๏ธBulk Content Management
  • Embedding
    • ๐Ÿ“ŒEmbed in Business apps
      • Notion
      • Clickup
      • Hubspot
      • Google Chrome
        • ๐ŸŒฑInstall
        • โš™๏ธConfigure the Chrome extension
    • ๐Ÿ‘ฉโ€๐Ÿ’ปEmbedding API
    • ๐ŸชŸPartner Portal
  • Workflows
    • ๐Ÿš€Push
      • Configure a Push
      • Manage Push
    • ๐Ÿ’ผManage Installed Actions
    • โšกActions catalog
      • Airtable
      • Google Sheets
      • Slack
      • Sendgrid
      • Webhook
  • Platform concepts
    • โœณ๏ธQuery Mode
    • ๐Ÿ’ซCaching
  • Guides
    • โ›‘๏ธSupport
  • User
    • ๐ŸคฉUpload your profile picture
  • Connectors
    • ๐Ÿ”ŒConnect your Sources
    • โš™๏ธWarehouse setup
      • BigQuery
        • Configure a Cloud Storage cleaning rule
      • Snowflake
    • โ˜๏ธWhitelisting Whaly connectors IPs
    • ๐Ÿ”SSH Tunneling
    • ๐Ÿ„Schema drift
    • ๐Ÿ”Replication method
    • ๐Ÿง™Source monitoring
    • ๐ŸŽSource catalog
      • Community
        • Github Stars
        • Slack
        • Orbit
      • Database
        • PostgreSQL / Postgres
          • ๐Ÿ’กTip: Extracting the relationships
        • MariaDB / MySQL
      • eCommerce
        • WooCommerce
      • Engineering
        • Github
      • Finance
        • Brex
        • Pennylane
          • Pennylane (Redshift) - General Ledger & Trial Balance
          • Pennylane API - Customer Invoices
        • Qonto
        • Stripe
        • QuickBooks
      • Marketing / Growth
        • Facebook Ads
        • Google Ads
        • Google Analytics
          • Google Analytics (V4)
          • Google Analytics (Universal Analytics)
        • LaGrowthMachine
        • lemlist
        • LinkedIn Ads
        • Salesloft
      • No-Code
        • Airtable
        • Bubble
        • Google Sheets
      • Support
        • Intercom
      • Product
        • Amplitude
        • MixPanel
        • Segment
      • Sales / CRMs
        • Aircall
        • Pipedrive
        • Hubspot
        • Recruit CRM
        • Salesforce
Powered by GitBook
On this page
  • A new column is created
  • A column change its type
  • A column is deleted

Was this helpful?

  1. Connectors

Schema drift

Schema drift is the case where an actual table schema in the data source is different from the schema of the target table in the Warehouse.

This is happening when the Data source schema is updated:

  • A new column is created

  • A column change its type

  • A column is deleted

When a Schema is drifting, there is an inconsistency between the Data Source schema and the destination schema.

Whaly connectors are handling such cases to reduce the level of data inconsistencies while ensuring that all analysis / queries made on the destination table in the Warehouse continue to work.

A new column is created

Whenever a new column is created in the data source, Whaly will create a new column in the destination table in the Warehouse at the next sync.

For "FULL TABLE" replicated tables, the new column will be completely filled at the next sync.

For "INCREMENTAL" replicated tables, the new column will be filled only for recently updated records at the next sync. A full resync of the table will be necessary to backfill all the records.

A column change its type

Whenever a column in the data source change its type, ex. String becomes a Number, a new column will be created on the destination table on the warehouse to hold the new values.

Previous values with the previous type will remain untouched in the previous column.

Newly created columns have the following format: columnName_NEWTYPE, ex. amount_eur_STRING

For "FULL TABLE" replicated tables, the new column will be completely filled at the next sync.

For "INCREMENTAL" replicated tables, the new column will be filled only for recently updated records at the next sync. A full resync of the table will be necessary to backfill all the records.

A column is deleted

Whenever a new column is deleted in the data source, Whaly will stop syncing it in the destination table in Warehouse. However, values already synced in the past will remain untouched and can still be read in the Warehouse so existing analysis will keep working.

PreviousSSH TunnelingNextReplication method

Last updated 2 years ago

Was this helpful?

๐Ÿ„