LogoLogo
HomeUser GuidesAPI Reference
  • ๐Ÿ‘Welcome to Whaly ๐Ÿณ
  • Team
    • ๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘งโ€๐Ÿ‘ฆWhat is a team?
    • ๐Ÿ›ก๏ธSingle Sign On
    • ๐ŸฅทImpersonate
  • Organisation
    • ๐ŸซWhat is an organisation?
    • ๐Ÿ“คUpload your Organisation logo
    • ๐Ÿ”‘Manage Access to your organisation
    • โ“Understanding Licences
    • ๐Ÿ‘ฎUnderstanding User Roles
  • User Management
    • ๐Ÿท๏ธUser Attributes
    • ๐Ÿ‘ญUser Groups
    • ๐Ÿค–Service Accounts
  • Workspace
    • โœ๏ธWorkspace
    • ๐Ÿ“‚Report Folders
    • โœจSharing & Collaboration
      • Share a report to the Web
    • ๐Ÿ“—Catalog
    • โš™๏ธSettings
  • Warehouse
    • ๐ŸฆConnect your Warehouse
    • โš”๏ธAmazon Athena
      • Connect your Athena
    • ๐ŸฎAmazon Redshift
      • Connect your Redshift
    • ๐ŸงฑDatabricks
      • Connect your Databricks
    • ๐Ÿ”ทGoogle BigQuery
      • Connect your BigQuery
      • Grant access to BigQuery datasets
      • Enable multi project support
    • ๐Ÿ˜Postgres
      • Connect your Postgres
      • Whitelisting Whaly IPs
    • โ„๏ธSnowflake
      • Connect your Snowflake
      • Giving access to Snowflake data
  • Models
    • ๐Ÿ’žModels sync
      • Where should my models be managed?
      • dbt Cloud
        • Configuration
        • Exposing models into Whaly
    • ๐Ÿ“ฅPersistence Engine
      • Configuration
        • Snowflake
      • Check Materialisation runs status
  • Workbench
    • ๐Ÿš€Navigating the workbench
    • ๐Ÿ› ๏ธModeling
      • Understanding Datasets
        • General Information
        • Drills
        • Relationships
        • Primary Keys
        • Cache
      • Model Data
        • SQL Models
        • Flow Models
          • Create a Flow
          • Update a Flow
          • Flow steps
            • From Model
            • From Raw
            • Hide Column
            • Filter
            • Lookup
            • Rollup
            • Formulas
            • Group
            • Union
      • Import raw data
        • From your warehouse
        • From third party data
    • ๐ŸงญExplorations
      • Configure an exploration
      • Exploration Templates
      • Tables
        • Configure a table
        • Add related data
      • Metrics
        • Create a Metric
        • Create a Calculated Metric
        • Create Drill Downs
        • Using custom formatting
      • Dimensions
        • Create a dimension
      • Check measure usage
      • Row Level Access
  • Data consumption
    • ๐Ÿ’กExploring data
      • How to explore data
      • Drill Down
      • Forecasting
    • ๐Ÿ’นWhat is a Report?
    • ๐Ÿ“ŠDashboards
      • Create a dashboard
      • Manage tiles
        • Add chart tiles
        • Add text tiles
        • Add navigation tiles
        • Arranging tiles
      • Add a description
      • Share a dashboard
      • Filter a dashboard
      • Push dashboard
      • Delete a dashboard
    • ๐Ÿ“ˆQuestions
      • Create a question
      • Add a description
      • Share a question
      • Push question data
      • Delete a question
    • ๐Ÿ”Refreshing a report
  • Data visualisation
    • ๐ŸŽจTheming
    • ๐Ÿ–Œ๏ธChart your data
      • Bar chart
      • Calendar chart
      • Funnel chart
      • Gauge chart
      • Geo map chart
      • Heatmap chart
      • Interactive map chart
      • Line chart
      • Metric chart
      • Pie chart
      • Retention chart
      • Table chart
      • Treemap chart
      • Waterfall chart
      • Custom time format in time series
  • Content management
    • โญExplorations Section
    • โœ‚๏ธBulk Content Management
  • Embedding
    • ๐Ÿ“ŒEmbed in Business apps
      • Notion
      • Clickup
      • Hubspot
      • Google Chrome
        • ๐ŸŒฑInstall
        • โš™๏ธConfigure the Chrome extension
    • ๐Ÿ‘ฉโ€๐Ÿ’ปEmbedding API
    • ๐ŸชŸPartner Portal
  • Workflows
    • ๐Ÿš€Push
      • Configure a Push
      • Manage Push
    • ๐Ÿ’ผManage Installed Actions
    • โšกActions catalog
      • Airtable
      • Google Sheets
      • Slack
      • Sendgrid
      • Webhook
  • Platform concepts
    • โœณ๏ธQuery Mode
    • ๐Ÿ’ซCaching
  • Guides
    • โ›‘๏ธSupport
  • User
    • ๐ŸคฉUpload your profile picture
  • Connectors
    • ๐Ÿ”ŒConnect your Sources
    • โš™๏ธWarehouse setup
      • BigQuery
        • Configure a Cloud Storage cleaning rule
      • Snowflake
    • โ˜๏ธWhitelisting Whaly connectors IPs
    • ๐Ÿ”SSH Tunneling
    • ๐Ÿ„Schema drift
    • ๐Ÿ”Replication method
    • ๐Ÿง™Source monitoring
    • ๐ŸŽSource catalog
      • Community
        • Github Stars
        • Slack
        • Orbit
      • Database
        • PostgreSQL / Postgres
          • ๐Ÿ’กTip: Extracting the relationships
        • MariaDB / MySQL
      • eCommerce
        • WooCommerce
      • Engineering
        • Github
      • Finance
        • Brex
        • Pennylane
          • Pennylane (Redshift) - General Ledger & Trial Balance
          • Pennylane API - Customer Invoices
        • Qonto
        • Stripe
        • QuickBooks
      • Marketing / Growth
        • Facebook Ads
        • Google Ads
        • Google Analytics
          • Google Analytics (V4)
          • Google Analytics (Universal Analytics)
        • LaGrowthMachine
        • lemlist
        • LinkedIn Ads
        • Salesloft
      • No-Code
        • Airtable
        • Bubble
        • Google Sheets
      • Support
        • Intercom
      • Product
        • Amplitude
        • MixPanel
        • Segment
      • Sales / CRMs
        • Aircall
        • Pipedrive
        • Hubspot
        • Recruit CRM
        • Salesforce
Powered by GitBook
On this page
  • Why syncing my Postgres to Whaly?
  • Requirements
  • Configuring the source
  • Configuring a database read only user
  • Troubleshooting

Was this helpful?

  1. Connectors
  2. Source catalog
  3. Database

PostgreSQL / Postgres

PreviousDatabaseNextTip: Extracting the relationships

Last updated 11 months ago

Was this helpful?

Why syncing my Postgres to ?

Syncing your Postgres can be useful for various cases

  • Share your product usage with your support and customer success team

  • Combine Postgres data with your CRM data to detect churn or upsells

  • ....

Requirements

  1. Postgres v9.4.x or above

  2. Allow connections from Whaly to your Postgres database by

Configuring the source

The Postgres connector will need the following information:

  • Host: This should be an IP address or an URL pointing to your Postgres server

  • Port: This should be the Port of your Postgres, Default to 5432.

  • User: This should be the username of a read only Postgres user.

  • Password: The password associated with the user

  • Database name: The name of the database to be synced. In a Postgres server, there can be multiple databases, depending on your setup. If you want to replicate multiple databases, you'll have to create one connector per database.

  • Schema name: The name of the schema to be synced. In a Postgres database, there can be multiple schema, depending on your setup. If you want to replicate multiple schemas, you'll have to create one connector per schema. Default: public

Configuring a database read only user

Whaly will need a Postgres user in order to connect to your database. We recommend to create a specific user with only Read Only access for the replication.

Using Postgres access control features, you'll be able to finely configure which data (tables, columns) should be replicated within Whaly.

Below instructions, executed with a Postgres Admin user, will create a read only user that can be used to sync all tables of your Postgres server.

CREATE USER <username> PASSWORD '<some-password>';
GRANT USAGE ON SCHEMA "public" TO <username>;
GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO <username>;
ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO <username>;

Note:

  • Replace <username> and <some-password> by the login and password that you want to give to the read only user used for the replication

  • If you're using another SCHEMA that the default "public" one in Postgres, please replace "public" with your SCHEMA name.

Troubleshooting

I got the "Error: terminating connection due to conflict with recovery" message in the logs

When the connector is reading from a Postgres replica that is configured as a Hot Standby, any update from the primary server will terminate queries on the replica after a certain amount of time, default to 30 seconds.

Here is the typical exception:

terminating connection due to conflict with recovery

Possible solutions include:

  • [Recommended] Sync data when there is no update running in the primary server, or sync data from the primary server.

This default waiting time is not enough to sync any meaning amount of data. See the Handling Query Conflicts section in the Postgres for detailed explanation.

[Recommended] Set to true on the replica server. This parameter will prevent the primary server from deleting the write-ahead logs when the replica is busy serving user queries. However, the downside is that the write-ahead log will increase in size.

[Not Recommended] Increase and to be larger than the amount of time needed to complete the data sync. However, it is usually hard to tell how much time it will take to sync all the data. This approach is not very practical.

๐ŸŽ
documentation
hot_standby_feedback
max_standby_archive_delay
max_standby_streaming_delay
Whaly
whitelisting Whaly IPs
You'll find instruction below to create it