Tiered Tables in Hybrid Manager

Suggest edits

Tiered Tables in Hybrid Manager (HM) provide an automated solution for managing large time-series or historical datasets by moving older data from PGD clusters to cost-effective object storage in Apache Iceberg format.

Hybrid Manager integrates this capability across:

  • PGD clusters with BDR AutoPartition
  • PGAA extensions for offloading
  • Lakehouse Clusters for querying
  • Centralized catalog services (Lakekeeper or external Iceberg catalogs)

For a conceptual overview of Tiered Tables, see Understanding Tiered Tables with EDB Postgres.

Why use Tiered Tables with Hybrid Manager

  • Lower storage costs: Offload "cold" data to object storage (Iceberg) and shrink primary PGD transactional tables.
  • Faster transactional performance: Keep "hot" data partitions small for efficient PGD operations.
  • Automated lifecycle management: Move data across tiers automatically based on age.
  • Transparent analytics: Query both hot and cold data via PGD parent table or Lakehouse Cluster.
  • Unified management: Configure and monitor all components through Hybrid Manager.

Key terms and architecture overview

For definitions of core analytics terms used in Hybrid Manager—such as PGFS, PGAA, Analytics Offload, and Lakehouse Cluster—see Analytics Concepts in Hybrid Manager.

When should I use Tiered Tables in Hybrid Manager?

Use Tiered Tables in Hybrid Manager when you want to:

  • Manage large time-series datasets in a cost-efficient way.
  • Keep PGD operational tables lean for better performance.
  • Meet compliance needs by keeping older data available but outside of PGD storage.
  • Enable BI users to run historical trend queries without impacting production databases.
  • Automate your data lifecycle with minimal manual intervention.

Use cases for Tiered Tables

  • Time-series data: Logging, IoT sensor readings, application telemetry.
  • Archival: Long-term retention of cold data for compliance.
  • Historical trend analysis: BI tools querying years of data without impacting PGD performance.
  • Large, append-mostly tables: Keep transactional footprint small while retaining full analytical access.

How Tiered Tables work in your HM architecture

  • PGD clusters: Manage partitioning and automatic offload of old partitions to Iceberg.
  • PGFS storage locations: Define object storage targets for offload.
  • Iceberg catalogs: Optionally manage offloaded tables in a catalog (Lakekeeper or external).
  • Lakehouse Clusters: Provide scalable analytical compute to query offloaded Iceberg data.
  • Monitoring: Use HM monitoring tools and observability queries to track offload status and storage savings.

Prerequisites within EDB Hybrid Manager

Before implementing Tiered Tables in HM:

  • Active Hybrid Manager instance
  • Provisioned PGD cluster: Version 6.0+ with PGAA and PGFS extensions enabled
  • Lakehouse Cluster (recommended): For querying offloaded data
  • Catalog service: Optional, but recommended — HM-managed Lakekeeper or external REST-compatible catalog
  • Machine user for catalog (if using catalog): With appropriate catalog data writer/reader permissions
  • Object storage: S3-compatible, with credentials if private
  • User permissions: Database user must have create/alter/execute privileges for BDR and PGAA functions

Main capabilities

  • Automated partitioning: Define BDR AutoPartition strategy and analytics_offload_period.
  • Storage tiering: Use PGFS or Iceberg catalog targets for offloaded data.
  • Query transparently: PGD parent table queries can hit both local and Iceberg tiers. Lakehouse Clusters can query Iceberg tables directly.
  • Monitor status: Track offload progress, validate Iceberg content, and observe space savings.

Getting started with Tiered Tables in Hybrid Manager

To begin using Tiered Tables with Hybrid Manager:

  1. Configure PGFS storage for Tiered Tables.
  2. Configure PGD node group for analytics offload.
  3. Configure BDR AutoPartition with analytics offload.
  4. (Optional) Query Tiered Tables from PGD and Lakehouse.
  5. Monitor Tiered Tables status and storage savings.

Related How-Tos

Observability tips

  • Use HM dashboards for PGD cluster health and offload progress.
  • Run analytics queries on bdr.analytics_table and partition views.
  • Use pg_total_relation_size() to observe space reclaimed on PGD nodes.
  • Use cloud storage console or analytics to track Iceberg object size growth.

Next topic

Lakehouse Clusters in Hybrid Manager

← Prev

Solving Business Problems with Analytics in Hybrid Manager

↑ Up

Analytics in Hybrid Manager

Next →

Sovereign AI and Data Factory

Could this page be better? Report a problem or suggest an addition!