Troubleshooting WarehousePG Disaster Recovery

Suggest edits

WarehousePG Disaster Recovery (whpg-dr) issues typically surface as backup failures, restore errors, or configuration problems. In each case, the operational logs contain the detailed output needed to identify the cause.

Finding logs

whpg-dr writes operational logs to $HOME/gpAdminLogs by default. The log file for a given day is whpg_dr-YYYY-MM-DD.log. Barman per-segment logs are written to the same directory as barman-<cluster_name>.log.

To increase log file verbosity for a single command, use --log-file-level debug:

whpg-dr check my_cluster --log-file-level debug

Backup configuration failures

Symptom:

Error: distribute barman configs: rsync barman_conf to host sdw1 failed: exit status 23
rsync: [receiver] failed to set permissions on ".../.whpg-lab-global.conf.yU9Gmu": No such file or directory

Cause: The WHPG_DR_HOME directory doesn't exist on the target segment host. whpg-dr uses rsync to distribute Barman configuration to segment hosts, and rsync can't create the intermediate directories.

Resolution: Create the WHPG_DR_HOME directory on all segment hosts (as gpadmin), then re-run configure backup:

gpssh -f all_hosts -e "mkdir -p $WHPG_DR_HOME/my_cluster/backup/barman_conf"
whpg-dr configure backup /home/gpadmin/backup.yaml

Verifying backup integrity

Use whpg-dr check-backup to verify that all backup files for each segment are present and accessible in the storage repository:

whpg-dr check-backup my_cluster --backup-name 20260601T120000_base_backup
Output
Finding backup...
Checked backup 20260601T120000 on server my_cluster-seg-1
Checked backup 20260601T120000 on server my_cluster-seg0
Checked backup 20260601T120000 on server my_cluster-seg1
Checked backup 20260601T120000 on server my_cluster-seg2
Checked backup 20260601T120000 on server my_cluster-seg3
All backup checks completed successfully

whpg-dr check failures

These errors surface when running whpg-dr check, which verifies that Barman can reach each segment and that WAL archiving is configured correctly.

Barman can't connect to a segment

Symptom:

check completed with 2 errors:
  - my_cluster-seg0: Server my_cluster-seg0:
      PostgreSQL: FAILED

Cause: Barman can't connect to the segment's Postgres process. Check the Barman log for the actual connection error:

cat $HOME/gpAdminLogs/barman-my_cluster.log

A typical entry looks like:

barman.postgres WARNING: Error retrieving PostgreSQL status: connection to server at "sdw1" (192.168.1.10), port 6000 failed: FATAL: no pg_hba.conf entry for host "192.168.1.10", user "gpadmin", database "postgres", SSL off

Barman uses the connection string:

host=<segment_host> user=<pg_user> dbname=postgres port=<segment_port> options='-c gp_role=utility'
Note

On WarehousePG 6, use gp_session_role=utility instead of gp_role=utility.

Resolution:

  1. Confirm that pg_hba.conf has an entry allowing the Barman user to connect from the segment host using a non-local socket:

    host    all    gpadmin    <segment_host_ip>/32    trust

  2. After editing pg_hba.conf, reload:

    gpstop -u

  3. Re-run the check:

    whpg-dr check my_cluster

whpg-dr check fails immediately after enabling whpg-dr

Symptom:

whpg-dr check my_cluster
Running barman check for server my_cluster-seg-1
Running barman check for server my_cluster-seg0
Running barman check for server my_cluster-seg1
Running barman check for server my_cluster-seg2
Running barman check for server my_cluster-seg3
Server w9-2-seg1:
	WAL archive: FAILED (please make sure WAL shipping is setup)

Cause: Running whpg-dr check immediately after enabling whpg-dr and restarting the cluster, a segment hasn't archived a WAL file yet.

Resolution: Generate some database activity, or wait for activity to occur, then run the check again:

whpg-dr check my_cluster

WAL archive check keeps failing after activity

Symptom:

Unlike whpg-dr check fails immediately after enabling whpg-dr, the check still fails on every segment after generating real write activity or forcing a WAL switch. The Barman log shows each segment finding nothing to archive:

cat $HOME/gpAdminLogs/barman-my_cluster.log
barman.wal_archiver INFO: No xlog segments found from file archival for my_cluster-seg0.
barman.server ERROR: Check 'WAL archive' failed for server 'my_cluster-seg0'

Cause: The storage repository path exists on the coordinator but wasn't created or mounted on one or more segment hosts. Each segment archives its own WAL locally to that path, if the path is missing there, WAL never reaches the shared repository, no matter how much write activity occurs. Confirm this by checking the path directly on the segment hosts:

gpssh -f all_hosts -e "ls -ld /path/to/storage"

Resolution: Create or mount the storage path on every segment host, owned by the WarehousePG cluster owner, then re-run the check:

gpssh -f all_hosts -e "sudo mkdir -p /path/to/storage && sudo chown gpadmin:gpadmin /path/to/storage"
whpg-dr check my_cluster

If the path is meant to be a shared NFS mount rather than local disk, mount the actual share on each segment host instead of creating a bare local directory, otherwise each segment archives to its own local disk rather than the shared repository. See Setting up the storage repository for the full requirement.

Restore point failures

If a whpg-dr create-restore-point command is interrupted, it may leave a lock file at:

$WHPG_DR_HOME/state/my_cluster_backup.lock

If subsequent runs fail with a lock error and you're certain no other whpg-dr process is running, remove the lock file:

whpg-dr clean-lock-file my_cluster

Restore failures

WAL replay did not complete

Symptom:

WARNING: WAL replay did not complete. Data directories for 1 coordinator and N segments
were modified, but the restore-last state file has NOT been updated.
Error: WAL replay failed: segments are not running and not in expected recovery state

Cause: One or more segments failed to replay WAL to the target restore point. A WAL file may be missing from the storage repository, or there's a network or permission issue accessing storage.

Resolution:

  1. Check the segment startup log in $HOME/gpAdminLogs. Look for pg_ctl start failures and the last line before the failure.

  2. Confirm all WAL files between the current restore state and the target are present in storage. For S3, list the WAL objects for the failing segment:

    aws s3 ls s3://my-bucket/whpg/my_cluster/my_cluster-seg0/wals/ --recursive

    WAL file names are hex-sequential (for example 000000010000000000000019). A gap in the sequence indicates a missing file.

  3. If the repository is intact, re-run the restore with --resume-failed:

    whpg-dr restore my_cluster --resume-failed

WAL file is missing during S3 restore

This issue affects whpg-dr 1.1.0 with S3 storage.

Symptom:

The restore fails at backup verification, even though the WAL files exist in the bucket:

Verifying backup 20260825T082845_base_backup integrity...
Error: backup verification failed
  ↳ check-backup completed with 4 errors:
  - w9-2-seg0: ERROR: This backup has been marked as FAILED due to the following reason: At least one WAL file is missing. The first missing WAL file is 000000010000000F0000000E

Later attempts fail with:

ERROR: The validity of a failed backup cannot be checked

Cause: The backup configuration enables WAL compression with barman_options.compression, but the restore YAML doesn't set a matching parameter. The DR site can't find the compressed WAL files and records the backup as status=FAILED in the backup.info file of each server in the bucket. The backup data is intact, but every retry fails until you reset the status.

Resolution:

Warning

Don't retry the restore until the compression is set in the restore YAML and you've re-run configure restore. Every failed attempt marks the backup as FAILED again.

  1. In the restore YAML, set the same barman_options.compression value as in the backup configuration:

    barman_options:
  compression: gzip   # must match the backup configuration

  2. Re-run configure restore:

    whpg-dr configure restore /home/gpadmin/restore.yaml

  3. If a restore attempt has already failed, the backup is still marked as failed in the bucket. Clear the failed status depending on your situation:

    If the primary cluster is available: run the integrity check on the primary cluster. The primary cluster still has the backup recorded as DONE, and the check uploads that status to the bucket:

    whpg-dr check-backup my_cluster --backup-name 20260825T082845_base_backup

    If the primary cluster isn't available: edit the status by hand. Only edit the status if the error= line in backup.info reads At least one WAL file is missing. Any other error means the backup actually failed, so don't change it.

    For each server reported as failed:

    1. Find the server's Barman backup ID, a timestamp such as 20260825T082846. List s3://<bucket>/<prefix>/<cluster>/basebackup/<server>/base/ and pick the directory whose backup.info has backup_name=<backup name>, where <backup name> is the name you pass to whpg-dr check-backup with the --backup-name option. The Barman ID has a different format from the backup name, and its timestamp can differ by a second. For example, the ID 20260825T082846 can belong to the backup named 20260825T082845_base_backup.
    2. Download s3://<bucket>/<prefix>/<cluster>/basebackup/<server>/base/<backup_id>/backup.info from the bucket.
    3. In the file, change status=FAILED to status=DONE and change the error= line to error=None. Leave all other lines unchanged.
    4. Upload the file back to the same location.

    After you've updated all servers, run the integrity check on the DR coordinator to confirm the backup passes:

    whpg-dr check-backup my_cluster --backup-name 20260825T082845_base_backup

  4. Run the restore again.

Could this page be better? Report a problem or suggest an addition!