For this exercise, I used a local three-node PostgreSQL 16 environment managed by Patroni and etcd, with pgBackRest handling backups and WAL archiving. The goal was to create a known recovery point, introduce data after that point, and then recover PostgreSQL to the earlier timestamp.

The exercise also became a troubleshooting exercise. The restore encountered several real operational issues, including a stale postmaster.pid, a non-empty PostgreSQL data directory, and the interaction between pgBackRest and a Docker-mounted PostgreSQL data volume.

Rather than treating those errors as obstacles to hide, I documented them as part of the recovery process. The final verification showed that the data written before the recovery target remained, the data written after the target disappeared, and PostgreSQL completed recovery on a new timeline.

This article documents the environment, recovery procedure, failures, diagnosis, and verification.

Environment

The recovery exercise was performed locally on Ubuntu 24.04.4 LTS using Docker.

The PostgreSQL environment consisted of three PostgreSQL 16 nodes managed by Patroni, with etcd providing the distributed configuration store. pgBackRest was used for full backups and WAL archiving.

The relevant components were:

• PostgreSQL 16.15

• Patroni

• etcd

• pgBackRest 2.59.2

• Docker

• Ubuntu 24.04.4 LTS

The cluster used the following PostgreSQL nodes:

pg1
pg2
pg3

At the beginning of the recovery exercise, pg1 was the Patroni leader. The other two nodes were configured as replicas, although they were not healthy enough to participate in the recovery procedure at that point.

For this exercise, the PITR operation was performed directly against pg1. Patroni cluster management was paused before the restore so that automated cluster management would not interfere with the recovery operation.

Establishing a Recoverable State

Before testing point-in-time recovery, I first established a known-good backup and verified that WAL archiving was working.

I ran a pgBackRest configuration check against the patroni-dev291 stanza:

docker exec patroni-pg1 pgbackrest --stanza=patroni-dev291 check

The check completed successfully, including verification that a WAL segment had been archived.

I then created a fresh full backup:

docker exec patroni-pg1 pgbackrest \
  --stanza=patroni-dev291 \
  --type=full \
  backup

The backup completed successfully with backup label:

20261006-075738F

The backup started at 2026-10-06 07:57:38 UTC and completed at 2026-10-06 07:58:39 UTC.

At this point, I had both a validated pgBackRest configuration and a known full backup from which recovery could be performed.

Creating the recovery scenario

I created a small test table and inserted a row that represented data I wanted to preserve:

CREATE TABLE IF NOT EXISTS pitr_demo (
    id SERIAL PRIMARY KEY,
    event TEXT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

INSERT INTO pitr_demo (event)
VALUES ('BEFORE_PITR');

The row was created at:

2026-10-06 08:03:06.138954+00

I selected the PITR target as:

2026-10-06 08:06:01.308827+00

After the target time, I inserted another row:

INSERT INTO pitr_demo (event)
VALUES ('AFTER_PITR_TARGET');

This row was created at:

2026-10-06 08:29:28.494015+00

The intention was straightforward: after recovery to the target timestamp, BEFORE_PITR should remain while AFTER_PITR_TARGET should no longer exist.

To make sure the relevant WAL was archived, I also forced a WAL switch:

SELECT pg_switch_wal();

This returned:

0/110023D0

A subsequent pgbackrest info check showed that the archived WAL range had advanced, confirming that WAL containing the later activity had been archived before starting recovery.

Performing the Point-in-Time Recovery

Before starting the restore, I checked the Patroni cluster state. pg1 was the leader on timeline 9, while pg2 and pg3 were not healthy enough to participate in the recovery.

Because the restore was going to modify the PostgreSQL data directory, I paused Patroni cluster management first:

patronictl -c /etc/patroni/patroni.yml pause

The command returned:

Success: cluster management is paused

I then stopped the pg1 container so that PostgreSQL would not be running while the data directory was restored.

The PITR restore was performed with pgBackRest using the previously created full backup and the target timestamp:

docker run --rm \
  --volumes-from patroni-pg1 \
  --entrypoint pgbackrest \
  patroni-dev291-pg1 \
  --stanza=patroni-dev291 \
  --type=time \
  --target="2026-10-06 08:06:01.308827+00" \
  --target-action=promote \
  --force \
  --log-level-console=info \
  restore

The restore completed successfully:

restore command end: completed successfully

The --target-action=promote option was important because the goal was not simply to leave PostgreSQL in recovery. I wanted the recovered instance to exit recovery and become the new primary state at the requested point in time.

After the restore completed, I started the pg1 container. PostgreSQL was not immediately running, so I checked its process state and started PostgreSQL manually. PostgreSQL then performed WAL replay from the restored backup toward the requested recovery target.

Troubleshooting the Recovery

The restore was successful, but getting there was not completely straightforward. Several errors appeared during the recovery process. I treated each one as a diagnostic step rather than simply retrying commands.

1. Using the container name instead of the image name

My first restore attempt used patroni-pg1 as the Docker image name. Docker could not find that image locally and attempted to resolve it as an image reference.

The error was:

Unable to find image 'patroni-pg1:latest' locally

I checked the actual image associated with the PostgreSQL container and found:

patroni-dev291-pg1

I then reran the restore using the correct image name.

2. A stale postmaster.pid

The next restore attempt was rejected by pgBackRest:

ERROR [038]: unable to restore while PostgreSQL is running
HINT: presence of postmaster.pid...

The PostgreSQL container itself was stopped, so I inspected the data directory from a helper container and found a stale postmaster.pid.

There was no PostgreSQL process running. The PID file was left behind from the previous PostgreSQL state.

I removed the stale PID file and attempted the restore again.

3. The PostgreSQL data directory was not empty

pgBackRest then reported:

ERROR [040]: unable to restore to path '/var/lib/postgresql/data' because it contains files
HINT: try using --delta

I inspected the data directory and confirmed that it still contained the existing PostgreSQL cluster.

I initially considered moving the entire data directory aside, but the directory was the root of a Docker volume mount. Attempting to rename it resulted in:

mv: cannot move ... Device or resource busy

The correct approach for this recovery exercise was to let pgBackRest cleanly overwrite the existing contents. I reran the restore with --force, which completed successfully and removed invalid files before restoring the backup.

4. PostgreSQL was not immediately available after the restore

I checked the PostgreSQL process state directly:

docker exec patroni-pg1 sh -c 'pg_ctl -D /var/lib/postgresql/data status'

The result was:

pg_ctl: no server running

This showed that the PITR restore itself had completed, but PostgreSQL was not yet running.

I then started PostgreSQL manually and captured its startup output:

docker exec patroni-pg1 sh -c 'pg_ctl -D /var/lib/postgresql/data -l /tmp/postgres-startup.log start'

The server started successfully. The startup log then showed that PostgreSQL was performing the requested point-in-time recovery, including WAL replay toward the target timestamp.

The recovery log reported:

starting point-in-time recovery to 2026-10-06 08:06:01.308827+00

It then stopped recovery before the later transaction and reported:

recovery stopping before commit of transaction 746

The log also confirmed the timeline change:

selected new timeline ID: 10

Finally, PostgreSQL reported:

archive recovery complete
database system is ready to accept connections

This clarified that the initial connection refusal was not a failed PITR. PostgreSQL simply needed to be started so that the recovery process could run to completion.

Verifying the Recovery

After PostgreSQL completed recovery, I verified both the recovery state and the data that existed after the target timestamp.

First, I checked whether PostgreSQL was still in recovery:

SELECT pg_is_in_recovery();

The result was:

f

This confirmed that PostgreSQL had completed recovery and was accepting normal read-write connections.

I then queried the test table:

SELECT * FROM pitr_demo ORDER BY id;

The result contained only the row created before the PITR target:

 id |    event
----+-------------
  1 | BEFORE_PITR

The AFTER_PITR_TARGET row was no longer present.

This gave me the expected recovery result:

• Data committed before the target timestamp remained.

• Data committed after the target timestamp was removed.

• PostgreSQL was no longer in recovery.

• PostgreSQL had created a new timeline during recovery.

The recovery log explicitly reported:

selected new timeline ID: 10

The cluster had been on timeline 9 before recovery, so the PITR resulted in a new timeline, timeline 10.

This timeline change is an important part of PostgreSQL recovery. The recovered database did not simply continue the old history. It created a new branch of WAL history from the point at which recovery stopped.

Patroni and Cluster Management Considerations

Running PITR in a Patroni-managed PostgreSQL environment requires coordination between the recovery procedure and the cluster manager.

Patroni is designed to manage PostgreSQL availability and cluster state. During a manual recovery operation, allowing Patroni to continue normal cluster management could interfere with the recovery process.

For this exercise, I therefore paused Patroni before stopping pg1 and restoring its data directory.

The recovery was performed against pg1 rather than attempting to restore all three nodes simultaneously. The objective was to establish and verify the recovered PostgreSQL state first.

The other nodes were already in a failed replica state before the PITR exercise, so I did not use them as part of the recovery verification. I also did not claim that the entire three-node cluster had been successfully rebuilt as part of this exercise.

That distinction matters. Successfully recovering one PostgreSQL instance to a point in time is different from completing a full Patroni cluster rebuild and resynchronisation.

A production recovery procedure would require additional consideration for the remaining members, cluster membership, replication state, and how the recovered node should be introduced back into the HA topology.

What This Exercise Taught Me

This exercise reinforced several practical lessons about PostgreSQL backup and recovery.

Backups are only useful when recovery is understood

Creating a backup is not the same as having a reliable recovery strategy. The recovery test exposed assumptions that would have been easy to miss without actually restoring the database.

Running pgbackrest check before the exercise also provided confidence that the repository and WAL archiving were functioning before the recovery test began.

WAL archiving is central to PITR

The full backup provided the base state, while archived WAL allowed PostgreSQL to replay changes forward until the requested recovery target.

For a PITR workflow, both pieces matter. A valid base backup without the required WAL history is not sufficient to recover to an arbitrary point in time.

Recovery changes the timeline

The recovery created timeline 10 from the previous timeline 9.

This is an important PostgreSQL concept because timeline history records branching in WAL history. After PITR, the recovered database has a new history from the recovery point onward.

Recovery errors contain useful information

The failed restore attempts were not simply obstacles. Each error pointed toward a specific condition that needed to be understood:

• An incorrect Docker image reference meant I was invoking the wrong image.

• A stale postmaster.pid indicated leftover PostgreSQL state.

• The non-empty data directory showed that pgBackRest was protecting the existing cluster from an accidental overwrite.

• The Docker volume mount explained why the data directory could not simply be renamed.

• The initial connection refusal required checking PostgreSQL's actual process and recovery state instead of assuming the restore had failed.

HA management and recovery are different concerns

Patroni manages PostgreSQL availability and cluster state, while pgBackRest handles backup and restore operations.

A recovery procedure needs to account for both systems. Pausing cluster management before modifying the PostgreSQL data directory prevented Patroni from attempting to manage the instance while the restore was being performed.

Most importantly, the exercise demonstrated that a successful recovery should be verified with evidence, not inferred from a successful restore command. The final checks showed the expected data state, confirmed that PostgreSQL had exited recovery, and demonstrated the creation of a new WAL timeline.

Recovery Failures and Resolutions

The important lesson from these failures was that recovery troubleshooting should be evidence-driven. Each error was investigated to determine the actual system state before choosing the next action.

Conclusion

This PITR exercise gave me a much better understanding of PostgreSQL recovery than simply reading about the process.

I started with a validated pgBackRest configuration and a fresh full backup, created a known recovery scenario, archived the required WAL, and recovered PostgreSQL to a specific timestamp.

The recovery also exposed practical operational issues around stale PostgreSQL state, Docker volume mounts, restore safety, WAL replay, and Patroni cluster management.

Most importantly, I verified the result rather than assuming the restore had worked. The data created before the recovery target remained, the data created after the target disappeared, PostgreSQL exited recovery, and the recovery created timeline 10 from the previous timeline 9.

The next step for me is to continue building on this work by studying the PostgreSQL backup and recovery ecosystem more deeply and contributing where I can identify a genuine documentation or technical gap.

The lab itself is not an upstream contribution. It is the practical foundation for that next stage.

Project Repository

The complete lab configuration, recovery documentation, and supporting files are available on GitHub:

https://github.com/david-onwuka/DEV-291-patroni-postgresql-disaster-recovery

The repository contains the PostgreSQL 16, Patroni, etcd, and pgBackRest lab used for this recovery exercise.

The lab itself is not an upstream contribution. It is a practical recovery exercise that I used to strengthen my understanding of PostgreSQL high availability, backup, WAL archiving, point-in-time recovery, and disaster recovery procedures.