This page shows you how to protect your data by configuring AlloyDB Omni to work with pgBackRest, an open-source database backup server.
For Kubernetes-based information, see Back up and restore in Kubernetes.
For more information about pgBackRest, see its user guide.
Before you begin
Before configuring AlloyDB Omni to work with pgBackRest, complete the following prerequisites:
- Install and run AlloyDB Omni on a server that you control.
- Set up persistent data storage for your AlloyDB Omni instance.
Basic configuration with local backups
The steps in this section guide you through a basic setup of pgBackRest. Basic configurations don't enable backups on a schedule. For more information, see Set up scheduled backups.
Because pgBackRest is a flexible third-party product compatible with AlloyDB Omni, you can modify any of these steps as appropriate for your own needs and preferences.
Mount a backup volume to your container
For organization purposes, we strongly recommend that you create and mount a separate volume for pgBackRest backups. To set up a new volume, complete the following steps:
If your container is running, stop and remove it.
Docker
docker stop CONTAINER_NAME
docker rm CONTAINER_NAME
Docker
docker stop CONTAINER_NAME
docker rm CONTAINER_NAME
Podman
podman stop CONTAINER_NAME
podman rm CONTAINER_NAME
Podman
podman stop CONTAINER_NAME
podman rm CONTAINER_NAME
Replace the following variable:
CONTAINER_NAME
: Name you used for your container.
Create a backup directory in your container's file system.
Docker
mkdir -p BACKUP_DIR
Docker
mkdir -p BACKUP_DIR
Podman
mkdir -p BACKUP_DIR
Podman
mkdir -p BACKUP_DIR
Replace the following variable:
BACKUP_DIR
: Host directory that your backups will be stored in.
Recreate your container with the new backups volume.
Docker
docker run -d --name CONTAINER_NAME \ -e POSTGRES_PASSWORD=NEW_PASSWORD \ -v DATA_DIR:/var/lib/postgresql/data \ -v BACKUP_DIR:/var/lib/postgresql/backups \ -p HOST_PORT:5432 \ --restart=always \ google/alloydbomni:latest
Docker
docker run -d --name CONTAINER_NAME \ -e POSTGRES_PASSWORD=NEW_PASSWORD \ -v DATA_DIR:/var/lib/postgresql/data \ -v BACKUP_DIR:/var/lib/postgresql/backups \ -p HOST_PORT:5432 \ --restart=always \ google/alloydbomni:latest
Podman
podman run -d --name CONTAINER_NAME \ -e POSTGRES_PASSWORD=NEW_PASSWORD \ -v DATA_DIR:/var/lib/postgresql/data \ -v BACKUP_DIR:/var/lib/postgresql/backups \ -p HOST_PORT:5432 \ --restart=always \ docker.io/google/alloydbomni:latest
Podman
podman run -d --name CONTAINER_NAME \ -e POSTGRES_PASSWORD=NEW_PASSWORD \ -v DATA_DIR:/var/lib/postgresql/data \ -v BACKUP_DIR:/var/lib/postgresql/backups \ -p HOST_PORT:5432 \ --restart=always \ docker.io/google/alloydbomni:latest
Replace the following variables:
CONTAINER_NAME
: Name you used for your container.NEW_PASSWORD
: Password assigned to the new container'spostgres
user after its creation. Note thatNEW_PASSWORD
will only set a new password whenDATA_DIR
is a new location.DATA_DIR
: Host directory path that your data is stored in.DATA_DIR
: Host directory path that your data is stored in.BACKUP_DIR
: Host directory that your backups will be stored in.HOST_PORT
: TCP port on the host machine that the container should publish its own port5432
to. To use the PostgreSQL default port on the host machine as well, specify5432
.
Make the backups directory, in your container, readable by
postgres
.Docker
docker exec CONTAINER_NAME chown -R postgres:postgres /var/lib/postgresql/backups
Docker
docker exec CONTAINER_NAME chown -R postgres:postgres /var/lib/postgresql/backups
Podman
podman exec CONTAINER_NAME chown -R postgres:postgres /var/lib/postgresql/backups
Podman
podman exec CONTAINER_NAME chown -R postgres:postgres /var/lib/postgresql/backups
Replace the following variable:
CONTAINER_NAME
: Name you used for your container.
Configure local backups
pgBackRest creates subdirectories for each backup taken and includes a plain-text manifest file.
pgBackRest uses the term stanza to refer to the configuration for a PostgreSQL database cluster. The stanza name is arbitrary and doesn't need to match the host server, PostgreSQL cluster, or database name. The pgBackRest documentation suggests naming the stanza after the cluster's function.
The repository is where backups are written. pgBackRest supports writing to more
than one repository in a given stanza. Most configuration parameters related to
repositories are indexed with a numeric value. For example, repo1-
. Parameters
related to PostgreSQL clusters are also, independently, indexed. For example,
pg1-
.
pgBackRest leverages a configuration file, called pgbackrest.conf
, to hold
global and stanza-specific parameters.
To build and initialize a configuration file for backing up your
AlloyDB Omni cluster, create the pgbackrest.conf
file in the
host-side backups directory you created in
Mount a backup volume to your container.
[global]
# Paths (all mandatory):
repo1-path=/var/lib/postgresql/backups
spool-path=/var/lib/postgresql/backups/spool
lock-path=/var/lib/postgresql/backups
# Retention details:
repo1-retention-full=3
repo1-retention-full-type=count
repo1-retention-diff=16
# Force a checkpoint to start backup immediately:
start-fast=y
# Logging parameters:
log-path=/var/lib/postgresql/backups
log-level-console=info
log-level-file=info
# Recommended ZSTD compression:
compress-type=zst
# Other performance parameters:
archive-async=y
archive-push-queue-max=1024MB
archive-get-queue-max=256MB
archive-missing-retry=y
[global:archive-push]
process-max=2
[global:archive-get]
process-max=2
[omni]
pg1-user=postgres
pg1-socket-path=/tmp
pg1-path=/var/lib/postgresql/data
Some parameters are mandatory, but others can be adjusted to meet your specific requirements if needed, such as the following:
repo1-path
: Directory location where the backups are written to. We recommend a location visible to both the host server and the container.log-path
: Directory location where the log files are written to. If you want to write the log files to a separate location, not mixed with the backups themselves, adjust this parameter.repo1-retention-full
: Number of full backups to retain.repo1-retention-full-type
: Whether the retention is measured by count or time period (days).repo1-retention-diff
: Number of differential backups to retain.
Other non-critical, but recommended, parameter settings that are compatible with AlloyDB Omni in the configuration file include the following:
log-level-console
: Level of logging written to the screen (STDOUT) when running pgBackup commands. You can adjust this to meet your needs in the configuration file, or override this value with the--log-level-console
command line argument. The default iswarn
.start-fast
: Forces a checkpoint to start the backups quickly. The default isn
.archive-async
: Push WAL segment files asynchronously for performance. The default isn
.process-max
: Maximum number of processes to use for compression and transfer. This is typically set tomax_cpu/4
on a primary ormax_cpu/2
on a standby cluster. The default is1
.compress-type
: Compression algorithm to use. The default isgz
.
Many other pgBackRest configuration parameters exist and can be adjusted. This documentation covers only the parameters mandatory for the default AlloyDB Omni configuration and some recommended parameter settings. For the full list of configuration parameters, refer to the pgBackRest Configuration Reference online documentation.
After configuring pgBackRest, the target repositories where backups are written to must be initialized by creating the stanza, which uses the parameters as set in the configuration file.
To configure your database for backups, complete the following steps:
Create the stanza using the
stanza-create
command.Docker
docker exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni stanza-create
Docker
docker exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni stanza-create
Podman
podman exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni stanza-create
Podman
podman exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni stanza-create
Replace the following variables:
CONTAINER_NAME
: Name you used for your container.
Configure your database for backups.
Docker
docker exec -u postgres CONTAINER_NAME psql \ -c "ALTER SYSTEM SET archive_command='pgbackrest --config-path=/var/lib/postgresql/backups --stanza=omni archive-push %p';" \ -c "ALTER SYSTEM SET archive_mode=on;" \ -c "ALTER SYSTEM SET max_wal_senders=10;" \ -c "ALTER SYSTEM SET wal_level=replica;"
Docker
docker exec -u postgres CONTAINER_NAME psql \ -c "ALTER SYSTEM SET archive_command='pgbackrest --config-path=/var/lib/postgresql/backups --stanza=omni archive-push %p';" \ -c "ALTER SYSTEM SET archive_mode=on;" \ -c "ALTER SYSTEM SET max_wal_senders=10;" \ -c "ALTER SYSTEM SET wal_level=replica;"
Podman
podman exec -u postgres CONTAINER_NAME psql \ -c "ALTER SYSTEM SET archive_command='pgbackrest --config-path=/var/lib/postgresql/backups --stanza=omni archive-push %p';" \ -c "ALTER SYSTEM SET archive_mode=on;" \ -c "ALTER SYSTEM SET max_wal_senders=10;" \ -c "ALTER SYSTEM SET wal_level=replica;"
Podman
podman exec -u postgres CONTAINER_NAME psql \ -c "ALTER SYSTEM SET archive_command='pgbackrest --config-path=/var/lib/postgresql/backups --stanza=omni archive-push %p';" \ -c "ALTER SYSTEM SET archive_mode=on;" \ -c "ALTER SYSTEM SET max_wal_senders=10;" \ -c "ALTER SYSTEM SET wal_level=replica;"
Replace the following variables:
CONTAINER_NAME
: Name you used for your container.
Restart your container.
Docker
docker restart CONTAINER_NAME
Docker
docker restart CONTAINER_NAME
Podman
podman restart CONTAINER_NAME
Podman
podman restart CONTAINER_NAME
Replace the following variables:
CONTAINER_NAME
: Name you used for your container.
Validate the backup configuration.
Docker
docker exec -u postgres CONTAINER_NAME psql \ -c "SELECT name, setting FROM pg_catalog.pg_settings WHERE name IN ('archive_command', 'archive_mode', 'max_wal_senders', 'wal_level') ORDER BY name"
Docker
docker exec -u postgres CONTAINER_NAME psql \ -c "SELECT name, setting FROM pg_catalog.pg_settings WHERE name IN ('archive_command', 'archive_mode', 'max_wal_senders', 'wal_level') ORDER BY name"
Podman
podman exec -u postgres CONTAINER_NAME psql \ -c "SELECT name, setting FROM pg_catalog.pg_settings WHERE name IN ('archive_command', 'archive_mode', 'max_wal_senders', 'wal_level') ORDER BY name"
Podman
podman exec -u postgres CONTAINER_NAME psql \ -c "SELECT name, setting FROM pg_catalog.pg_settings WHERE name IN ('archive_command', 'archive_mode', 'max_wal_senders', 'wal_level') ORDER BY name"
Replace the following variables:
CONTAINER_NAME
: Name you used for your container.
Perform a pgBackRest check.
Docker
docker exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ check
Docker
docker exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ check
Podman
podman exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ check
Podman
podman exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ check
Replace the following variables:
CONTAINER_NAME
: Name you used for your container.
Perform a backup
Perform a full backup.
Docker
docker exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ --type=full \ backup
Docker
docker exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ --type=full \ backup
Podman
podman exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ --type=full \ backup
Podman
podman exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ --type=full \ backup
Replace the following variables:
CONTAINER_NAME
: Name you used for your container.
Perform a differential backup.
Docker
docker exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ --type=diff \ backup
Docker
docker exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ --type=diff \ backup
Podman
podman exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ --type=diff \ backup
Podman
podman exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ --type=diff \ backup
Replace the following variables:
CONTAINER_NAME
: Name you used for your container.
Report backups.
Docker
docker exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ info
Docker
docker exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ info
Podman
podman exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ info
Podman
podman exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ info
Replace the following variables:
CONTAINER_NAME
: Name you used for your container.
Set up scheduled backups
To set up scheduled backups, create a cron job that runs the pgbackrest backup
command as often as needed. For more information, see
Schedule a backup.
Custom configuration and remote backups
After you have a basic configuration working, you can tune your configuration file to suit your needs and preferences using the options documented in the pgBackRest configuration reference.
This includes specifying additional backup repositories located on remote machines, or in the cloud. If you define multiple repositories, then pgBackRest simultaneously writes to them all as its default backup action.
For example, pgBackRest supports using a Cloud Storage bucket as a backup repository, with a number of related configuration options. The following section demonstrates one way to use these options.
An example configuration using Cloud Storage
The steps in this section build on the configuration file introduced in Basic configuration with local backups. These modifications to that file define a second backup repository in a Cloud Storage bucket, accessed through Identity and Access Management (IAM).
The automatic authentication style in this example requires an AlloyDB Omni cluster to run on a Compute Engine VM instance. If you don't run AlloyDB Omni on a Compute Engine VM instance, then you can still back up to a Cloud Storage bucket by using another authentication method, such a Google Cloud service account key saved to the local file system.
To extend the earlier configuration file to define a Cloud Storage-based pgBackRest repository, follow these steps:
Configure the bucket permissions to allow the service account attached to your VM instance to write to the bucket. This requires the Storage Object User IAM role set on that service account.
Add these lines to the
[global]
section of yourpgbackrest.conf
file:# Cloud Storage access details: repo2-type=gcs repo2-gcs-key-type=auto repo2-storage-verify-tls=n # Cloud Storage bucket and path details: repo2-gcs-bucket=BUCKET_NAME repo2-path=PATH_IN_BUCKET # Cloud Storage backup retention parameters: repo2-retention-full=8 repo2-retention-full-type=count
Replace the following variable:
BUCKET_NAME
: Name of the Cloud Storage bucket that you want pgBackRest to store backups to.PATH_IN_BUCKET
: Directory path in the Cloud Storage bucket that you want pgBackRest to store backups in.
Initialize the cloud-based backup location using the
pgbackrest stanza-create
command:Docker
docker exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ stanza-create
Docker
docker exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ stanza-create
Podman
podman exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ stanza-create
Podman
podman exec -u postgres CONTAINER_NAME pgbackrest \ --config-path=/var/lib/postgresql/backups \ --stanza=omni \ stanza-create
Replace the following variable:
CONTAINER_NAME
: Name you used for your container.
After you initialize the backup repository in your Cloud Storage bucket
using the pgbackrest stanza-create
command, pgBackRest may back up to two
locations:
Location in the local file system, defined elsewhere in the configuration file as
repo1-path
. This is the default location when--repo
is not specified in thepgbackrest
command.Cloud Storage bucket, defined using the
repo2-gcs-bucket
configuration directives set up by this example. To use the Cloud Storage bucket, specify--repo=2
in thepgbackrest
command.