Creating a Linux application consistent persistent disk snapshot

You can create application consistent snapshots of persistent disks attached to Linux virtual machine (VM) instances. In general, the quality of your persistent disk snapshot depends on how well your applications can recover from snapshots that you create during heavy write workloads. Application consistent snapshots capture the state of application data at the time of backup with all application transactions completed and all pending writes flushed to the disk.

To create snapshots that are application consistent, pause apps or operating system processes that write data to the persistent disk, flush the disk buffers, and sync the file system before you create the snapshot. Depending on your application, these and other steps might be required to ensure that all application transactions are complete and captured in the backup.

To create an application consistent snapshot of your persistent disks, use the following process:

  1. To prepare the guest environment for application consistency, create custom shell scripts to run before and after the snapshot is captured
  2. Configure snapshot settings on your VM (virtual machine) instance.
  3. Create a snapshot with the guest-flush option enabled. The guest-flush option starts your pre and post snapshot scripts.

Before you begin

Restrictions

  • Application consistency is guaranteed only by the behavior of your custom pre and post snapshot scripts, not by the snapshot operation itself.
  • When using the guest-flush option in your snapshot creation request, no snapshot is created in the event of a script error or timeout.
  • You can only create application consistent snapshots of persistent disks attached to certain public or custom Linux images. The following images are supported:
    • CentOS 7
    • CentOS 8
    • Debian 9 Stretch
    • Debian 10 Buster
    • RHEL 7
    • RHEL 8
    • SLES 12
    • SLES 15
    • Ubuntu 18.04
    • Ubuntu 20.04

Create pre and post snapshot scripts

Before you proceed, update the guest environment so that you are running the latest software on your linux VM.

To facilitate application consistency, create pre and post snapshot shell scripts to run before and after the snapshot is captured. Use the pre and post scripts for operations such as:

  • Pause apps or operating system processes running on the VM that writes data to the persistent disk.
  • Flush the disk buffers. For example, MySQL has a FLUSH statement. Use whichever tool is available for your app.
  • Sync your file system.

The following code example shows a simple pre snapshot script. Note the leading #! characters. Running fsfreeze -f blocks any running process that tries to access the filesystem, so use this with caution if your application is latency-sensitive.

#!/bin/bash
fsfreeze -f [example-disk-location]

The following code example shows a simple post snapshot script. Note the leading #! characters.

#!/bin/bash
fsfreeze -u [example-disk-location]

You must save your scripts on your VM in the directory /etc/google/snapshots/. The full path of your pre script must be /etc/google/snapshots/pre.sh and the full path of your post script must be /etc/google/snapshots/post.sh.

Referencing specific disks in your scripts

The first argument passed to your pre and post snapshot scripts is a list of disks that are being snapshotted. You can use this argument in your scripts for various checks. For example, if your VM has multiple disks attached but you only specified one disk in your snapshot request, you can check which disk is being snapshotted.

The argument is formatted as follows:

  • SCSI-attached disks: a comma-separated list of <target/lun> pairs.
  • NVME-attached disks: a comma-separated list of <nvme:namespace> pairs.

For example, your boot disk might appear as 1/0 while an additional disk attached to the VM might appear as 2/0.

Edit your guest environment configuration file

Configure your application consistent snapshot settings by updating a specific configuration file on your VM.

  1. Open or create your guest environment configuration file:

    /etc/default/instance_configs.cfg
    
  2. Add the following section to the configuration file:

    [Snapshots]
    enabled = enabled
    timeout_in_seconds = timeout
    

    Replace the values according to the descriptions below:

    Key Value Default Description
    enabled true or false false Whether the application consistent snapshot feature is enabled.
    timeout_in_seconds Integer [0, 300] 60 Number of seconds the pre or post snapshot script can take to finish running before a timeout error. Note that the number of seconds the entire snapshot operation can take to complete before a timeout error is 300 seconds per disk, and this is not configurable.
  3. Save your configuration settings:

    sudo systemctl restart google-guest-agent.service
    

Create a snapshot with guest-flush enabled

Using the Google Cloud Console, the gcloud command-line tool, or the Compute Engine API, create a snapshot with the guest-flush option enabled. This starts running the pre and post snapshot scripts before and after the snapshot is captured.

Console

  1. Go to the Create a new snapshot page.
  2. Enter a snapshot Name.
  3. Optionally, enter a Description of the snapshot.
  4. Under Source disk, select an existing disk from which you want to create a snapshot.
  5. Select a storage location.
  6. Check the Enable Application Consistent Snapshot option.
  7. Click Create.

gcloud

Run the command

gcloud compute disks snapshot DISK_NAME [DISK_NAME ...] --guest-flush

Replace the following:

  • DISK_NAME is the name of one or more persistent disks of which you want to create snapshots.

The gcloud command-line tool waits until the operation returns a status of READY, FAILED, or reaches the maximum timeout and returns the last known details of the snapshot.

API

Make a POST request to the disks.createSnapshot method method with the guestFlush option enabled:

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/disks/DISK_NAME/createSnapshot?guestFlush=true

Replace the following:

  • PROJECT_ID: your project ID.
  • ZONE: the zone where your VM and disk are located.
  • DISK_NAME: the name of the persistent disk of which you want to create a snapshot.

Create a snapshot schedule with guest-flush enabled

Use scheduled snapshots to regularly and automatically back up your zonal and regional persistent disks. If you want to schedule application consistent snapshots for your backup, use the --guest-flush option when you create the snapshot schedule so that the pre and post snapshot scripts execute before and after each scheduled snapshot.

For example, after configuring my settings and creating custom scripts, the following command creates hourly application consistent snapshots:

gcloud compute resource-policies create snapshot-schedule SCHEDULE_NAME \
  --description "MY HOURLY SNAPSHOT SCHEDULE" \
  --start-time 22:00 \
  --hourly-schedule 4 \
  --guest-flush

To learn more, see Creating scheduled snapshots of persistent disks.

Troubleshooting

  • Review the logs from your snapshot creation events:

    1. In the Cloud Console, go to the Google Cloud's operations suite Logging  > Logs page:
      Go to the Logs Viewer page
    2. Navigate to Logs Viewer in the left navigation.
    3. In the Filter by label or text search list, select Convert to advanced filter.

    Convert to advanced filter.

    Replace the filter text box with the following text:

    resource.type="gce_disk"
    jsonPayload.event_subtype="compute.disks.createSnapshot"
    
  • If no script is found, no snapshot is created. Make sure you've followed the steps in Create pre and post snapshot scripts.

  • If there is a script error or timeout, no snapshot is created. Review Preparing for consistent snapshots. Note that the maximum timeout period that you can configure in your settings is 300 seconds. Try repeating the entire process using the sample scripts.

  • A persistent disk must be attached to a VM to capture a snapshot of the disk with guest-flush enabled. To learn how to create a Linux VM and attach a persistent disk, see Creating a VM.

What's next