Replicate volumes asynchronously

This page describes how to set up and perform asynchronous replication of Google Distributed Cloud (GDC) air-gapped block storage volumes.

Asynchronous replication is used to replicate data from one GDC zone to another. The replicated data can be utilized for failover if the source zone data becomes unavailable. Note that once a failover is created, the original volume cannot be replicated to the same destination volume. Instead, a new replication relationship must be created.

Before you begin

To enable asynchronous block replication between two zones, your Infrastructure Operator (IO) must first establish the necessary storage infrastructure by peering the relevant storage clusters from each zone. Next, they need to peer the storage virtual machine associated with the organization in which the block storage is provisioned.

After, ensure that you have the app-volume-replication-admin-global role to administer the VolumeReplicationRelationship resource. If the global API is not available, the volume-replication-admin role can be used to directly modify the zonal VolumeReplicationRelationshipReplica resource.

Set up replication

The VolumeReplicationRelationship Custom Resource (CR) services the asynchronous block replication API. This CR exists in the global management API. In order to enable replication for a given block device, a VolumeReplicationRelationship CR needs to be created on the global management API:

apiVersion: storage.global.gdc.goog/v1
kind: VolumeReplicationRelationship
metadata:
  name: my-pvc-repl
  namespace: my-project
spec:
  source:
    pvc:
      clusterRef: my-pvc-cluster
      pvcRef: my-block-pvc
    zoneRef: xx-xxxx-zone1
 destination:
    pvc:
      clusterRef: my-pvc-cluster
    zoneRef: xx-xxxx-zone2

This example assumes that a project named my-project is created in an org named my-org and that a PVC named my-block-pvc has already been provisioned. The clusterRef is the name of the cluster on which the PVC exists.

The source and destination fields of the specification indicate where the data is being replicated from and to respectively. In this example, the data is replicated from xx-xxxx-zone1 to xx-xxxx-zone2.

Note: The asynchronous block replication service automatically creates a PVC in the destination zone for writing the replicated data. By default, this PVC has the same name as the source PVC, however, the VolumeOverrideName field can be used to specify the name of the created PVC. Until a VolumeReplicationFailover is created, though, the destination PVC remains read-only and won't be mountable by any workloads.

Check the status of the replication relationship by retrieving the VolumeReplicationRelationship CR from the global API. Reference the following example. Note that the output has been truncated for simplification:

apiVersion: storage.global.gdc.goog/v1
kind: VolumeReplicationRelationship
metadata:
  name: my-pvc-repl
  namespace: my-project
spec:
  destination:
    pvc:
      clusterRef: my-pvc-cluster
    zoneRef: xx-xxxx-zone2
  source:
    pvc:
      clusterRef: my-pvc-cluster
      pvcRef: my-block-pvc
    zoneRef: xx-xxxx-zone1
status:
  zones:
  - name: xx-xxxx-zone1
    replicaStatus:
      message: SnapMirror relationship has been established. Please check the destination
        zone for relationship state
      replicationID: a096621e-f062-11ef-ad24-00a0b89f23fb
      state: Established
  - name: xx-xxxx-zone2
    replicaStatus:
      exportedSnapshotName: snapmirror.c34f8845-e8c0-11ef-ad24-00a0b89f23fb_2150007868.2025-02-21_150000
      message: SnapMirror relationship has been successfully established
      replicationID: a096621e-f062-11ef-ad24-00a0b89f23fb
      state: Idle

Create failover

If the source zone is unavailable for any reason, a VolumeFailover CR can be created in the organization's management plane of the destination zone. For a v2 organization, this would be the management API server. For a v1 organization, this would be the organization admin cluster. For example, if a VolumeReplicationRelationship was created which specifies xx-xxxx-zone2 as the destination zone and PVC exists in the my-org organization, then the VolumeFailover CR is created in the my-org management plane in xx-xxxx-zone2. This breaks the replication relationship between the two zones, and allows the PVC in the destination zone to be mounted by a workload:

apiVersion: storage.gdc.goog/v1
kind: VolumeFailover
metadata:
  name: my-pvc-failover
  namespace: my-project
spec:
  volumeReplicationRelationshipRef: my-pvc-repl

After, a successful failover is reflected in the status of the CR:

apiVersion: storage.gdc.goog/v1
kind: VolumeFailover
metadata:
  name: my-pvc-failover
  namespace: my-project
spec:
  volumeReplicationRelationshipRef: my-pvc-repl
status:
    state: Completed

After the failover is created, the my-pvc-repl VolumeReplicationRelationship transitions to a Broken Off state. The PVC in xx-xxxx-zone2 is now mountable.

At this point, the VolumeReplicationRelationship will look similar to the following example. Again, this output has been truncated for simplification:

apiVersion: storage.global.gdc.goog/v1
kind: VolumeReplicationRelationship
metadata:
  name: my-pvc-repl
  namespace: my-project
spec:
  destination:
    pvc:
      clusterRef: my-pvc-cluster
    zoneRef: xx-xxxx-zone2
  source:
    pvc:
      clusterRef: my-pvc-cluster
      pvcRef: my-block-pvc
    zoneRef: xx-xxxx-zone1
status:
  zones:
  - name: xx-xxxx-zone1
    replicaStatus:
      message: SnapMirror relationship has been broken off
      replicationID: a096621e-f062-11ef-ad24-00a0b89f23fb
      state: Broken Off
  - name: xx-xxxx-zone2
    replicaStatus:
      exportedSnapshotName: snapmirror.c34f8845-e8c0-11ef-ad24-00a0b89f23fb_2150007868.2025-02-21_150000
      message: SnapMirror relationship has been broken off
      replicationID: a096621e-f062-11ef-ad24-00a0b89f23fb
      state: Broken Off

Now, it's safe to delete the VolumeReplicationRelationship as it's the only action remaining that can be done on this CR.

Resize volumes

If at any point the source volume is resized, the corresponding volume in the destination zone, which is created on behalf of the user when a VolumeReplicatioRelationship is created, should also be resized to match.

Replicate virtual machine disks

VolumeReplicationRelationship also services the asynchronous virtual machine disk (VM disk) replication API. The source disk being replicated is called the primary disk. The destination disk that is being replicated to is called the secondary disk. Starting asynchronous replication on a primary disk will automatically create the secondary disk.

Request permissions and access

To replicate VM disks, you must have the Project VirtualMachine Admin role. Follow the steps to verify that you have the Project VirtualMachine Admin (project-vm-admin) role in the namespace of the project where the VM disk resides.

For VM operations using the gdcloud CLI, request your Project IAM Admin to assign you both the Project VirtualMachine Admin role and the Project Viewer (project-viewer) role.

Start asynchronous replication

Start asynchronous replication on a VM disk using gdcloud or kubectl.

gdcloud

gdcloud compute disks start-async-replication PRIMARY_DISK_NAME \
  --project PROJECT --zone PRIMARY_ZONE \
  --secondary-disk SECONDARY_DISK_NAME --secondary-zone SECONDARY_ZONE

Replace the following:

Variable	Definition
PRIMARY_DISK_NAME	The name of the source disk being replicated.
PROJECT	The GDC project of the primary disk.
PRIMARY_ZONE	The zone where the primary disk resides.
SECONDARY_DISK_NAME	The name of the destination disk to replicate to.
SECONDARY_ZONE	The zone where the secondary disk must reside.

API

kubectl --kubeconfig GLOBAL_MANAGEMENT_API \
  apply -f - <<EOF
apiVersion: storage.global.gdc.goog/v1
kind: VolumeReplicationRelationship
metadata:
  name: VRR_NAME
  namespace: PROJECT
spec:
  source:
    virtualMachineDisk:
      virtualMachineDiskRef: PRIMARY_DISK_NAME
    zoneRef: PRIMARY_ZONE
  destination:
    volumeOverrideName: SECONDARY_DISK_NAME
    zoneRef: SECONDARY_ZONE
EOF

Replace the following:

Variable	Definition
GLOBAL_MANAGEMENT_API	The kubeconfig file for the global management API server.
VRR_NAME	The name of the volume replication relationship. The same name must be used when stopping asynchronous replication.
PROJECT	The GDC project of the primary disk.
PRIMARY_DISK_NAME	The name of the source disk being replicated.
PRIMARY_ZONE	The zone where the primary disk resides.
SECONDARY_DISK_NAME	The name of the destination disk to replicate to.
SECONDARY_ZONE	The zone where the secondary disk must reside.

List asynchronous replication relationships

List asynchronous replication relationships in a project using kubectl.

kubectl --kubeconfig GLOBAL_MANAGEMENT_API get volumereplicationrelationships -n my-project

Replace the following:

• PROJECT: The GDC project of the primary disk.
• GLOBAL_MANAGEMENT_API: The kubeconfig file for the global management API server.

The output will look like this:

NAME       AGE     SOURCE ZONE   SOURCE PVC   SOURCE PVC CLUSTER   SOURCE VM DISK      DEST. ZONE   DEST. PVC CLUSTER   DEST. VOLUME OVERRIDE     STATE
my-vrr     3m21s   zone1                                           my-vm-boot-disk     zone2                            my-vm-boot-disk-replica
test-vrr   7s      zone1                                           test-vm-boot-disk   zone2

Stop asynchronous replication

Stop asynchronous replication on a primary VM disk using gdcloud or kubectl.

gdcloud

gdcloud compute disks stop-async-replication PRIMARY_DISK_NAME \
  --project PROJECT --zone PRIMARY_ZONE

Replace the following:

Variable	Definition
PRIMARY_DISK_NAME	The name of the source disk being replicated.
PROJECT	The GDC project of the primary disk.
PRIMARY_ZONE	The zone where the primary disk resides.

API

1. Find the volume replication relationships corresponding to the primary VM disk.

   kubectl --kubeconfig GLOBAL_MANAGEMENT_API get volumereplicationrelationships \
     -n PROJECT -o json | \
     jq -r '.items[] | select(.spec.source.virtualMachineDisk.virtualMachineDiskRef == "PRIMARY_DISK_NAME"
     and .spec.source.zoneRef == "PRIMARY_ZONE") | .metadata.name'
   

2. Delete each of the volume replication relationships listed in the previous step. Replace VRR_NAMES with the names of the volume replication relationships.

   kubectl --kubeconfig GLOBAL_MANAGEMENT_API delete volumereplicationrelationships \
     -n PROJECT VRR_NAMES
   

   Replace the following:

   Variable	Definition
   GLOBAL_MANAGEMENT_API	The kubeconfig file for the global management API server.
   PROJECT	The GDC project of the primary disk.
   PRIMARY_DISK_NAME	The name of the source disk being replicated.
   PRIMARY_ZONE	The zone where the primary disk resides.

If the source zone is unavailable for any reason, create a volume failover to stop replication.

Attach the replicated disk to a VM

While replication is enabled, the secondary disk cannot be attached to a VM. After replication is stopped, you can attach the secondary disk to a newly created VM or to an existing VM.