Move an instance

This page describes moving an instance in Spanner.

You can move your Spanner instance from any instance configuration to any other instance configuration, including between regional and multi-region configurations. Moving your instance doesn't cause downtime, and Spanner continues to provide the usual transaction guarantees, including strong consistency, during the move.

You can also move your instance from its source instance configuration to a custom instance configuration (for example, a nam3 base configuration with a us-west2 read-only replica). Because you can't update the topology of existing instance configurations, you need to create a new custom instance configuration with the topology you want first. After creating the new custom instance configuration, you can move your instance from the source instance configuration to the new custom instance configuration.

Why move your Spanner instance?

Benefits of moving your instance include:

• Increase availability: Obtain 99.999% availability with zero downtime after performing a regional to multi-region move.
• Reduce latency: Reduce latency and increase geographic coverage with additional read-only replicas through a regional to multi-region or multi-region to multi-region move.
• Reduce cost: Reduce hourly costs by moving from a multi-region configuration to a regional configuration.
• Colocate database: Colocate the Spanner database with the client application by moving the instance to a more optimized location.

Pricing

When moving an instance, both the source and destination instance configurations are subject to hourly compute and storage charges. Once the move is complete, you are billed for the instance storage at the destination configuration.

If you are moving your instance to a new regional or multi-region instance configuration, you might be subject to outbound data transfer charges. For more information, see Spanner Pricing.

Limitations

• To move your instance, it must have a minimum of 1 node (1000 processing units).
• You cannot move your instance across projects and Google Cloud accounts.
• You cannot move a Spanner free trial instance. You can move the instance after upgrading to a paid instance.
• If you have active requests using a regional service endpoint on any of the instance resources, the instance move impacts all the requests that are using the regional endpoint because regional enforcement blocks access to cross region instances. Requests that use a global endpoint are unaffected.
• Spanner backups are specific to an instance configuration and are not included when moving an instance. For more information, see Backups.
• The following APIs are disabled during an instance move:
  • InstanceAdmin.DeleteInstance
  • InstanceAdmin.UpdateInstance
  • DatabaseAdmin.CreateDatabase
  • DatabaseAdmin.UpdateDatabaseDdl (Disabled if default_leader is specified in the request.)
  • DatabaseAdmin.RestoreDatabase
  • DatabaseAdmin.CreateBackup
  • DatabaseAdmin.CopyBackup
• You cannot move instances that contain any CMEK-enabled databases.
• If a database has a modified default leader, the selection is preserved if it names a read-write region in the destination instance configuration, and that configuration is multi-region. If the destination configuration is regional, or doesn't include the named read-write region, the default leader selection is cleared.
• Moving an instance changes the instance configuration attribute of your instance. If you manage your Spanner resources through automation, make sure to prepare and address any inconsistencies that might arise.
  • For example, if you use Terraform to manage your Spanner instances and databases, and you enable terraform apply --auto-approve to keep your resources in sync, all instances and child resources are deleted when we move the instance. Update the configuration accordingly to avoid deletion and data loss. See Terraform Apply Options for more information about the apply command.
• While the instance is being moved, the Spanner monitoring metrics and charts might show data in both the source and destination instance configurations, or it might only reflect performance in one instance configuration.
• If you've configured the open source Autoscaler tool, then you don't need to disable it. It fails because InstanceAdmin.UpdateInstance (used for node and processing unit changes) is disabled.

• You can't move an instance if the Spanner managed autoscaler feature is enabled on it. To move the instance, you need to disable the managed autoscaler, move the instance, and then re-enable the managed autoscaler.

  Additionally, if you're using autoscaling, you must provision enough nodes for peak CPU usage according to the maximum recommendations noted, and then disable autoscaling before you move the instance.

Performance considerations

When an instance is being moved, it experiences higher read-write latencies and a higher transaction abort rate. The CPU utilization during the move might go up to 100% because the instance move is performed using spare CPU provisioned by the user. However, moving an instance does not cause any downtime. The time it takes to move an instance depends on various factors including the size of the databases, the number of nodes, and the kind of move (e.g., regional to multi-region).

After moving an instance, the performance of the instance varies depending on the details of the instance configuration. For example, multi-region configurations generally have higher write latency and lower read latency than regional configurations.

Backups

When you move an instance, the backups in the source instance are not moved to the new destination configuration automatically. The instance move is aborted if backups exist in the source instance configuration when you start the instance move. It is important that you copy your backups and consider your data recovery plan before moving your instance.

If there are backups in your source instance that you need to keep, we recommend that you copy your backups to the destination instance configuration and another instance with the same instance configuration as the source instance to be moved. This is so that:

• You can copy your backups to the destination instance configuration immediately after the instance move completes.
• If you need to cancel the instance move, then you can also quickly restore your backups from the instance with the same configuration as the source instance configuration.

After you copy your backups to another instance, you must delete any existing backups in the source instance before you can move the instance. Then, once the instance move completes, you have a copy of the backup in the destination configuration already. You can also create a new backup.

To learn more about copying backups and associated costs, see Copy a backup.

How to move an instance

You can move an instance with the Google Cloud console Cloud Shell and the gcloud CLI using gcloud commands.

Prerequisites

Before moving your instance configuration, make sure that you have read the Limitations and Performance considerations sections. Then, follow these steps:

1. Check that you have the spanner.instances.update IAM permission on the source instance.
2. If applicable, move your non-production instances (such as test and staging) before moving your production instances to help assess and understand the performance impact on workloads during an instance move.
3. When you move a Spanner instance, the moving process deletes the instance tags that you created in Data Catalog. To preserve your tags, you need to export your tags before the move and import them after the move. For more information, see Export and import tags.

For best practices, also follow these guidelines:

• Test performance workloads in non-production instances in the destination instance configuration before moving your production instance. Try moving a staging instance that is similar to your production instance to get a sense of how long it'll take to move your production instance.
• Check that there are no hotspots in your databases using the Key Visualizer.
• Review to ensure that you have enough node quota in the destination instance configuration to support the expected peak usage of the instance. For more information, see Spanner Quotas & Limits.
• Make sure that the peak CPU utilization of your instance is below 40% for the instance configuration you moved and the amount of storage per node is less than 1 Tebibyte (TiB).
• Don't make changes to the instance during the move. This includes changing the instance node count, changing database schemas, creating or dropping databases, and creating or deleting backups.

If you move your instance according to these recommendations, then the move typically completes within 24 hours. However, depending on the application workload, the completion time might be longer or shorter.

Move an instance

Google Cloud console

1. Click Activate Cloud Shell at the top of the Google Cloud console.

   A Cloud Shell session opens inside a new frame at the bottom of the Google Cloud console and displays a command-line prompt. It can take a few seconds for the session to initialize.

2. Use the gcloud spanner instances move command to move the instance.

   gcloud spanner instances move INSTANCE-ID \
   --target-config=TARGET_CONFIG
   

   Replace the following:

   • INSTANCE-ID: the permanent identifier for the instance that you want to move.
   • TARGET-CONFIG: a permanent identifier of the instance configuration where you want to move your instance. The new geographic location of your instance. This could be a regional, multi-region or custom instance configuration (for example, nam3, us-central1, or custom-us-c1).

   For example, you want to add a read-only replica us-west2 to the base instance configuration nam3.

   1. Clone the base configuration and add the read-only replica:

        gcloud spanner instance-configs create custom-nam3-us-west2 \
        --clone-config=nam3 --add-replicas=location=us-west2, type=READ_ONLY
        ```
      

   2. Move your instance test-instance from its current instance configuration to this new custom-nam3-us-west2 instance configuration:

      gcloud spanner instances move test-instance --target-config=custom-nam3-us-west2
      

gcloud CLI

Use the gcloud spanner instances move command to move the instance.

gcloud spanner instances move INSTANCE-ID \
--target-config=TARGET_CONFIG

Replace the following:

• INSTANCE-ID: the permanent identifier for the instance that you want to move.
• TARGET_CONFIG: a permanent identifier of the instance configuration where you want to move your instance. The new geographic location of your instance. This could be a regional or multi-region instance configuration (for example, nam3, us-central1, or custom-us-c1).

For example, you want to add a read-only replica us-west2 to the base instance configuration nam3.

1. Clone the base configuration and add the read-only replica:

   gcloud spanner instance-configs create custom-nam3-us-west2 \
   --clone-config=nam3 --add-replicas=location=us-west2, type=READ_ONLY
   

2. Move your instance test-instance from its current instance configuration to this new custom-nam3-us-west2 instance configuration:

   gcloud spanner instances move test-instance --target-config=custom-nam3-us-west2
   

How to monitor instance move and cancellation progress

You can use gcloud spanner operations describe or create a custom Cloud Monitoring dashboard to monitor the progress of an instance move.

View move and cancellation operation progress

To track the progress of an instance move or instance move cancellation operation, use the gcloud spanner operations describe command. This command requires the operation ID of the in progress instance move operation.

1. Get the operation ID for your instance move operation by running:

   gcloud spanner operations list --instance="INSTANCE-ID"
   

   Replace the following:

   • INSTANCE-ID: the permanent identifier for the instance that you want to move.

   The output shows a list of long-running operations, including the instance move operation.

2. Run the gcloud spanner operations describe command to view progress percentage and status:

   gcloud spanner operations describe OPERATION-ID
   

   Replace the following:

   • OPERATION-ID: the operation ID of the instance move operation that you want to check.

Monitor an instance move operation

You can create a custom Cloud Monitoring dashboard to display and monitor metrics during the instance move, a long-running operation with potential service implications.

The Total storage and Total database storage by databases graphs in the dashboard are helpful to monitor the progress of the move. You can see the storage in the source configuration gradually drop while the storage in the destination configuration increases.

Google Cloud console

1. Download the move-instance-dashboard.json file. This file has the information needed to populate a custom dashboard in Monitoring.

2. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Dashboards:

   Go to Dashboards

3. In the Dashboards Overview page, click Create dashboard.
4. In the dashboard toolbar, click the Dashboard settings drop-down. Then select JSON, followed by JSON Editor.
5. In the JSON Editor pane, copy and paste the contents of the move-instance-dashboard.json file you downloaded.
6. To apply your changes to the dashboard, click Apply changes. If you don't want to use this dashboard, navigate back to the Dashboards Overview page.
7. After the dashboard is created, click Add Filter. Then select either project_id or instance_id to monitor the progress of your instance move.

gcloud CLI

1. Download the move-instance-dashboard.json file. This file has the information needed to populate a custom dashboard in Monitoring.

2. To create a dashboard in a project, use the gcloud monitoring dashboards create command:

   gcloud monitoring dashboards create --config-from-file=move-instance-dashboard.json
   

   For more information, see the gcloud monitoring dashboards create reference.

How to cancel an instance move

You can only cancel an instance move that is still in progress. If you want to revert an already completed instance move, you must start a new move.

You can use gcloud spanner operations cancel to cancel instance move operations. The cancellation is not instantaneous and takes roughly the same amount of time as the time that has elapsed since the start of the move. This is because data has to be moved back to source instance configuration.

This command requires the operation ID of the in progress instance move operation.

1. Get the operation ID by running:

   gcloud spanner operations list --type=INSTANCE --instance="INSTANCE-ID"
   --filter="done:False AND metadata.@type:MoveInstanceMetadata
   

   Replace the following:

   • INSTANCE-ID: the permanent identifier for the instance that you want to move.

   The output shows a list of in progress instance move operations.

2. Run the gcloud spanner operations cancel command to cancel the instance move:

   gcloud spanner operations cancel OPERATION-ID
   

   Replace the following:

   • OPERATION-ID: the operation ID of the instance move operation that you want to cancel.

What's next

• Learn more about Spanner's Regional and multi-region configurations.
• Learn more about Google Cloud regions and zones.