Manage and observe long-running operations

This page describes how to manage the lifecycle of a Spanner long-running operation using the operations REST API.

Long-running operations are method calls that may take a substantial amount of time to complete. Spanner creates long-running operations for several instance, database, and backup actions. An example is the method to restore a database, projects.instances.databases.restore.

When you restore a database, the Spanner service creates a long-running operation to track restore progress. Spanner provides operations APIs that let you check the progress of long-running operations. You can also list and cancel long-running operations, and delete long-running instance operations.

You can check and manage long-running operations with the following:

Manage long-running Spanner operations with the REST API

Manage your Spanner long-running operations using the following REST methods:

Action Long-running database operations Long-running instance operations
Cancel a long-running operation cancel cancel
Delete a long-running operation Unsupported Unsupported
Check the progress of a long-running operation get get
List long-running operations list list

For information about using REST with Spanner, see Getting started with Spanner using REST.

Database operations

The following are long-running database operations.

Check the progress of a long-running database operation

Use projects.instances.databases.operations.get to check the progress of a long-running database operation.

As an example, this is a response from projects.instances.databases.create:

  {
    "name": "projects/<VAR>PROJECT-ID</VAR>/instances/test-instance/databases/example-db/operations/_auto_1492721321097206",
    "metadata": {
      "@type": "type.googleapis.com/google.spanner.admin.database.v1.CreateDatabaseMetadata",
      "database": "projects/<VAR>PROJECT-ID</VAR>/instances/test-instance/databases/example-db"
    }
  }

The name value at the top of the response shows the Spanner service created a long-running database operation named projects/PROJECT-ID/instances/test-instance/databases/example-db/operations/_auto_1492721321097206.

To check the progress of the long-running database operation:

  1. Click projects.instances.databases.operations.get.

  2. For name, enter the long-running database operation name as shown in the response to projects.instances.databases.create or projects.instances.databases.updateDdl. For example:

    projects/PROJECT-ID/instances/test-instance/databases/example-db/operations/OPERATION-ID

    Replace the following:

    • PROJECT-ID with the project ID.
    • OPERATION-ID with the operations ID.

    retrieve the database operation name by listing long-running database operations.

  3. Click Execute. When an operation is done, the done field is set to true.

To get continuous updates, repeatedly invoke the projects.instances.databases.operations.get method until the operation is done. Use a backoff between each request. For example, make a request every 10 seconds.

List long-running database operations

Use projects.instances.databases.operations.list to list long-running database operations.

  1. Click projects.instances.databases.operations.list.
  2. For name, enter:

    projects/PROJECT-ID/instances/test-instance/databases/example-db/operations

  3. Click Execute. The response contains a list of long-running operations.

Cancel a long-running database operation

Use projects.instances.databases.operations.cancel to cancel a long-running database operation.

  1. Click projects.instances.databases.operations.cancel.
  2. For name, enter the long-running database operation name as shown in the long-running database operation response. For example:

    projects/PROJECT-ID/instances/test-instance/databases/example-db/operations/_auto_1492721321097206

    (Use the operation name you received instead of this value.) You can also retrieve the database operation name by listing long-running database operations.

  3. Click Execute.

Schema update operations

The following are long-running schema update operations.

Check the progress of a long-running schema update operation

Console

  1. In the Spanner navigation menu, select the Operations tab. The Operations page shows a list of currently running operations.

  2. Find the schema operation in the list. If it's still running, the progress bar in the End time column shows the percentage of the operation that is complete, as shown in the following image:

Screenshot of progress indicator showing 98%

gcloud

Use gcloud spanner operations describe to check the progress of an operation.

  1. Get the operation ID:

    gcloud spanner operations list --instance=INSTANCE-NAME \
    --database=DATABASE-NAME --type=DATABASE_UPDATE_DDL
    

    Replace the following:

    • INSTANCE-NAME with the Spanner instance name.
    • DATABASE-NAME with the name of the database.
  2. Run gcloud spanner operations describe:

    gcloud spanner operations describe OPERATION-ID\
    --instance=INSTANCE-NAME \
    --database=DATABASE-NAME
    

    Replace the following:

    • OPERATION-ID: The operation ID of the operation that you want to check.
    • INSTANCE-NAME: The Spanner instance name.
    • DATABASE-NAME: The Spanner database name.

    The progress section in the output shows the percentage of the operation that's complete. The output looks similar to the following:

    done: true
    metadata:
    ...
      progress:
      - endTime: '2022-03-01T00:28:06.691403Z'
        progressPercent: 100
        startTime: '2022-03-01T00:28:04.221401Z'
      - endTime: '2022-03-01T00:28:17.624588Z'
        startTime: '2022-03-01T00:28:06.691403Z'
        progressPercent: 100
    ...
    

REST v1

Get the operation ID:

gcloud spanner operations list --instance=INSTANCE-NAME \
--database=DATABASE-NAME --type=DATABASE_UPDATE_DDL

Replace the following:

  • INSTANCE-NAME with the Spanner instance name.
  • DATABASE-NAME with the name of the database.

Before using any of the request data, make the following replacements:

  • project-id: The project ID.
  • instance-id: The instance ID.
  • database-id: The database ID.
  • operation-id: The operation ID.

HTTP method and URL:

GET https://spanner.googleapis.com/v1/projects/project-id/instances/instance-id/databases/database-id/operations/operation-id

To send your request, expand one of these options:

You should receive a JSON response similar to the following:

{
...
    "progress": [
      {
        "progressPercent": 100,
        "startTime": "2023-05-27T00:52:27.366688Z",
        "endTime": "2023-05-27T00:52:30.184845Z"
      },
      {
        "progressPercent": 100,
        "startTime": "2023-05-27T00:52:30.184845Z",
        "endTime": "2023-05-27T00:52:40.750959Z"
      }
    ],
...
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.protobuf.Empty"
  }
}

If the operation takes too long, you can cancel it. For more information, see Cancel a long-running schema update operation.

List long-running schema update operations

gcloud

gcloud spanner operations list --instance=INSTANCE-NAME \
--database=DATABASE-NAME --type=DATABASE_UPDATE_DDL

Replace the following:

  • INSTANCE-NAME with the Spanner instance name.
  • DATABASE-NAME with the name of the database.

The output looks similar to the following:

OPERATION-ID     STATEMENTS                                                                                          DONE   @TYPE
_auto_op_123456  CREATE INDEX SingersByFirstLastName ON Singers(FirstName, LastName)                                 False  UpdateDatabaseDdlMetadata
               CREATE INDEX SongsBySingerAlbumSongName ON Songs(SingerId, AlbumId, SongName), INTERLEAVE IN Albums
_auto_op_234567                                                                                                      True   CreateDatabaseMetadata

Cancel a long-running schema update operation

gcloud

  1. Get the operation ID:

    gcloud spanner operations list --instance=INSTANCE-NAME \
    --database=DATABASE-NAME --type=DATABASE_UPDATE_DDL
    

    Replace the following:

    • INSTANCE-NAME with the Spanner instance name.
    • DATABASE-NAME with the name of the database.
  2. Use the gcloud operations cancel to cancel a long-running schema update operation.

    gcloud operations cancel OPERATION-ID \
    --instance=INSTANCE-NAME \
    

    Replace the following:

    • OPERATION-ID: The operation ID of the operation that you want to check.
    • INSTANCE-NAME: The Spanner instance name.

REST V1

Use projects.instances.databases.operations.cancel to cancel a long-running schema update operation.

  1. Get the operation ID:

    gcloud spanner operations list --instance=INSTANCE-NAME \
    --database=DATABASE-NAME --type=DATABASE_UPDATE_DDL
    

    Replace the following:

    • INSTANCE-NAME with the Spanner instance name.
    • DATABASE-NAME with the name of the database.
  2. Click projects.instances.databases.operations.cancel.

  3. For name, enter the long-running schema update operation name as shown in the long-running schema update operation response. For example:

    projects/PROJECT-ID/instances/test-instance/databases/example-db/operations/OPERATION-ID

    Replace the following:

    • PROJECT-ID with the project ID.
    • OPERATION-ID with the operations ID.

    You can also list long-running instance operations to retrieve the schema update operation name.

  4. Click Execute. The operation stops running.

Backup and restore operations

The following are long-running backup operations.

Check the progress of a long-running backup or restore operation

Console

Backup

  1. In the Spanner navigation menu, select the Operations tab. The Operations page shows a list of currently running operations.

  2. Find the schema operation in the list. If it's still running, the progress bar in the End time column shows the percentage of the operation that is complete, as shown in the following image:

Screenshot of progress indicator showing 98%

Restore

To check the progress of the restore operation, see the progress indicator that is displayed during the restore, as shown in the following image:

Screenshot of progress indicator showing 56%

If the operation takes too long, you can cancel it. For more information, see Cancel a long-running instance operation.

gcloud

Use gcloud spanner operations describe to check the progress of a backup or restore operation.

  1. Get the operation ID:

    gcloud spanner operations list --instance=INSTANCE-NAME \
    --database=DATABASE-NAME --type=DATABASE_UPDATE_DDL
    

    Replace the following:

    • INSTANCE-NAME with the Spanner instance name.
    • DATABASE-NAME with the name of the database.
  2. Run gcloud spanner operations describe:

    gcloud spanner operations describe OPERATION-ID\
    --instance=INSTANCE-NAME \
    --database=DATABASE-NAME
    

    Replace the following:

    • OPERATION-ID: The operation ID of the operation that you want to check.
    • INSTANCE-NAME: The Spanner instance name.
    • DATABASE-NAME: The Spanner database name.

    The progress section in the output shows the percentage of the operation that's complete. The output looks similar to the following:

    done: true
    metadata:
    ...
      progress:
      - endTime: '2022-03-01T00:28:06.691403Z'
        progressPercent: 100
        startTime: '2022-03-01T00:28:04.221401Z'
      - endTime: '2022-03-01T00:28:17.624588Z'
        startTime: '2022-03-01T00:28:06.691403Z'
        progressPercent: 100
    ...
    

REST v1

Get the operation ID:

  gcloud spanner operations list --instance=INSTANCE-NAME 
--database=DATABASE-NAME --type=DATABASE_UPDATE_DDL

Replace the following:

  • INSTANCE-NAME with the Spanner instance name.
  • DATABASE-NAME with the name of the database.

Before using any of the request data, make the following replacements:

  • project-id: The project ID.
  • instance-id: The instance ID.
  • database-id: The database ID.
  • operation-id: The operation ID.

HTTP method and URL:

GET https://spanner.googleapis.com/v1/projects/project-id/instances/instance-id/databases/database-id/operations/operation-id

To send your request, expand one of these options:

You should receive a JSON response similar to the following:

{
...
    "progress": [
      {
        "progressPercent": 100,
        "startTime": "2023-05-27T00:52:27.366688Z",
        "endTime": "2023-05-27T00:52:30.184845Z"
      },
      {
        "progressPercent": 100,
        "startTime": "2023-05-27T00:52:30.184845Z",
        "endTime": "2023-05-27T00:52:40.750959Z"
      }
    ],
...
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.protobuf.Empty"
  }
}

If the operation takes too long, you can cancel it. For more information, see Cancel a long-running backup operation.

List long-running backup or restore operations

Use projects.instances.backups.operations.list to list the operations on a single backup or projects.instances.backupOperations.list to list all backup operations in the instance.

  1. Click projects.instances.backups.operations.list.

  2. For name, enter:

    projects/project-id/instances/test-instance/backups/example-db-backup-4/operations

  3. Click Execute. The response contains a list of long-running operations.

Cancel a long-running backup operation

Use projects.instances.backups.operations.cancel to cancel a long-running backup operation.

  1. Click projects.instances.backups.operations.cancel.

  2. For name, enter the long-running backup operation name as shown in the long-running backup operation response. For example:

    projects/project-id/instances/test-instance/backups/example-db-backup-4/operations/_auto_1492721321097206

    Replace _auto_1492721321097206 with your operation ID.

    You can retrieve the backup operation name by listing long-running backup operations.

  3. Click Execute.

Instance operations

The following are long-running instance operations.

Check the progress of a long-running instance operation

Use projects.instances.operations.get to check the progress of a long-running instance operation.

As an example, this is a response from projects.instances.create:

  {
    "name": "projects/<VAR>PROJECT-ID</VAR>/instances/test-instance/operations/<VAR>OPERATION-ID</VAR>",
    "metadata": {
      "@type": "type.googleapis.com/google.spanner.admin.instance.v1.CreateInstanceMetadata",
      "instance": {
        "name": "projects/<VAR>PROJECT-ID</VAR>/instances/test-instance",
        "config": "projects/<VAR>PROJECT-ID</VAR>/instanceConfigs/regional-us-central1",
        "displayName": "Test Instance",
        "nodeCount": 1,
        "state": "READY"
      },
      "startTime": "2017-04-24T22:45:41.130854Z"
    }
  }

The name value at the top of the response shows the Spanner service created a long-running instance operation named projects/PROJECT-ID/instances/test-instance/operations/<VAR>OPERATION-ID</VAR>.

To Check the progress of the long-running instance operation:

  1. Click projects.instances.operations.get.
  2. For name, enter the long-running instance operation name as shown in the response to projects.instances.create or projects.instances.patch. For example:

    projects/PROJECT-ID/instances/test-instance/operations/OPERATION-ID

    Replace the following:

    • PROJECT-ID with the project ID.
    • OPERATION-ID with the operations ID.

    You can retrieve the instance operation name by listing long-running instance operations.

  3. Click Execute. When an operation is done, the done field is set to true.

To get continuous updates, repeatedly invoke the projects.instances.databases.operations.get method until the operation is done. Use a backoff between each request. For example, make a request every 10 seconds.

List long-running instance operations

Use projects.instances.operations.list to list long-running instance operations.

  1. Click projects.instances.operations.list.
  2. For name, enter:

    projects/PROJECT-ID/instances/test-instance/operations

  3. Click Execute. The response contains a list of long-running operations.

Cancel a long-running instance operation

Use projects.instances.operations.cancel to cancel a long-running instance operation.

  1. Click projects.instances.operations.cancel.
  2. For name, enter the long-running instance operation name as shown in the long-running instance operation response. For example:

    projects/PROJECT-ID/instances/test-instance/operations/OPERATION-ID

    Replace the following:

    • PROJECT-ID with the project ID.
    • OPERATION-ID with the operations ID.

    You can retrieve the instance operation name by listing long-running instance operations.

  3. Click Execute.

Delete a long-running instance operation

Use projects.instances.operations.delete to delete a long-running instance operation.

  1. Click projects.instances.operations.delete.
  2. For name, enter the long-running instance operation name as shown in the long-running instance operation response. For example:

    projects/PROJECT-ID/instances/test-instance/operations/OPERATION-ID

    Replace the following:

    • PROJECT-ID with the project ID.
    • OPERATION-ID with the operations ID.

    You can retrieve the instance operation name by listing long-running instance operations.

  3. Click Execute. The operation is deleted.