Managing long-running operations

This page describes how to manage the lifecycle of a Cloud Healthcare API long-running operation (LRO).

When an API method might take a long time to complete, it can return an Operation to the client. The client can use the Operation interface to retrieve the status of the API method asynchronously by polling the operation. LROs in the Cloud Healthcare API adhere to the Google Cloud LRO design pattern.

The Cloud Healthcare API creates LROs for several methods, such as projects.locations.datasets.fhirStores.import. When projects.locations.datasets.fhirStores.import is called, the Cloud Healthcare API creates an LRO to track the import status. The LRO has a unique identifier that you can use to view the status of the LRO.

LROs are managed at the dataset level. If you are calling a method that returns an LRO, like projects.locations.datasets.fhirStores.import, you can view the status of the LRO by sending a request containing the LRO ID to the parent dataset of the FHIR store where the import is occurring.

In addition to the REST API, the following sources generate long-running operations when you call the methods listed in Methods that return an LRO:

You can manage your Cloud Healthcare API LROs using the Google Cloud console, the Google Cloud CLI, or the REST API.

The record of an LRO is kept for approximately 30 days after the LRO finishes, meaning that you cannot view or list an LRO after that point.

Methods that return an LRO

The following methods return an LRO.

Consent management methods:

Dataset methods:

DICOM methods:

FHIR methods:

HL7v2 methods:

Getting details about an LRO

The following samples show how to get details about an LRO.

The version of the Cloud Healthcare API shown in the response when getting details about an LRO is the same as the API version of the method that initiated the LRO.

Console

After calling a method using either the gcloud CLI or the API that returns an LRO, you can view details about the LRO in the Google Cloud console.

  1. In the Google Cloud console, go to the Datasets page.

    Go to Datasets

  2. Click the name of the dataset containing the LRO you want to view.

  3. Click Operations.

  4. To view error logs for the operation in Cloud Logging, click Actions and then click View details in Cloud Logging. For more information, see Viewing error logs in Cloud Logging.

  5. To view more details about the operation, click a running operation ID. The Long-running Operation Details page is displayed. The page shows the following information:

    • The progress of the LRO
    • The details of the LRO, such as the operation ID and the method that invoked the LRO
    • A subset of log entries

gcloud

Suppose that you receive the following response after calling gcloud healthcare dicom-stores deidentify:

Request issued for: [DATASET_ID]
Waiting for operation [OPERATION_ID] to complete...

The response shows that the Cloud Healthcare API created an LRO with an operation ID. You can also retrieve the operation ID by listing long-running database operations. The command continues to run until it finishes, after which it outputs the following:

Request issued for: [DATASET_ID]
Waiting for operation [OPERATION_ID] to complete...done
name: projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID

To get details about the LRO, run the gcloud healthcare operations describe command.

Before using any of the command data below, make the following replacements:

  • PROJECT_ID: the ID of your Google Cloud project
  • DATASET_ID: the dataset ID
  • LOCATION: the dataset location
  • OPERATION_ID: the ID returned from the long-running operation

Execute the following command:

Linux, macOS, or Cloud Shell

gcloud healthcare operations describe OPERATION_ID \
    --project=PROJECT_ID \
    --dataset=DATASET_ID \
    --location=LOCATION

Windows (PowerShell)

gcloud healthcare operations describe OPERATION_ID `
    --project=PROJECT_ID `
    --dataset=DATASET_ID `
    --location=LOCATION

Windows (cmd.exe)

gcloud healthcare operations describe OPERATION_ID ^
    --project=PROJECT_ID ^
    --dataset=DATASET_ID ^
    --location=LOCATION

You should receive a response similar to the following:

Response

done: true
// If there were any errors, an `error` field displays instead of a `response` field.
// See Troubleshooting long-running operations for a list of response codes.
error: ERROR
  code: ERROR_CODE
  message: DESCRIPTION
metadata:
  '@type': 'type.googleapis.com/google.cloud.healthcare.v1.OperationMetadata'
  apiMethodName: 'google.cloud.healthcare.v1.deidentify.DeidentifyService.DeidentifyDicomStore'
  counter:
    success: 'SUCCESS_COUNT'
    // If there were any failures, they display in the `failure` field.
    failure: 'FAILURE_COUNT'
  createTime: 'YYYY-MM-DDTHH:MM:SS+ZZ:ZZ'
  endTime: 'YYYY-MM-DDTHH:MM:SS+ZZ:ZZ'
  logsUrl: https://console.cloud.google.com/CLOUD_LOGGING_URL
name: projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID
// The `response` field only displays if there were no errors.
response:
  '@type': 'type.googleapis.com/google.cloud.healthcare.v1.deidentify.DeidentifySummary'

REST

Suppose that you receive the following response after calling projects.locations.datasets.dicomStores.deidentify:

{
  "name": "projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID"
}

The name value in the response shows that the Cloud Healthcare API created an LRO named projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID. You can also retrieve the LRO name by listing LROs.

To get the status of the LRO, use the projects.locations.datasets.operations.get method. To poll an LRO, repeatedly call the projects.locations.datasets.operations.get method until the operation finishes. Use a backoff between each poll request, such as 10 seconds.

Before using any of the request data, make the following replacements:

  • PROJECT_ID: the ID of your Google Cloud project
  • DATASET_ID: the dataset ID
  • LOCATION: the dataset location
  • OPERATION_ID: the ID returned from the long-running operation

To send your request, choose one of these options:

curl

Execute the following command:

curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID"

PowerShell

Execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
-Method GET `
-Headers $headers `
-Uri "https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID" | Select-Object -Expand Content

APIs Explorer

Open the method reference page. The APIs Explorer panel opens on the right side of the page. You can interact with this tool to send requests. Complete any required fields and click Execute.

The output is the following. When the response contains "done": true, the long-running operation has finished.

Listing LROs

The following samples show how to list the LROs in a dataset.

Console

To view a list of all LROs in a dataset in the Google Cloud console, complete the following steps:

  1. In the Google Cloud console, go to the Datasets page.

    Go to Datasets

  2. Click the name of the dataset containing the LRO you want to view.

  3. Click Operations.

A list of LROs in the dataset and their status displays. To view the error logs in Cloud Logging, click the more information icon in the last column and then click View details in Cloud Logging. For more information, see Viewing error logs in Cloud Logging.

gcloud

To list the LROs in a dataset, run the gcloud healthcare operations list command.

Before using any of the command data below, make the following replacements:

  • DATASET_ID: the dataset ID
  • LOCATION: the dataset location

Execute the following command:

Linux, macOS, or Cloud Shell

gcloud healthcare operations list --dataset=DATASET_ID --location=LOCATION

Windows (PowerShell)

gcloud healthcare operations list --dataset=DATASET_ID --location=LOCATION

Windows (cmd.exe)

gcloud healthcare operations list --dataset=DATASET_ID --location=LOCATION

You should receive a response similar to the following:

Response

ID             LOCATION     DONE
OPERATION_ID   LOCATION {TRUE|FALSE}
...

REST

To list the LROs in a dataset, use the projects.locations.datasets.operations.get method.

Before using any of the request data, make the following replacements:

  • PROJECT_ID: the ID of your Google Cloud project
  • DATASET_ID: the dataset ID
  • LOCATION: the dataset location

To send your request, choose one of these options:

curl

Execute the following command:

curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations"

PowerShell

Execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
-Method GET `
-Headers $headers `
-Uri "https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations" | Select-Object -Expand Content

APIs Explorer

Open the method reference page. The APIs Explorer panel opens on the right side of the page. You can interact with this tool to send requests. Complete any required fields and click Execute.

You should receive a JSON response similar to the following:

Cancelling an LRO

The following samples show how to cancel an LRO in a dataset.

Console

To cancel an LRO in the Google Cloud console, complete the following steps:

  1. In the Google Cloud console, go to the Datasets page.

    Go to Datasets

  2. Click the name of the dataset containing the LRO you want to view.

  3. Click Operations.

  4. In the same row as the LRO you want to cancel, open the Actions list and click Stop Operation.

REST

To cancel an LRO, use the projects.locations.datasets.operations.cancel method.

Before using any of the request data, make the following replacements:

  • PROJECT_ID: the ID of your Google Cloud project
  • DATASET_ID: the dataset ID
  • LOCATION: the dataset location
  • OPERATION_ID: the ID returned from the long-running operation

To send your request, choose one of these options:

curl

Execute the following command:

curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json; charset=utf-8" \
-d "" \
"https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID:cancel"

PowerShell

Execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
-Method POST `
-Headers $headers `
-Uri "https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID:cancel" | Select-Object -Expand Content

APIs Explorer

Open the method reference page. The APIs Explorer panel opens on the right side of the page. You can interact with this tool to send requests. Complete any required fields and click Execute.

You should receive a JSON response similar to the following:

To view the status of the cancel request, use the projects.locations.datasets.operations.get method.

Before using any of the request data, make the following replacements:

  • PROJECT_ID: the ID of your Google Cloud project
  • DATASET_ID: the dataset ID
  • LOCATION: the dataset location
  • OPERATION_ID: the ID returned from the long-running operation

To send your request, choose one of these options:

curl

Execute the following command:

curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID"

PowerShell

Execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
-Method GET `
-Headers $headers `
-Uri "https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID" | Select-Object -Expand Content

APIs Explorer

Open the method reference page. The APIs Explorer panel opens on the right side of the page. You can interact with this tool to send requests. Complete any required fields and click Execute.

You should receive a JSON response similar to the following:

Cancelling multiple LROs

To cancel multiple LROs, complete the following steps:

  1. Call the operations.list method to get the names of the operations in a dataset.
  2. Call the operations.cancel method on each operation.

Google provides a Python script that you can use to cancel all operations for a given dataset.

Troubleshooting LROs

When a LRO fails, its response includes a canonical Google Cloud error code. The following table provides an explanation of the cause for each code and a recommendation for how to handle the code. For many errors, the recommended action is to try the request again using exponential backoff. For information on how to implement exponential backoff in the Cloud Healthcare API, see Retry failed requests.

Code Enum Description Recommended action
1 CANCELLED The operation was cancelled, typically by the caller. Re-run the operation if desired.
2 UNKNOWN This error might be returned when a Status value received from another address space belongs to an error space that is not known in this address space. If an API's error doesn't return enough information, the error might be converted to this error. Retry with exponential backoff.
3 INVALID_ARGUMENT The client specified an invalid argument. This error differs from FAILED_PRECONDITION. INVALID_ARGUMENT indicates arguments that are problematic regardless of the state of the system, such as a malformed file name. Don't retry without fixing the problem.
4 DEADLINE_EXCEEDED The deadline expired before the operation could complete. For operations that change the state of the system, this error might be returned even if the operation has completed successfully. For example, a successful response from a server could have been delayed long enough for the deadline to expire. Retry with exponential backoff.
5 NOT_FOUND Some requested entity, such as a FHIR resource, was not found. Don't retry without fixing the problem.
6 ALREADY_EXISTS The entity that a client attempted to create, such as a DICOM instance, already exists. Don't retry without fixing the problem.
7 PERMISSION_DENIED The caller doesn't have permission to execute the specified operation. This error code doesn't imply the request is valid or the requested entity exists or satisfies other preconditions. Don't retry without fixing the problem.
8 RESOURCE_EXHAUSTED Some resource has been exhausted, such as a per-project quota. See Best practices for quota management for recommended actions. Retry with exponential backoff. Quota might become available over time.
9 FAILED_PRECONDITION The operation was rejected because the system is not in a state required for the operation's execution. For example, the directory to be deleted is non-empty, or an rmdir operation is applied to a non-directory. Don't retry without fixing the problem.
10 ABORTED The operation was aborted, typically due to a concurrency issue such as a sequencer check failure or transaction abort. Retry with exponential backoff.
11 OUT_OF_RANGE The operation was attempted past the valid range, such as seeking or reading past end-of-file. Unlike INVALID_ARGUMENT, this error indicates a problem that may be fixed if the system state changes. Don't retry without fixing the problem.
12 UNIMPLEMENTED The operation isn't implemented or isn't supported/enabled in the Cloud Healthcare API. Don't retry.
13 INTERNAL Internal errors. This indicates that an unexpected error was encountered in processing on the underlying system. Retry with exponential backoff.
14 UNAVAILABLE The Cloud Healthcare API is currently unavailable. This is most likely a transient condition, which can be corrected by retrying with a backoff. Note that it is not always safe to retry non-idempotent operations. Retry with exponential backoff.
15 DATA_LOSS Unrecoverable data loss or corruption. Contact your system administrator. The system administrator might want to contact a support representative if data loss or corruption occurred.
16 UNAUTHENTICATED The request doesn't have valid authentication credentials for the operation. Don't retry without fixing the problem.