Troubleshooting

API not enabled or service account deleted error

When calling the Cloud Life Sciences API, you might encounter one or both of the following errors:

  • API not enabled or service account deleted
  • checking service account permission: Account deleted: PROJECT_ID

To solve these issues, complete the following steps in order:

  1. Make sure that the Cloud Life Sciences and Compute Engine APIs are enabled.
  2. Make sure that the Cloud Life Sciences Service Agent service account is configured correctly.
  3. Make sure that the Compute Engine default service account is configured correctly.

Enabling the Cloud Life Sciences and Compute Engine APIs

Make sure that the Cloud Life Sciences and Compute Engine APIs are enabled in your Google Cloud project:

  1. Enable the Cloud Life Sciences API:

    Enable Cloud Life Sciences API

  2. Enable the Compute Engine API:

    Enable Compute Engine API

If you encounter a permissions error indicating that you don't have permission to enable Google Cloud APIs for your project, then see Enabling and disabling APIs for information on how to enable Google Cloud APIs.

Missing the Cloud Life Sciences service account or Cloud Life Sciences Service Agent role

The Cloud Life Sciences Service Agent service account is automatically created when you enable Cloud Life Sciences. This is a Google-managed service account. You cannot delete the service account entirely, but under certain circumstances it might not appear in the Cloud Identity and Access Management page and you might encounter issues with the Cloud Life Sciences API.

For the Cloud Life Sciences API to function correctly and complete tasks like running pipelines on Compute Engine VMs, the Cloud Life Sciences Service Agent service account must exist and must have the Life Sciences Service Agent Cloud IAM role.

You can recreate the Cloud Life Sciences Service Agent service account or grant it the Life Sciences Service Agent Cloud IAM role if you encounter any of the following issues:

  • You cannot find the Cloud Life Sciences Service Agent service account in the Cloud Identity and Access Management page.
  • You can find the Cloud Life Sciences Service Agent service account, but it does not contain the Life Sciences Service Agent role.

Use the gcloud command-line tool to add the lifesciences.serviceAgent role to the Cloud Life Sciences Service Agent service account using the service account's identifier, which uses the format service-PROJECT_NUMBER@gcp-sa-lifesciences.iam.gserviceaccount.

To recreate the service account or grant it the Life Sciences Service Agent Cloud IAM role, run the gcloud projects add-iam-policy-binding command. To find the PROJECT_ID and PROJECT_NUMBER, see Identifying projects.

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member=serviceAccount:service-PROJECT_NUMBER@gcp-sa-lifesciences.iam.gserviceaccount.com \
    --role=roles/lifesciences.serviceAgent

If the request is successful, the command prompt displays a message similar to the following sample:

Updated IAM policy for project [PROJECT_ID].
bindings:
...
- members:
  - serviceAccount:service-PROJECT_NUMBER@gcp-sa-lifesciences.iam.gserviceaccount.com
  role: roles/lifesciences.serviceAgent
...
etag: VALUE
version: VALUE

Return to the Cloud Identity and Access Management page again and verify the following:

  • The Member column contains a service account identifier in the format service-PROJECT_NUMBER@gcp-sa-lifesciences.iam.gserviceaccount.
  • In the same row as the Member column, the Name column contains Cloud Life Sciences Service Agent.
  • In the same row as the Member column, the Role column contains Life Sciences Service Agent.

Missing the Compute Engine default service account

Newly created Google Cloud projects come with the Compute Engine default service account, identifiable using the following email:

PROJECT_NUMBER-compute@developer.gserviceaccount.com

The service account must exist in your Google Cloud project, otherwise the Cloud Life Sciences API cannot run pipelines on Compute Engine VMs. If you delete the service account from your project, any applications that depend on the service account's credentials might fail. If you accidentally delete the Compute Engine default service account, you can try to recover the account within 30 days. See Undeleting a service account for more information.

Cannot authenticate to the Cloud Life Sciences API

If you are running a pipeline with the Cloud Life Sciences API using a service account as your credentials (as opposed to, for example, using gcloud auth application-default login as your credentials), ensure that the service account has the following roles:

  • roles/lifesciences.workflowsRunner
  • roles/iam.serviceAccountUser

To add these roles to your service account, complete the following steps using either the Google Cloud Console or the gcloud command-line tool:

Console

  1. Make sure that you have enabled the Cloud Life Sciences API.
  2. On the Cloud IAM page in Google Cloud Console, find your service account.
  3. In the Inheritance column that matches the service account, click the pencil icon. The Edit permissions pane opens.
  4. Click Add another role and then search for the Life Sciences Workflows Runner and Service Account User roles.
  5. Select the role and then click Save. The lifesciences.workflowsRunner and iam.serviceAccountUser roles are then added to the service account.

gcloud

To add the service account permissions, run the gcloud projects add-iam-policy-binding command. To find the PROJECT_ID and PROJECT_NUMBER, see Identifying projects.

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member=serviceAccount:service-PROJECT_NUMBER@SERVICE_ACCOUNT_ID.iam.gserviceaccount.com \
    --role=roles/lifesciences.workflowsRunner
gcloud projects add-iam-policy-binding PROJECT_ID \
    --member=serviceAccount:service-PROJECT_NUMBER@SERVICE_ACCOUNT_ID.iam.gserviceaccount.com \
    --role=roles/iam.serviceAccountUser

Cannot authenticate using Application Default Credentials

When calling the Cloud Life Sciences API, you might receive an error message indicating that your "Application Default Credentials" are unavailable.

See Setting up authentication for server-to-server applications for information on how to configure Application Default Credentials or how to pass in authentication credentials manually to an application or command.

Status codes

Error codes

The Cloud Life Sciences API can return the following error codes:

RESOURCE EXHAUSTED (8)

Code: 8

Status: RESOURCE_EXHAUSTED

Category: User error

Description: A resource has been exhausted. This can indicate that your application has exhausted a project-level admin API quota.

Recommended action: Retry the operation.

FAILED_PRECONDITION (9)

Code: 9

Status: FAILED_PRECONDITION

Category: User error

Full error: Execution failed: while running "[USER_COMMAND_LINE]": unexpected exit status [NUMBER] was not ignored

Description: The operation was rejected because a user action returned a non-zero exit status. A snippet of the standard error output from the action displays that you can use to diagnose the issue. To upload the full logs from the Compute Engine virtual machine (VM), use the ALWAYS_RUN action when making the pipeline request, like so:

{
  "commands": [
    "-c",
    "gsutil -q cp /google/logs/output gs://CLOUD_STORAGE_BUCKET/output"
  ],
  "entrypoint": "bash",
  "flags": [ "ALWAYS_RUN" ],
  "imageUri": "gcr.io/cloud-genomics-pipelines/io"
}

Recommended action: Do not retry without fixing the problem.

ABORTED (10)

Code: 10

Status: ABORTED

Category: System error

Full error: The assigned worker has failed to complete the operation

Description: The operation was aborted because the Compute Engine VM running the pipeline failed, possibly because it was preempted and could not report its status before terminating.

Recommended action: Retry the operation. If the error reoccurs consistently, there might be an issue causing the Compute Engine VM to fail, such as using too many resources. Inspect the Compute Engine logs in Stackdriver Logging for further information.

UNAVAILABLE (14)

Code: 14

Status: UNAVAILABLE

Category: System error

Full error: Execution failed: worker was terminated

Description: The Compute Engine VM running the pipeline was preempted.

Recommended action: Retry the operation.

Retrying after encountering errors

A pipeline might fail and return an error code. Failures can occur due issues unrelated to the work that the pipeline was doing, and in most cases you should retry the pipeline operation. Pipelines are particularly prone to failures when you use preemptible VMs, which are cheaper but more likely to encounter interruptions. The Cloud Life Sciences API cannot retry pipeline operations automatically because not all pipelines are idempotent.

As shown in the error codes section, retrying is typically recommended when you encounter any of the following error codes:

  • RESOURCE EXHAUSTED (8)
  • ABORTED (10)
  • UNAVAILABLE (14)

Enabling Stackdriver Monitoring

You can enable Stackdriver Monitoring on your pipelines to monitor the health and resource usage of the worker VMs used to run the pipeline. However, enabling Monitoring can incur additional costs. To enable Monitoring, specify the enableStackdriverMonitoring flag on the VirtualMachine object when making the pipeline request.

Pipeline is running out of disk space

If your pipeline is running out of disk space and cannot pull Docker images, or if it needs more disk space to log or perform its tasks, you can choose either of the following options:

  • Increase the boot disk size using the bootDiskSizeGb flag in the VirtualMachine object when making the pipeline request.
  • Attach a separate disk and add it to the Mount object inside of the Action object when making the pipeline request.

Encountering quota delays

The Cloud Life Sciences API will not allocate VMs if your Google Cloud project is out of Compute Engine quota. Any further allocation attempts are delayed to give existing pipelines time to complete. If this delay continues to occur, you can request an increase in quota.

Pipelines are hanging

If your pipelines are hanging and repeatedly assigning and releasing VMs because they cannot communicate with the Cloud Life Sciences API service, it might be because the VMs' default network was deleted and another network was not specified in the Network object.

To solve this issue, either:

  • Specify a network in the Network object

OR

Cancelling or deleting VMs

Rather than deleting unwanted worker VMs, you should cancel their associated operations. If you delete the VM, the pipeline will fail slowly and, if it is in the process of starting up, a new VM might be assigned.

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Life Sciences Documentation