Process genomic data by using Cloud Life Sciences

Process genomic data by using Cloud Life Sciences

Tip: Using a higher-level workflow specification can simplify implementing and managing the Cloud Life Sciences API. See Tutorials for examples of workflow specifications that have integrated with the Cloud Life Sciences API.

This page explains how to run a genomics pipeline that uses the Cloud Life Sciences API to create an index file (BAI file) from a binary file containing DNA sequences (BAM file).

BAM files are typically large and can take a long time to read using a genome viewer. You use a BAI file to locate the portions of the BAM file that contain the genome position you are interested in.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. Enable the Cloud Life Sciences, Compute Engine, and Cloud Storage JSON APIs.

   Enable the APIs

5. Install the Google Cloud CLI.

6. To initialize the gcloud CLI, run the following command:

   gcloud init

7. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

8. Make sure that billing is enabled for your Google Cloud project.

9. Enable the Cloud Life Sciences, Compute Engine, and Cloud Storage JSON APIs.

   Enable the APIs

10. Install the Google Cloud CLI.

11. To initialize the gcloud CLI, run the following command:

    gcloud init

Alternatively, you can use Cloud Shell, which comes with the gcloud CLI already installed.

12. Install Python 3.8.

    If you are using Windows and you left the relevant checkbox selected when you installed the Google Cloud CLI, this was done automatically.

Note: If you are using Cloud Shell or a Compute Engine instance, Python and the Google Cloud CLI are pre-installed and authenticated. If your Compute Engine instance is set up without a service account scope to Cloud Life Sciences, run gcloud init and follow the instructions.

Run the pipeline

To run the pipeline, complete the following steps:

1. Create a bucket where you store the BAI file. Buckets are the basic containers that hold your data in Cloud Storage. To create a bucket named PROJECT_ID-life-sciences, run the gcloud storage buckets create command:

   gcloud storage buckets create gs://PROJECT_ID-life-sciences

   Replace PROJECT_ID with your Google Cloud project ID. You must use a globally unique bucket name.

   See bucket naming requirements.

   • Bucket names can only contain lowercase letters, numeric characters, dashes (-), and underscores (_). Spaces are not allowed.
   • Bucket names must start and end with a number or letter.
   • Bucket names must contain 3-63 characters. Names containing dots can contain up to 222 characters, but each dot-separated component can be no longer than 63 characters.
   • Bucket names cannot be represented as an IP address in dotted-decimal notation (for example, 192.168.5.4).
   • Bucket names cannot begin with the "goog" prefix.
   • Bucket names cannot contain "google" or close misspellings, such as "g00gle".
   Caution: Do not include sensitive information in the bucket name, because the bucket namespace is global and publicly visible.

   If successful, the command returns the following:

   Creating gs://PROJECT_ID-life-sciences

2. To start the pipeline, run the gcloud beta lifesciences pipelines run command:

   gcloud beta lifesciences pipelines run \
       --regions us-east1 \
       --command-line 'samtools index ${BAM} ${BAI}' \
       --docker-image "gcr.io/cloud-lifesciences/samtools" \
       --inputs BAM=gs://genomics-public-data/NA12878.chr20.sample.bam \
       --outputs BAI=gs://PROJECT_ID-life-sciences/NA12878.chr20.sample.bam.bai

   If successful, the command returns the following:

   Running [projects/PROJECT_ID/operations/OPERATION_ID]

   Note the OPERATION_ID, which you use in the next step.

3. To track the pipeline's status, run the gcloud beta lifesciences operations wait command. Replace OPERATION_ID with the value printed in the previous step. The pipeline takes a few minutes to finish.

   gcloud beta lifesciences operations wait OPERATION_ID

   After the operation finishes, it returns the following message:

   Waiting for [projects/PROJECT_ID/operations/OPERATION_ID]...done.

4. To verify that the BAI file was generated, run the gcloud storage ls command:

   gcloud storage ls gs://PROJECT_ID-life-sciences

   If successful, the command returns the following:

   gs://PROJECT_ID-life-sciences/NA12878.chr20.sample.bam.bai

You've run a pipeline using the Cloud Life Sciences API to create a BAI file from a BAM file. Use a genome viewer to examine the NA12878.chr20.sample.bam BAM file using the NA12878.chr20.sample.bam.bai index file.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used on this page, follow these steps.

Delete the BAI file

To delete the generated BAI file but keep the project and bucket you created, run the gcloud storage rm command:

gcloud storage rm PROJECT_ID-life-sciences/NA12878.chr20.sample.bam.bai

Delete the bucket

If you created the bucket specifically for this quickstart and no longer need it, but want to keep your project, delete the bucket using the gcloud storage rm command. Deleting the bucket also deletes the generated BAI file.

gcloud storage rm gs://PROJECT_ID-life-sciences --recursive

Delete the project

If you created the project specifically for this quickstart and no longer need it, you can delete the project. Deleting the project also deletes the BAI file and the Cloud Storage bucket.

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

How did it go?

It worked!

Great! What did you like about the quickstart? What could we have done better? Let us know!.

I got stuck.

We're sorry to hear that. Let us know what went wrong. We'll want to fix it.

What's next

• Learn more about Cloud Life Sciences API public datasets.
• Learn how to load variant data into Cloud Storage or BigQuery.
• Learn how to analyze variants with BigQuery.