GPU-accelerated video transcoding with FFmpeg on Cloud Run jobs

This tutorial describes how to transcode low-priority offline videos using Cloud Run jobs.

Preview — GPU support for Cloud Run jobs

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

Objectives

In this tutorial, you will do the following:

• Create Cloud Storage buckets to store the videos for processing and to store the encoding results.
• Deploy a Cloud Run job using GPUs to accelerate video transcoding.
• Run the job and make sure the video transcoded successfully.

Costs

In this document, you use the following billable components of Google Cloud:

• Cloud Build
• Artifact Registry
• Cloud Run

To generate a cost estimate based on your projected usage, use the pricing calculator.

New Google Cloud users might be eligible for a free trial.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

To initialize the gcloud CLI, run the following command:

gcloud init

Create or select a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

• Create a Google Cloud project:

  gcloud projects create PROJECT_ID

  Replace PROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set project PROJECT_ID

  Replace PROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

To initialize the gcloud CLI, run the following command:

gcloud init

Create or select a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

• Create a Google Cloud project:

  gcloud projects create PROJECT_ID

  Replace PROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set project PROJECT_ID

  Replace PROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

1. Enable the Cloud Run, Artifact Registry, and Cloud Build APIs:

   Roles required to enable APIs

   To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.

   gcloud services enable run.googleapis.com cloudbuild.googleapis.com artifactregistry.googleapis.com

2. Set your region as an environment variable:

   export REGION=REGION

3. Create a service account:

   gcloud iam service-accounts create video-encoding

4. Request Total Nvidia L4 GPU allocation without zonal redundancy, per project per region, under Cloud Run Admin API in the Quotas and system limits page to complete this tutorial. Alternatively, you can deploy a Cloud Run service to automatically receive a grant of 3 nvidia-l4 GPU quota (zonal redundancy off) for a region.

Required roles

To get the permissions that you need to complete the tutorial, ask your administrator to grant you the following IAM roles on your project:

• Artifact Registry Repository Administrator (roles/artifactregistry.repoAdmin)
• Cloud Build Editor (roles/cloudbuild.builds.editor)
• Cloud Run Admin (roles/run.admin)
• Create Service Accounts (roles/iam.serviceAccountCreator)
• Service Account User (roles/iam.serviceAccountUser)

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Note: IAM basic roles might also contain permissions to complete the tutorial. You shouldn't grant basic roles in a production environment, but you can grant them in a development or test environment.

Grant the roles

Console

1. In the Google Cloud console, go to the IAM page.

   Go to IAM
2. Select the project.
3. Click Grant access.

4. In the New principals field, enter your user identifier. This is typically the Google Account email address that is used to deploy the Cloud Run service.

5. In the Select a role list, select a role.
6. To grant additional roles, click Add another role and add each additional role.
7. Click Save.

gcloud

To grant the required IAM roles to your account on your project:

     gcloud projects add-iam-policy-binding PROJECT_ID \
         --member=PRINCIPAL \
         --role=ROLE
     

Replace:

• PROJECT_NUMBER with your Google Cloud project number.
• PROJECT_ID with your Google Cloud project ID.
• PRINCIPAL with the account you are adding the binding for. This is typically the Google Account email address that is used to deploy the Cloud Run service.
• ROLE with the role you are adding to the deployer account.

Prepare your application

To retrieve the code sample for use:

1. Clone the sample repository to your local machine:

   git clone https://github.com/GoogleCloudPlatform/cloud-run-samples
   

2. Change to the directory that contains the Cloud Run sample code:

   cd cloud-run-samples/jobs-video-encoding
   

Create Cloud Storage buckets

To store the videos for processing, and to save the results of encoding, create the following two Cloud Storage buckets:

1. Create a bucket to store videos before processing:

   gcloud storage buckets create gs://preprocessing-PROJECT_ID \
     --location LOCATION
   

   Replace the following:

   • PROJECT_ID: your project ID.
   • LOCATION: the Cloud Storage location.

2. Grant the service account access to read from this bucket:

   gcloud storage buckets add-iam-policy-binding gs://preprocessing-PROJECT_ID \
     --member="serviceAccount:video-encoding@PROJECT_ID.iam.gserviceaccount.com" \
     --role="roles/storage.objectViewer"
   

   Replace PROJECT_ID with your project ID.

3. Create a bucket to store transcoded videos after processing:

   gcloud storage buckets create gs://transcoded-PROJECT_ID \
     --location LOCATION
   

   Replace the following:

   • PROJECT_ID: your project ID.
   • LOCATION: the Cloud Storage location.

4. Grant the service account access to read from and write to this bucket:

   gcloud storage buckets add-iam-policy-binding gs://transcoded-PROJECT_ID \
     --member="serviceAccount:video-encoding@PROJECT_ID.iam.gserviceaccount.com" \
     --role="roles/storage.objectAdmin"
   

   Replace PROJECT_ID with your project ID.

Deploy a Cloud Run job

Create a Cloud Run job by using the Dockerfile in the sample repository and mounting the buckets that you created:

1. Navigate to the sample directory:

   cd cloud-run-samples/jobs-video-encoding
   

1. Create an Artifact Registry if the default Cloud Run registry doesn't already exist:

   gcloud artifacts repositories create cloud-run-source-deploy \
     --repository-format=docker \
     --location LOCATION
   

   Replace LOCATION with the name of the location of the registry.

2. Build the container image:

   gcloud builds submit \
     --tag LOCATION-docker.pkg.dev/PROJECT_ID/cloud-run-source-deploy/IMAGE_NAME \
     --machine-type E2-HIGHCPU-32
   

   Replace the following:

   • PROJECT_ID: your project ID.
   • LOCATION:name of the location of the registry.
   • IMAGE_NAME: name for the container image, for example: ffmpeg-image.

   Cloud Run uses a larger machine type to reduce build time.

3. Deploy the job:

   gcloud beta run jobs create video-encoding-job \
       --image LOCATION-docker.pkg.dev/PROJECT_ID/cloud-run-source-deploy/IMAGE_NAME \
       --region REGION \
       --memory 32Gi \
       --cpu 8 \
       --gpu 1 \
       --gpu-type nvidia-l4 \
       --no-gpu-zonal-redundancy \
       --max-retries 1 \
       --service-account video-encoding@PROJECT_ID.iam.gserviceaccount.com \
       --add-volume=name=input-volume,type=cloud-storage,bucket=preprocessing-PROJECT_ID,readonly=true \
       --add-volume-mount=volume=input-volume,mount-path=/inputs \
       --add-volume=name=output-volume,type=cloud-storage,bucket=transcoded-PROJECT_ID \
       --add-volume-mount=volume=output-volume,mount-path=/outputs
   

   Replace the following:

   • PROJECT_ID: your project ID.
   • REGION: the name of the region. Note: This must be the same region that you have GPU quota for.
   • IMAGE_NAME: name for the container image, for example, ffmpeg-image.

   If this is the first time you deployed from source in this project, Cloud Run prompts you to create a default Artifact Registry repository.

Run the job

To run the job, follow these steps:

1. Upload an example video to encode:

   gcloud storage cp gs://cloud-samples-data/video/cat.mp4 gs://preprocessing-PROJECT_ID
   

2. Run the job:

   gcloud run jobs execute video-encoding-job  \
       --region REGION \
       --wait \
       --args="cat.mp4,encoded_cat.mp4,-vcodec,h264_nvenc,-cq,21,-movflags,+faststart"
   

   The entrypoint.sh file requires an input file, output file, and any arguments to send to FFmpeg.

3. Review the Cloud Run logs to make sure the video trancoded:

   gcloud run jobs logs read video-encoding-job --region REGION
   

4. Download the transcoded video:

   gcloud storage cp gs://transcoded-PROJECT_ID/encoded_cat.mp4 .
   

Clean up

To avoid additional charges to your Google Cloud account, delete all the resources you deployed with this quickstart.

Delete your repository

Cloud Run only charges for the time your job executes. However, you might still be charged for storing the container image in Artifact Registry. To delete Artifact Registry repositories, follow the steps in Delete repositories in the Artifact Registry documentation.

Delete your job

Cloud Run jobs only incur cost when a job task is executing. To delete your Cloud Run job, follow one of these steps:

Console

To delete a job:

1. In the Google Cloud console, go to Cloud Run:

   Go to Cloud Run

2. Locate the job you want to delete in the jobs list, and click its checkbox to select it.

3. Click Delete. This terminates all the job executions in progress and all running container instances.

gcloud

To delete a job, run the following command:

gcloud run jobs delete JOB_NAME

Replace JOB_NAME with the name of the job.

Delete your test project

Deleting your Google Cloud project stops billing for all resources in that project. To release all Google Cloud resources in your project, follow these steps:

Caution: Deleting a project has the following effects:
• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

Delete a Google Cloud project:

gcloud projects delete PROJECT_ID

What's next

• Explore other Cloud Run demos, tutorials, and samples.