Create and run an example job
Learn how to create and run an example video-transcoding job on Batch.
To follow step-by-step guidance for this task directly in the Google Cloud console, click Guide me:
Before you begin
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
- Install the Google Cloud CLI.
-
To initialize the gcloud CLI, run the following command:
gcloud init
-
Create or select a Google Cloud project.
-
Create a Google Cloud project:
gcloud projects create PROJECT_ID
Replace
PROJECT_ID
with a name for the Google Cloud project you are creating. -
Select the Google Cloud project that you created:
gcloud config set project PROJECT_ID
Replace
PROJECT_ID
with your Google Cloud project name.
-
-
Make sure that billing is enabled for your Google Cloud project.
-
Enable the Batch, Compute Engine, Logging and Cloud Storage APIs:
gcloud services enable batch.googleapis.com
compute.googleapis.com logging.googleapis.com storage.googleapis.com - Install the Google Cloud CLI.
-
To initialize the gcloud CLI, run the following command:
gcloud init
-
Create or select a Google Cloud project.
-
Create a Google Cloud project:
gcloud projects create PROJECT_ID
Replace
PROJECT_ID
with a name for the Google Cloud project you are creating. -
Select the Google Cloud project that you created:
gcloud config set project PROJECT_ID
Replace
PROJECT_ID
with your Google Cloud project name.
-
-
Make sure that billing is enabled for your Google Cloud project.
-
Enable the Batch, Compute Engine, Logging and Cloud Storage APIs:
gcloud services enable batch.googleapis.com
compute.googleapis.com logging.googleapis.com storage.googleapis.com -
Make sure that you and the job's service account have the required permissions to complete this tutorial. This tutorial uses default service account for a job, which is the Compute Engine default service account.
-
To get the permissions that you need to complete this tutorial, ask your administrator to grant you the following IAM roles:
-
To create, view, and delete jobs:
-
Batch Job Editor (
roles/batch.jobsEditor
) on the project -
Service Account User (
roles/iam.serviceAccountUser
) on the Compute Engine default service account
-
Batch Job Editor (
-
To create, view, and delete Cloud Storage buckets:
Storage Admin (
roles/storage.admin
) on the project -
To view logs from jobs:
Logs Viewer (
roles/logging.viewer
) on the project
For more information about granting roles, see Manage access to projects, folders, and organizations.
You might also be able to get the required permissions through custom roles or other predefined roles.
-
To create, view, and delete jobs:
-
To ensure that the Compute Engine default service account has the necessary permissions to complete this tutorial, ask your administrator to grant the Compute Engine default service account the following IAM roles:
-
Batch Agent Reporter (
roles/batch.agentReporter
) on the project -
To let jobs access Cloud Storage buckets:
Storage Admin (
roles/storage.admin
) on the project -
To let jobs generate logs in Logging:
Logs Writer (
roles/logging.logWriter
) on the project
-
Batch Agent Reporter (
-
-
Clone the
Batch git repository into the current directory:
git clone https://github.com/GoogleCloudPlatform/batch-samples.git
-
Go to the
transcoding
directory:cd batch-samples/transcoding/
Prepare job inputs
Create a Cloud Storage bucket:
gcloud storage buckets create gs://BUCKET_NAME
Replace
BUCKET_NAME
with a globally unique name for the bucket.The output is similar to the following:
Creating gs://BUCKET_NAME/...
Copy the
transcode.sh
script and the folder containing the video files to your Cloud Storage bucket:gcloud storage cp -R transcode.sh input gs://BUCKET_NAME
The output is similar to the following:
Copying file://transcode.sh to gs://BUCKET_NAME/transcode.sh Copying file://input/video-2.mp4 to gs://BUCKET_NAME/input/video-2.mp4 Copying file://input/video-1.mp4 to gs://BUCKET_NAME/input/video-1.mp4 Copying file://input/video-0.mp4 to gs://BUCKET_NAME/input/video-0.mp4 Completed files 4/4 | 37.5MiB/37.5MiB Average throughput: 48.4MiB/s
Create a job
In a text editor of your choice, open the
job.json
configuration file.Set the value of the
remotePath
field to the name of your Cloud Storage bucket:{ "taskGroups": [ { "taskSpec": { "runnables": [ { "script": { "text": "bash /mnt/share/transcode.sh" } } ], "computeResource": { "cpuMilli": 2000, "memoryMib": 2048 }, "volumes": [ { "gcs": { "remotePath": "BUCKET_NAME" }, "mountPath": "/mnt/share" } ], "maxRetryCount": 2, "maxRunDuration": "600s" }, "taskCount": 3, "parallelism": 3 } ], "allocationPolicy": { "instances": [ { "policy": { "machineType": "n2d-standard-4", "provisioningModel": "SPOT" } } ] }, "labels": { "department": "creative", "env": "testing" }, "logsPolicy": { "destination": "CLOUD_LOGGING" } }
Save your changes and close the text editor.
Create the
transcode
job:gcloud batch jobs submit transcode \ --config=job.json \ --location=us-central1
The output is similar to the following:
Job transcode-7a1654ca-211c-40e8-b0fb-8a00 was successfully submitted. ...
The job runs 3 tasks concurrently. Each task runs the
transcode.sh
script, which encodes 1 of 3 video files and uploads it to the Cloud Storage bucket.
Monitor the job
In the Google Cloud console, go to the Job list page.
In the Job name column, click transcode.
The Job details page opens.
Click the Events tab.
In the Events list section you can monitor the status of the
transcode
job. The time the job takes to finish being queued, scheduled, and running varies based on multiple factors. For this example, you can expect the job to be all finished in approximately 5 minutes.Optional: To update the page, click
Refresh.
Before proceeding to the next step, make sure that the status of the job is set to Succeeded. If your job fails, see Troubleshooting instead.
View the encoded videos
In the Google Cloud console, go to the Buckets page.
In the Name column, click BUCKET_NAME.
The Bucket details page opens.
In the Name column, click
output/, and then click one of the encoded video files.The Object details page opens.
To view the encoded video, click Preview, and then click
Play.
Clean up
To avoid incurring charges to your Google Cloud account for the resources used on this page, delete the Google Cloud project with the resources.
Delete the project
The easiest way to eliminate billing is to delete the project you used in this tutorial.
Delete a Google Cloud project:
gcloud projects delete PROJECT_ID
Delete individual resources
If you want to keep using the current project, then delete the individual resources used in this tutorial.
Delete the job
After the Batch job has finished running, delete the
transcode
job:
gcloud batch jobs delete transcode \
--location=us-central1
The output is similar to the following:
Job projects/example-project/locations/us-central1/jobs/transcode deletion is in progress
Deleting a job also deletes the job's details and history. The job's logs are automatically deleted at the end of the Cloud Logging log retention period.
Delete the bucket
If you no longer need the Cloud Storage bucket you used in this tutorial and its content, delete the bucket:
gcloud storage rm gs://BUCKET_NAME \
--recursive
The output is similar to the following:
Removing objects:
Removing gs://BUCKET_NAME/input/video-0.mp4#1694788495332395...
Removing gs://BUCKET_NAME/input/video-2.mp4#1694788495296173...
Removing gs://BUCKET_NAME/input/video-1.mp4#1694788495228839...
Removing gs://BUCKET_NAME/output/video-0.mp4#1694788495332395...
Removing gs://BUCKET_NAME/output/video-2.mp4#1694788495296173...
Removing gs://BUCKET_NAME/output/video-1.mp4#1694788495228839...
Removing gs://BUCKET_NAME/transcode.sh#1694788495039427...
Completed 4/4
Removing Buckets:
Removing gs://BUCKET_NAME/...
Completed 1/1
Delete the git repository
If you no longer need the Batch git repository that you cloned for this tutorial, you can delete it:
cd ../../ && rm -rf batch-samples
What's next
Learn how to get started with Batch.
Learn more about job creation and execution.