Build and run a Flex Template

---

Dataflow Flex Templates allow you to package a Dataflow pipeline for deployment. This tutorial shows you how to build a Dataflow Flex Template and then run a Dataflow job using that template.

Objectives

• Build a Dataflow Flex Template.
• Use the template to run a Dataflow job.

Costs

In this document, you use the following billable components of Google Cloud:

• Dataflow
• Cloud Storage
• Artifact Registry
• Cloud Build

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
2. Install the Google Cloud CLI.

3. To initialize the gcloud CLI, run the following command:

   gcloud init

4. Create or select a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   • Create a Google Cloud project:

     gcloud projects create PROJECT_ID

     Replace PROJECT_ID with a name for the Google Cloud project you are creating.

   • Select the Google Cloud project that you created:

     gcloud config set project PROJECT_ID

     Replace PROJECT_ID with your Google Cloud project name.

5. Make sure that billing is enabled for your Google Cloud project.

6. Enable the Dataflow, Compute Engine, Logging, Cloud Storage, Cloud Storage JSON, Resource Manager, Artifact Registry, and Cloud Build API:

   gcloud services enable dataflow compute_component logging storage_component storage_api cloudresourcemanager.googleapis.com artifactregistry.googleapis.com cloudbuild.googleapis.com

7. If you're using a local shell, then create local authentication credentials for your user account:

   gcloud auth application-default login

   You don't need to do this if you're using Cloud Shell.

8. Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/iam.serviceAccountUser

   gcloud projects add-iam-policy-binding PROJECT_ID --member="USER_IDENTIFIER" --role=ROLE

   • Replace PROJECT_ID with your project ID.

   • Replace USER_IDENTIFIER with the identifier for your user account. For example, user:[e-mail].

   • Replace ROLE with each individual role.
9. Install the Google Cloud CLI.

10. To initialize the gcloud CLI, run the following command:

    gcloud init

11. Create or select a Google Cloud project.

    Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

12. Make sure that billing is enabled for your Google Cloud project.

13. Enable the Dataflow, Compute Engine, Logging, Cloud Storage, Cloud Storage JSON, Resource Manager, Artifact Registry, and Cloud Build API:

    gcloud services enable dataflow compute_component logging storage_component storage_api cloudresourcemanager.googleapis.com artifactregistry.googleapis.com cloudbuild.googleapis.com

14. If you're using a local shell, then create local authentication credentials for your user account:

    gcloud auth application-default login

    You don't need to do this if you're using Cloud Shell.

15. Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/iam.serviceAccountUser

    gcloud projects add-iam-policy-binding PROJECT_ID --member="USER_IDENTIFIER" --role=ROLE

    • Replace PROJECT_ID with your project ID.

    • Replace USER_IDENTIFIER with the identifier for your user account. For example, user:[e-mail].

    • Replace ROLE with each individual role.

16. Grant roles to your Compute Engine default service account. Run the following command once for each of the following IAM roles:

    • roles/dataflow.admin
    • roles/dataflow.worker
    • roles/storage.objectAdmin
    • roles/artifactregistry.writer

    gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:[e-mail]" --role=SERVICE_ACCOUNT_ROLE

    Replace the following:

    • PROJECT_ID: your project ID
    • PROJECT_NUMBER your project number
    • SERVICE_ACCOUNT_ROLE: each individual role

Prepare the environment

Install the SDK and any requirements for your development environment.

Java

1. Download and install the Java Development Kit (JDK) version 11. Verify that the JAVA_HOME environment variable is set and points to your JDK installation.

2. Download and install Apache Maven by following Maven's installation guide for your specific operating system.

Python

Install the Apache Beam SDK for Python.

Go

Use Go's Download and install guide to download and install Go for your specific operating system. To learn which Go runtime environments are supported by Apache Beam, see Apache Beam runtime support.

Download the code sample.

Java

1. Clone the java-docs-samples repository.

   git clone https://github.com/GoogleCloudPlatform/java-docs-samples.git
   

2. Navigate to the code sample for this tutorial.

   cd java-docs-samples/dataflow/flex-templates/getting_started
   

3. Build the Java project into an Uber JAR file.

   mvn clean package

   This Uber JAR file has all the dependencies embedded in it. You can run this file as a standalone application with no external dependencies on other libraries.

Python

1. Clone the python-docs-samples repository.

   git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git
   

2. Navigate to the code sample for this tutorial.

   cd python-docs-samples/dataflow/flex-templates/getting_started
   

Go

1. Clone the golang-samples repository.

   git clone https://github.com/GoogleCloudPlatform/golang-samples.git
   

2. Navigate to the code sample for this tutorial.

   cd golang-samples/dataflow/flex-templates/wordcount
   

3. Compile the Go binary.

   CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o wordcount .

Create a Cloud Storage bucket

Use the gcloud storage buckets create command to create a Cloud Storage bucket:

gcloud storage buckets create gs://BUCKET_NAME

Replace BUCKET_NAME with a name for your Cloud Storage bucket. Cloud Storage bucket names must be globally unique and meet the bucket naming requirements.

Create an Artifact Registry repository

Create an Artifact Registry repository where you will push the Docker container image for the template.

1. Use the gcloud artifacts repositories create command to create a new Artifact Registry repository.

   gcloud artifacts repositories create REPOSITORY \
    --repository-format=docker \
    --location=LOCATION
   

   Replace the following:

   • REPOSITORY: a name for your repository. Repository names must be unique for each repository location in a project.
   • LOCATION: the regional or multi-regional location for the repository.

2. Use the gcloud auth configure-docker command to configure Docker to authenticate requests for Artifact Registry. This command updates your Docker configuration, so that you can connect with Artifact Registry to push images.

   gcloud auth configure-docker LOCATION-docker.pkg.dev
   

Flex Templates can also use images stored in private registries. For more information, see Use an image from a private registry.

Build the Flex Template

In this step, you use the gcloud dataflow flex-template build command to build the Flex Template.

A Flex Template consists of the following components:

• A Docker container image that packages your pipeline code. For Java and Python Flex Templates, the Docker image is built and pushed to your Artifact Registry repository when you run the gcloud dataflow flex-template build command.
• A template specification file. This file is a JSON document that contains the location of the container image plus metadata about the template, such as pipeline parameters.

The sample repository in GitHub contains the metadata.json file.

• java-docs-samples repository
• python-docs-samples repository
• golang-samples repository

To extend your template with additional metadata, you can create your own metadata.json file.

Java

gcloud dataflow flex-template build gs://BUCKET_NAME/getting_started-java.json \
 --image-gcr-path "LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/getting-started-java:latest" \
 --sdk-language "JAVA" \
 --flex-template-base-image JAVA11 \
 --metadata-file "metadata.json" \
 --jar "target/flex-template-getting-started-1.0.jar" \
 --env FLEX_TEMPLATE_JAVA_MAIN_CLASS="com.example.dataflow.FlexTemplateGettingStarted"

Replace the following:

• BUCKET_NAME: the name of the Cloud Storage bucket that you created earlier
• LOCATION: the location
• PROJECT_ID: the Google Cloud project ID
• REPOSITORY: the name of the Artifact Registry repository that you created earlier

Python

gcloud dataflow flex-template build gs://BUCKET_NAME/getting_started-py.json \
 --image-gcr-path "LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/getting-started-python:latest" \
 --sdk-language "PYTHON" \
 --flex-template-base-image "PYTHON3" \
 --metadata-file "metadata.json" \
 --py-path "." \
 --env "FLEX_TEMPLATE_PYTHON_PY_FILE=getting_started.py" \
 --env "FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE=requirements.txt"

Replace the following:

• BUCKET_NAME: the name of the Cloud Storage bucket that you created earlier
• LOCATION: the location
• PROJECT_ID: the Google Cloud project ID
• REPOSITORY: the name of the Artifact Registry repository that you created earlier

Go

1. Use the gcloud builds submit command to build the Docker image using a Dockerfile with Cloud Build. This command builds the file and pushes it to your Artifact Registry repository.

   gcloud builds submit --tag LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/dataflow/wordcount-go:latest .
   

   Replace the following:

   • LOCATION: the location
   • PROJECT_ID: the Google Cloud project ID
   • REPOSITORY: the name of the Artifact Registry repository that you created earlier

2. Use the gcloud dataflow flex-template build command to create a Flex Template named wordcount-go.json in your Cloud Storage bucket.

   gcloud dataflow flex-template build gs://BUCKET_NAME/samples/dataflow/templates/wordcount-go.json \
     --image "LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/dataflow/wordcount-go:latest" \
     --sdk-language "GO" \
     --metadata-file "metadata.json"

   Replace BUCKET_NAME with the name of the Cloud Storage bucket that you created earlier.

Run the Flex Template

In this step, you use the template to run a Dataflow job.

Java

1. Use the gcloud dataflow flex-template run command to run a Dataflow job that uses the Flex Template.

   gcloud dataflow flex-template run "getting-started-`date +%Y%m%d-%H%M%S`" \
    --template-file-gcs-location "gs://BUCKET_NAME/getting_started-java.json" \
    --parameters output="gs://BUCKET_NAME/output-" \
    --region "REGION"

   Replace the following:

   • BUCKET_NAME: the name of the Cloud Storage bucket that you created earlier
   • REGION: the region

2. To view the status of the Dataflow job in the Google Cloud console, go to the Dataflow Jobs page.

   Go to Jobs

If the job runs successfully, it writes the output to a file named gs://BUCKET_NAME/output--00000-of-00001.txt in your Cloud Storage bucket.

Python

1. Use the gcloud dataflow flex-template run command to run a Dataflow job that uses the Flex Template.

   gcloud dataflow flex-template run "getting-started-`date +%Y%m%d-%H%M%S`" \
    --template-file-gcs-location "gs://BUCKET_NAME/getting_started-py.json" \
    --parameters output="gs://BUCKET_NAME/output-" \
    --region "REGION"
   

   Replace the following:

   • BUCKET_NAME: the name of the Cloud Storage bucket that you created earlier
   • REGION: the region

2. To view the status of the Dataflow job in the Google Cloud console, go to the Dataflow Jobs page.

   Go to Jobs

If the job runs successfully, it writes the output to a file named gs://BUCKET_NAME/output--00000-of-00001.txt in your Cloud Storage bucket.

Go

1. Use the gcloud dataflow flex-template run command to run a Dataflow job that uses the Flex Template.

   gcloud dataflow flex-template run "wordcount-go-`date +%Y%m%d-%H%M%S`" \
    --template-file-gcs-location "gs://BUCKET_NAME/samples/dataflow/templates/wordcount-go.json" \
    --parameters output="gs://BUCKET_NAME/samples/dataflow/templates/counts.txt" \
    --region "REGION"
   

   Replace the following:

   • BUCKET_NAME: the name of the Cloud Storage bucket that you created earlier
   • REGION: the region

2. To view the status of the Dataflow job in the Google Cloud console, go to the Dataflow Jobs page.

   Go to Jobs

If the job runs successfully, it writes the output to a file named gs://BUCKET_NAME/samples/dataflow/templates/count.txt in your Cloud Storage bucket.

Note: If your job fails to run and the error message A Timeout in polling error message is displayed, see Troubleshoot Flex Template timeouts.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

Delete a Google Cloud project:

gcloud projects delete PROJECT_ID

Delete individual resources

1. Delete the Cloud Storage bucket and all the objects in the bucket.

   gcloud storage rm gs://BUCKET_NAME --recursive

2. Delete the Artifact Registry repository.

   gcloud artifacts repositories delete REPOSITORY \
       --location=LOCATION

3. Revoke the roles that you granted to the Compute Engine default service account. Run the following command once for each of the following IAM roles:
   • roles/dataflow.admin
   • roles/dataflow.worker
   • roles/storage.objectAdmin
   • roles/artifactregistry.writer

   gcloud projects remove-iam-policy-binding PROJECT_ID \
       --member=serviceAccount:[e-mail] \
       --role=SERVICE_ACCOUNT_ROLE

4. Optional: Revoke the authentication credentials that you created, and delete the local credential file.

   gcloud auth application-default revoke

5. Optional: Revoke credentials from the gcloud CLI.

   gcloud auth revoke

What's next

• Learn how to configure Flex Templates.
• See the list of Google-provided templates.