Provision Apache Kafka for BigQuery resources with Terraform

HashiCorp Terraform is an infrastructure-as-code (IaC) tool that lets you provision and manage cloud infrastructure. Terraform provides plugins called providers that let you interact with cloud providers and other APIs. You can use the Terraform provider for Google Cloud to provision and manage Google Cloud resources, including Apache Kafka for BigQuery.

How Terraform works

Terraform has a declarative and configuration-oriented syntax, which you can use to describe the infrastructure that you want to provision in your Google Cloud project. After you author this configuration in one or more Terraform configuration files, you can use the Terraform CLI to apply this configuration to your Apache Kafka for BigQuery resources.

The following steps explain how Terraform works:

  1. You describe the infrastructure you want to provision in a Terraform configuration file. You don't need to write code describing how to provision the infrastructure. Terraform provisions the infrastructure for you.
  2. You run the terraform plan command, which evaluates your configuration and generates an execution plan. You can review the plan and make changes as needed.

  3. You run the terraform apply command, which performs the following actions:

    1. It provisions your infrastructure based on your execution plan by invoking the corresponding Apache Kafka for BigQuery APIs in the background.
    2. It creates a Terraform state file, which is a JSON file that maps the resources in your configuration file to the resources in the real-world infrastructure. Terraform uses this file to keep a record of the most recent state of your infrastructure, and to determine when to create, update, and destroy resources.

    3. When you run terraform apply, Terraform uses the mapping in the state file to compare the existing infrastructure to the code, and make updates as necessary:

      • If a resource object is defined in the configuration file, but doesn't exist in the state file, Terraform creates it.
      • If a resource object exists in the state file, but has a different configuration from your configuration file, Terraform updates the resource to match your configuration file.
      • If a resource object in the state file matches your configuration file, Terraform leaves the resource unchanged.

Terraform-based guides for Apache Kafka for BigQuery

The following table lists all Terraform-based how-to guides and tutorials for Apache Kafka for BigQuery:

Guide Details
Create an Apache Kafka for BigQuery cluster This guide explains how to create an Apache Kafka for BigQuery cluster using Terraform.
Create an Apache Kafka for BigQuery topic This guide explains how to create an Apache Kafka for BigQuery topic using Terraform.

Terraform resources for Apache Kafka for BigQuery

Resources are the fundamental elements in the Terraform language. Each resource block describes one or more infrastructure objects, such as virtual networks or compute instances.

The following table lists the Terraform resources available for Apache Kafka for BigQuery:

Apache Kafka for BigQuery service Terraform resources
Manage a cluster Managed Kafka cluster
Manage a topic Managed Kafka topic

What's next