Set up an Apache Kafka for BigQuery project

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This document provides an overview of the procedures that you must follow to set up a Apache Kafka for BigQuery cluster.

Choose an interface option

With Apache Kafka for BigQuery, you can deploy, configure, and operate Kafka clusters by using a number of configuration options such as the following:

• Google Cloud console

• Google Cloud CLI

• Cloud APIs

• Terraform

• Client libraries.

Your choice of a configuration option depends on your use case.

If you're new to Google Cloud and want to test Apache Kafka for BigQuery, then use the Google Cloud console or the gcloud CLI.

The Apache Kafka for BigQuery client libraries use the Apache Kafka for BigQuery API. The Apache Kafka for BigQuery API and the other Google APIs are best for custom automation and the recommended way of accessing these is through the client libraries.

Use the latest version of the client library. The client libraries are constantly being updated with new features and bug fixes. Ensure that you are using the latest version of the client library for your language. For more information about Apache Kafka for BigQuery client libraries, see Overview of Apache Kafka for BigQuery client libraries.

Decide whether you need granular access control

The simplest and default way to manage authorization for Kafka clusters is with the Apache Kafka for BigQuery API and Identity and Access Management(IAM). However, IAM doesn't allow access controls on individual resources.

If you would like to manage access control to individual topics, you must manage Kafka ACLs. Set up your Kafka ACLs before you create any resources to avoid migration later.

For more information about access control for Apache Kafka for BigQuery, see the following:

• Authenticate to the Apache Kafka for BigQuery API.

• Authenticate to the open source Kafka API.

Workflow to create a Kafka cluster

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. Enable the Managed Kafka, Compute Engine, and Cloud DNS APIs.

   Enable the APIs

5. Create a service account:

   1. In the Google Cloud console, go to the Create service account page.

      Go to Create service account
   2. Select your project.

   3. In the Service account name field, enter a name. The Google Cloud console fills in the Service account ID field based on this name.

      In the Service account description field, enter a description. For example, Service account for quickstart.

   4. Click Create and continue.

   5. Grant the Managed Kafka Admin role to the service account.

      To grant the role, find the Select a role list, then select Managed Kafka Admin.

      Note: The Role field affects which resources the service account can access in your project. You can revoke these roles or grant additional roles later. In production environments, do not grant the Owner, Editor, or Viewer roles. Instead, grant a predefined role or custom role that meets your needs.
   6. Click Continue.

   7. In the Service account users role field, enter the identifier for the principal that will attach the service account to other resources, such as Compute Engine instances.

      This is typically the email address for a Google Account.

   8. Click Done to finish creating the service account.

      Do not close your browser window. You will use it in the next step.

6. Create a service account key:

   1. In the Google Cloud console, click the email address for the service account that you created.
   2. Click Keys.
   3. Click Add key, and then click Create new key.
   4. Click Create. A JSON key file is downloaded to your computer.
   5. Click Close.
7. Install the Google Cloud CLI.

8. To initialize the gcloud CLI, run the following command:

   gcloud init

   Note: If you installed the gcloud CLI previously, make sure you have the latest version by running gcloud components update.

9. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

10. Make sure that billing is enabled for your Google Cloud project.

11. Enable the Managed Kafka, Compute Engine, and Cloud DNS APIs.

    Enable the APIs

12. Create a service account:

    1. In the Google Cloud console, go to the Create service account page.

       Go to Create service account
    2. Select your project.

    3. In the Service account name field, enter a name. The Google Cloud console fills in the Service account ID field based on this name.

       In the Service account description field, enter a description. For example, Service account for quickstart.

    4. Click Create and continue.

    5. Grant the Managed Kafka Admin role to the service account.

       To grant the role, find the Select a role list, then select Managed Kafka Admin.

       Note: The Role field affects which resources the service account can access in your project. You can revoke these roles or grant additional roles later. In production environments, do not grant the Owner, Editor, or Viewer roles. Instead, grant a predefined role or custom role that meets your needs.
    6. Click Continue.

    7. In the Service account users role field, enter the identifier for the principal that will attach the service account to other resources, such as Compute Engine instances.

       This is typically the email address for a Google Account.

    8. Click Done to finish creating the service account.

       Do not close your browser window. You will use it in the next step.

13. Create a service account key:

    1. In the Google Cloud console, click the email address for the service account that you created.
    2. Click Keys.
    3. Click Add key, and then click Create new key.
    4. Click Create. A JSON key file is downloaded to your computer.
    5. Click Close.
14. Install the Google Cloud CLI.

15. To initialize the gcloud CLI, run the following command:

    gcloud init

    Note: If you installed the gcloud CLI previously, make sure you have the latest version by running gcloud components update.

You require the service account JSON key to later authenticate the Kafka consumer and producer applications. The process is described in the Quickstart.

16. Follow the procedures in the individual sections to complete the rest of the workflow:
1. Create a Kafka cluster.
2. Create a topic.
3. Configure the consumer and producer applications.

   For more information, see the Quickstart.