Create and view a Managed Service for Apache Kafka cluster

Create and view a Managed Service for Apache Kafka cluster

Learn how to create a Managed Service for Apache Kafka cluster with a topic, and view details about the cluster. You can use the Google Cloud console or the Google Cloud CLI to complete this tutorial.

Before you begin

Console

Sign in to your Google Account.
If you don't already have one, sign up for a new account.
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.
Go to project selector
Verify that billing is enabled for your Google Cloud project.
Enable the Managed Kafka API.
Enable the API
Make sure that you have the following role or roles on the project: Managed Kafka Cluster Editor, Managed Kafka Topic Editor
Check for the roles
1. In the Google Cloud console, go to the IAM page.

   Go to IAM
2. Select the project.

3. In the Principal column, find all rows that identify you or a group that you're included in. To learn which groups you're included in, contact your administrator.

4. For all rows that specify or include you, check the Role column to see whether the list of roles includes the required roles.
Grant the roles
1. In the Google Cloud console, go to the IAM page.

   Go to IAM
2. Select the project.
3. Click Grant access.

4. In the New principals field, enter your user identifier. This is typically the email address for a Google Account.

5. In the Select a role list, select a role.
6. To grant additional roles, click Add another role and add each additional role.
7. Click Save.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Managed Kafka API.

Enable the API

Make sure that you have the following role or roles on the project: Managed Kafka Cluster Editor, Managed Kafka Topic Editor

Check for the roles

1. In the Google Cloud console, go to the IAM page.

   Go to IAM
2. Select the project.

3. In the Principal column, find all rows that identify you or a group that you're included in. To learn which groups you're included in, contact your administrator.

4. For all rows that specify or include you, check the Role column to see whether the list of roles includes the required roles.

Grant the roles

1. In the Google Cloud console, go to the IAM page.

   Go to IAM
2. Select the project.
3. Click Grant access.

4. In the New principals field, enter your user identifier. This is typically the email address for a Google Account.

5. In the Select a role list, select a role.
6. To grant additional roles, click Add another role and add each additional role.
7. Click Save.

gcloud

Sign in to your Google Account.
If you don't already have one, sign up for a new account.
Install the Google Cloud CLI.
If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.
To initialize the gcloud CLI, run the following command:
gcloud init
Create or select a Google Cloud project.
Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.
• Create a Google Cloud project:

  gcloud projects create PROJECT_ID

  Replace PROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set project PROJECT_ID

  Replace PROJECT_ID with your Google Cloud project name.
Verify that billing is enabled for your Google Cloud project.
Enable the Managed Kafka API:
gcloud services enable managedkafka.googleapis.com
Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/managedkafka.clusterEditor, roles/managedkafka.topicEditor
gcloud projects add-iam-policy-binding PROJECT_ID --member="user:USER_IDENTIFIER" --role=ROLE
Replace the following:
• PROJECT_ID: your project ID.
• USER_IDENTIFIER: the identifier for your user account—for example, [e-mail].
• ROLE: the IAM role that you grant to your user account.

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

To initialize the gcloud CLI, run the following command:

gcloud init

Create or select a Google Cloud project.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

• Create a Google Cloud project:

  gcloud projects create PROJECT_ID

  Replace PROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set project PROJECT_ID

  Replace PROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

Enable the Managed Kafka API:

gcloud services enable managedkafka.googleapis.com

Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/managedkafka.clusterEditor, roles/managedkafka.topicEditor

gcloud projects add-iam-policy-binding PROJECT_ID --member="user:USER_IDENTIFIER" --role=ROLE

Replace the following:

• PROJECT_ID: your project ID.
• USER_IDENTIFIER: the identifier for your user account—for example, [e-mail].
• ROLE: the IAM role that you grant to your user account.

Create a cluster

Console

1. Go to the Managed Service for Apache Kafka > Clusters page.

   Go to Clusters

2. Click Create.

3. In the Cluster name box, enter a name for the cluster.

4. In the Region list, select a location for the cluster.

5. For Network configuration, configure the subnet where the cluster is accessible:

   1. For Project, select your project.
   2. For Network, select the VPC network.
   3. For Subnet, select the subnet.
   4. Click Done.

6. Click Create.

After you click Create, the cluster state is Creating. When the cluster is ready, the state is Active.

gcloud

To create a Kafka cluster, run the managed-kafka clusters create command.

gcloud managed-kafka clusters create KAFKA_CLUSTER \
--location=REGION \
--cpu=3 \
--memory=3GiB \
--subnets=projects/PROJECT_ID/regions/REGION/subnetworks/SUBNET_NAME \
--async

Replace the following:

• KAFKA_CLUSTER: a name for the Kafka cluster
• REGION: the location of the cluster
• PROJECT_ID: your project ID
• SUBNET_NAME: the subnet where you want to create the cluster, for example default

For information about supported locations, see Managed Service for Apache Kafka locations.

The command runs asynchronously and returns an operation ID:

Check operation [projects/PROJECT_ID/locations/REGION/operations/OPERATION_ID] for status.

To track the progress of the create operation, use the gcloud managed-kafka operations describe command:

gcloud managed-kafka operations describe OPERATION_ID \
  --location=REGION

When the cluster is ready, the output from this command includes the entry state: ACTIVE. For more information, see Monitor the cluster creation operation.

After you follow these steps, it can take up to 30 minutes for the cluster to be ready.

Optionally, while you wait for this operation to complete, look at the tutorials listed in What's next. They show how to produce messages for the cluster using various methods. Once the cluster is ready, you can complete any of those tutorials.

Create a topic

When the Managed Service for Apache Kafka cluster is ready, create a topic.

Console

1. Go to the Managed Service for Apache Kafka > Clusters page.

   Go to Clusters

2. Click the name of the cluster.

3. In the cluster details page, click Create Topic.

4. In the Topic name box, enter a name for the topic.

5. Click Create.

gcloud

To create a Kafka topic, run the managed-kafka topics create command.

gcloud managed-kafka topics create KAFKA_TOPIC_NAME \
--cluster=KAFKA_CLUSTER \
--location=REGION \
--partitions=10 \
--replication-factor=3

Replace the following:

• KAFKA_TOPIC_NAME: the name of the Kafka topic to create
• KAFKA_CLUSTER: the name of the Kafka cluster
• REGION: the region where you created the Kafka cluster

View the cluster

Once the cluster is ready, you can view information about the cluster, including the topics, consumer groups, networks, and other details.

Console

1. Go to the Managed Service for Apache Kafka > Clusters page.

   Go to Clusters

2. Click the name of the cluster. The Cluster details page shows information about the cluster. The Resources tab shows the resources associated with the cluster, including topics.

3. To view details about a topic, select the Resources tab and click the topic name. The Topic details page shows information about the topic, such as the number of partitions and replicas.

4. To return to the Cluster details page, click Back to parent page.

gcloud

1. To view information about the cluster, run the gcloud managed-kafka clusters describe command.

   gcloud managed-kafka clusters describe KAFKA_CLUSTER \
     --location=REGION
   

   Replace the following:

   • KAFKA_CLUSTER: the name of the cluster
   • REGION: the region where you created the cluster

   The output of this command is a list of cluster properties, similar to the following:

   bootstrapAddress: BOOTSTRAP_ADDRESS
   capacityConfig:
     memoryBytes: '3221225472'
     vcpuCount: '3'
   createTime: 'TIMESTAMP'
   

2. To get a list of topics in the cluster, run the gcloud managed-kafka topics list command.

   gcloud managed-kafka topics list KAFKA_CLUSTER \
     --location=REGION \
     --format="value(name)"
   

   The output includes the topics that you created, as well as topics that Managed Service for Apache Kafka creates for internal use.

3. To view information about a topic, run the gcloud managed-kafka topics describe command.

   gcloud managed-kafka topics describe TOPIC_NAME \
     --cluster=KAFKA_CLUSTER \
     --location=REGION
   

   Replace TOPIC_NAME with the name of the topic.

   The output of this command is a list of topic properties, similar to the following:

    name: projects/PROJECT_ID/locations//clusters/KAFKA_CLUSTER/topics/TOPIC_NAME
    partitionCount: 10
    replicationFactor: 3
   

Get the bootstrap address

Kafka clients use the cluster's bootstrap address to establish a connection with the cluster. To get the bootstrap address, perform the following steps:

Console

1. Go to the Managed Service for Apache Kafka > Clusters page.

   Go to Clusters

2. Click the name of the cluster.

3. Select the Configurations tab.

4. The bootstrap URL is listed under Bootstrap URL. Click Copy to copy the value.

gcloud

To get the bootstrap URL, run the managed-kafka clusters describe command.

gcloud managed-kafka clusters describe KAFKA_CLUSTER \
  --location=REGION \
  --format="value(bootstrapAddress)"

Replace the following:

• KAFKA_CLUSTER: the name of the cluster
• REGION: the region where you created the cluster

You are now ready to send messages to the cluster. For more information, see What's next in this document.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used on this page, delete the Google Cloud project with the resources.

Console

Delete the cluster.

1. Go to the Managed Service for Apache Kafka > Clusters page.

   Go to Clusters

2. Select the Kafka cluster and click Delete.

gcloud

1. To delete the cluster, use the gcloud managed-kafka clusters delete command.

   gcloud managed-kafka clusters delete KAFKA_CLUSTER \
     --location=REGION --async
   

What's next

Produce messages for Kafka, either by using the Kafka command-line tools, or by creating a client application:

• Use the Kafka command-line tools

• Develop a Java producer application

• Develop a Python producer application