Access control with IAM and Kafka ACLs

This document describes the access control options available to you in Google Cloud Managed Service for Apache Kafka. Managed Service for Apache Kafka uses Identity and Access Management (IAM) and open source Apache Kafka ACLs for access control.

Overview of IAM in Managed Service for Apache Kafka

Managed Service for Apache Kafka uses two levels of access control:

Identity and Access Management roles: These roles control who can connect and manage your Managed Service for Apache Kafka cluster using Google Cloud APIs and tools. You manage these roles through the Google Cloud console or the IAM API. For more information, see the IAM documentation.
Apache Kafka ACLs: For more granular control over access to resources within a cluster, such as topics and consumer groups, use Kafka ACLs. Kafka ACLs let you manage who can access specific topics and perform actions like reading and writing messages.

You can further refine access control by using Kafka ACLs in conjunction with IAM to manage permissions both within a cluster and across clusters. While Kafka ACLs provide fine-grained control within the cluster, IAM lets you manage access to the cluster itself and the Managed Kafka API.

In Managed Service for Apache Kafka, you can configure access control with IAM Conditions at the level of the cluster, consumer group, or topic.

You have to use the Managed Kafka API to create and manage the cluster. The Managed Kafka API is managed by Google Cloud. For more information, see the Managed Kafka API documentation.

Every Managed Service for Apache Kafka method requires a corresponding permission. For a list of the permissions and roles that IAM supports, see the next section.

Types of roles in Managed Service for Apache Kafka

Similar to other Google Cloud products, Managed Service for Apache Kafka supports three types of roles:

Basic roles: Basic roles are highly permissive roles that existed prior to the introduction of IAM. You can use basic roles to grant principals broad access to Google Cloud resources. In production environments, don't grant basic roles unless there is no alternative. Instead, grant the most limited predefined roles or custom roles that meet your needs. For more information about basic roles, see Basic roles.
Predefined roles: Predefined roles give granular access to specific Google Cloud resources. These roles are created and maintained by Google. Google automatically updates their permissions as necessary, such as when Google Cloud adds new features or services. For more information about predefined roles, see Predefined roles. The Managed Service for Apache Kafka predefined roles are included in a later part of this section.
Custom roles: Custom roles help you enforce the principle of least privilege, because they ensure that the principals in your organization have only the permissions that they need. Allow only a small number of highly trusted principals to edit custom roles. If a principal can edit custom roles in a project or organization, they can add any permission to any custom role in that project or organization. As a result, if you grant any custom role to the principal, they can use the custom role to get unlimited access. For more information about custom roles, see Custom roles.

Set access control at the project level

To set access controls at the project level, see Manage access to projects, folders, and organizations.

Set access control at the cluster, consumer group, or topic level

Some Managed Service for Apache Kafka requests, such as cluster creation or updates, are long-running operations. To allow a principal to perform these actions, grant them access to managedkafka.googleapis.com/Operation resources in addition to the specific cluster resource.

This ensures the principal can both initiate the operation and monitor its progress.

The following is an example of an IAM condition being set on a topic named "iam-conditions-topic" { 'expression': 'resource.name.endsWith(\'iam-conditions-topic') 'title': 'SampleIAMCondition' }

NOTE Certain Managed Service for Apache Kafka requests return long running operations. To grant access at the cluster level, include access for all resources of type managedkafka.googleapis.com/Operation in addition to the per-cluster resource condition.

IAM roles and permissions versus Kafka ACLs

If you're using Google Cloud to manage your cluster, IAM roles are the primary mechanism for controlling access.

You can restrict access to individual resources using Kafka ACLs in addition to IAM permissions.

For example, assume that you want to prevent a principal from editing topics. You can achieve this in two ways:

Fully through IAM: Deny the principal both the roles/managedkafka.topicEditor and roles/managedkafka.client roles. This completely restricts topic editing through Google Cloud APIs and prevents any Kafka API access.
Using Kafka ACLs in conjunction with IAM: If the principal requires the roles/managedkafka.client role for actions like publishing to a topic, but you want to restrict them from editing topics, you must use Kafka ACLs. Specifically, restrict the following operations:
- Create (for topic creation) at the cluster level.
- Alter, AlterConfigs, Delete (for topic modification and deletion) at the topic level.

Therefore, you can choose the appropriate method based on whether the principal needs access to the Kafka APIs for other actions.

Configure Apache Kafka ACLs for granular access control

Managed Service for Apache Kafka enables the out-of-the-box StandardAuthorizer, which stores ACLs in the KRaft-based Kafka cluster metadata. All access to a Managed Service for Apache Kafka cluster by using open source Apache Kafka APIs is first subject to the IAM check of the permission managedkafka.clusters.connect, before any Kafka ACLs are evaluated.

Apache Kafka ACLs have the format:

Principal P is [Allowed/Denied] Operation O From Host H on any
Resource R matching ResourcePattern RP.

The following is a list of important information about the format:

Principal (P): The user or service account requesting access.
Operation (O): The action performed such as Read, Write, or Create.
Host (H): The machine from which the request originates.
Resource (R): The Kafka resource being accessed such as Topic, Cluster, or Consumer Group.
ResourcePattern (RP): A pattern used to match specific resources such as "Topic:mytopic".

If RP doesn't match a specific resource R, then R has no associated ACLs. Managed Service for Apache Kafka clusters are configured with the Apache Kafka property allow.everyone.if.no.acl.found set to true by default. So, with Managed Service for Apache Kafka clusters, if you don't explicitly set ACLs on a resource, all principals can access that resource. If you enable ACLs on a resource, only the authorized principals can access it. For more information about adding, removing, and listing ACLs, see Kafka authorization command line interface.

Managed Service for Apache Kafka gives its service agent administrative access to your cluster. This access allows the service agent to perform essential functions. Managed Service for Apache Kafka accomplishes this by modifying the StandardAuthorizer implementation. The modification grants the service agent permissions similar to super users, except for Read and Write operations. You cannot change this configuration.

If you want to restrict access to a topic, add ACLs using the Apache Kafka authorizer CLI.

Kafka Principals

Kafka Principals for Managed Service for Apache Kafka clusters are specified as Google Cloud accounts, with the Kafka StandardAuthorizer prefix "User:". For example, to grant access to the service account test-kafka-client@my-project.iam.gserviceaccount.com, use the Kafka principal "User:test-kafka-client@my-project.iam.gserviceaccount.com".

Kafka ACL principals must specify a user, a service account, or an individual IAM principal, but not a group or principal set. Kafka ACLs don't support resolving group memberships for Google Cloud principals. For example, authorizing a service account as a member of an allowed Google Group is not supported natively with ACLs.

To create an ACL that applies to all members of a Google Group or principal set, you can use a proxy service account principal and service account impersonation:

Create a service account to use as a proxy for the group. Grant the Google Group or principal set the Service Account Token Creator role on the service account; see Manage access to service accounts
Add Kafka ACLs for the proxy service account. An example principal would be: User:group-proxy@my-project.iam.gserviceaccount.com.
Use service account impersonation in Kafka clients to authenticate to Kafka as the service account. Google Cloud IAM authorizes an individual principal as a member of the group that's allowed to impersonate the proxy service account. And Kafka authorizes the proxy service account against existing ACLs in the cluster.

Set up Kafka ACL administrator

To use Kafka ACLs, begin by setting up a non superuser ACL administrator as the first ACL on the cluster. After the first ACL is set on the cluster and allow.everyone.if.no.acl.found=true, all principals not allowed by the ACL are denied access to alter the cluster.

For example, to administer cluster ACLs with the service account clusterAdmin@my-project.iam.gserviceaccount.com, run this command on a client that can communicate with the Managed Service for Apache Kafka cluster:

<path-to-your-kafka-installation>/bin/kafka-acls.sh \
    --bootstrap-server BOOTSTRAP_ADDR \
    --command-config PATH_TO_CLIENT_PROPERTIES \
    --add \
    --allow-principal User:clusterAdmin@my-project.iam.gserviceaccount.com \
    --operation  ALTER --cluster

The following is a list of important information about the command:

Replace BOOTSTRAP_ADDR with the address of your Kafka broker.
PATH_TO_CLIENT_PROPERTIES: A path to a configuration file that specifies your Kafka client configuration, including security settings. For an example of this file, see step 4 of the Local authentication server section.

After granting ALTER --cluster to the ACL administrator, that user can create and delete ACLs for all resources in a cluster.

The ALTER operation on a Managed Service for Apache Kafka cluster lets a principal to create and delete ACLs. To maintain security, it's crucial to restrict this operation to a specific administrator.

Allow reading from a topic

To create an ACL for the service account test-kafka-client@my-project.iam.gserviceaccount.com to read from the topic topic-name, run the following command on a client that can communicate with the Managed Service for Apache Kafka cluster:

<path-to-your-kafka-installation>/bin/kafka-acls.sh \
    --bootstrap-server BOOTSTRAP_ADDR  \
    --command-config PATH_TO_CLIENT_PROPERTIES \
    --add \
    --allow-principal "User:test-kafka-client@my-project.iam.gserviceaccount.com" \
    --operation Read \
    --topic topic-name

The following is a list of important information about the command:

test-kafka-client@my-project.iam.gserviceaccount.com: The Google Cloud service account that needs read access.
topic-name: The name of the Managed Service for Apache Kafka topic to which you want to grant access.
BOOTSTRAP_ADDR: The bootstrap address of your Managed Service for Apache Kafka cluster. This address tells the ACL command how to connect to your Managed Service for Apache Kafka cluster.
PATH_TO_CLIENT_PROPERTIES: A path to a configuration file that specifies how to connect to your Managed Service for Apache Kafka securely. This file usually contains settings for authentication and encryption.

To remove read access, you can run the same command and replace --add with --remove.

Allow writing to a specific topic

To create an ACL for the service account test-kafka-client@my-project.iam.gserviceaccount.com to write to the topic topic-name, run the following command on a client that can communicate with the Managed Service for Apache Kafka cluster:

<path-to-your-kafka-installation>/bin/kafka-acls.sh \
    --bootstrap-server BOOTSTRAP_ADDR  \
    --command-config PATH_TO_CLIENT_PROPERTIES \
    --add \
    --allow-principal "User:test-kafka-client@my-project.iam.gserviceaccount.com" \
    --operation Write \
    --topic topic-name

The following is a list of important information about the command:

test-kafka-client@my-project.iam.gserviceaccount.com: The Google Cloud service account that needs write access.
topic-name: The name of the Managed Service for Apache Kafka topic to which you want to grant access.
BOOTSTRAP_ADDR: The bootstrap address of your Managed Service for Apache Kafka cluster. This address tells the ACL command how to connect to your Managed Service for Apache Kafka cluster.
PATH_TO_CLIENT_PROPERTIES: A path to a configuration file that specifies how to connect to your Managed Service for Apache Kafka cluster securely. This file usually contains settings for authentication and encryption.

To remove write access, you can run the same command and replace --add with --remove.

Deny modifying a topic

To prevent the service account test-kafka-client@my-project.iam.gserviceaccount.com from modifying the topic Topic-Name, run the following command on a client that can communicate with the Managed Service for Apache Kafka cluster:

<path-to-your-kafka-installation>/bin/kafka-acls.sh \
    --bootstrap-server BOOTSTRAP_ADDR  \
    --command-config PATH_TO_CLIENT_PROPERTIES \
    --add \
    --deny-principal "User:test-kafka-client@my-project.iam.gserviceaccount.com" \
    --operation Alter \
    --operation AlterConfigs \
    --operation Delete \
    --topic Topic-Name

The following is a list of important information about the command:

test-kafka-client@my-project.iam.gserviceaccount.com: The Google Cloud service account that you want to restrict.
Topic-Name: The name of the Managed Service for Apache Kafka topic you want to protect from modification.
BOOTSTRAP_ADDR: The bootstrap address of your Google Cloud cluster.
PATH_TO_CLIENT_PROPERTIES: A path to a client configuration file containing settings for securely connecting to your Managed Service for Apache Kafka cluster.

To remove the restriction, run the same command but replace --add with --remove to revoke the deny permission.

Managed Service for Apache Kafka predefined roles

The following table lists the Managed Service for Apache Kafka predefined roles.

Role	Description	Permissions
Managed Kafka Viewer `roles/managedkafka.viewer`	Read-only access to Managed Service for Apache Kafka resources. Lowest-level resources where you can grant this role: Cluster Topic ConsumerGroup	This role includes the following permissions: resourcemanager.projects.get resourcemanager.projects.list serviceusage.quotas.get serviceusage.services.get serviceusage.services.list serviceusage.consumerpolicy.get serviceusage.effectivepolicy.get serviceusage.groups.list serviceusage.groups.listMembers serviceusage.groups.listFlattenedMembers serviceusage.reverseclosure.get serviceusage.values.check serviceusage.values.fetchValueInfo serviceusage.values.fetchServiceApis managedkafka.operations.list managedkafka.operations.get managedkafka.locations.list managedkafka.locations.get managedkafka.clusters.list managedkafka.clusters.get managedkafka.consumerGroups.list managedkafka.consumerGroups.get managedkafka.topics.list managedkafka.topics.get
Managed Kafka Client role `roles/managedkafka.client`	Provides access to connect to the Managed Service for Apache Kafka servers in a cluster.	This role includes the following permissions: managedkafka.clusters.connect
Managed Kafka Topic Editor `roles/Managedkafka.topicEditor`	Provides read and write access to topic metadata. This role is intended for developers who configure topics. Lowest-level resources where you can grant this role: Topic	This role includes the following permissions: managedkafka.topics.create managedkafka.topics.update managedkafka.topics.delete This role includes the following roles: `roles/managedkafka.viewer`
Managed Kafka ConsumerGroup Editor `roles/managedkafka.consumerGroupEditor`	Provides read and write access to consumer group metadata. This role is intended for developers. Lowest-level resources where you can grant this role: ConsumerGroup	This role includes the following permissions: managedkafka.consumerGroups.create managedkafka.consumerGroups.update managedkafka.consumerGroups.delete This role includes the following roles: `roles/managedkafka.viewer`
Managed Kafka Cluster Editor `roles/managedkafka.clusterEditor`	Provides read and write access to Managed Service for Apache Kafka clusters. This role is intended for organizations that separate the duties of cluster administrators from application developers who work with topics. Lowest-level resources where you can grant this role: Cluster	This role includes the following permissions: managedkafka.clusters.create managedkafka.clusters.update managedkafka.clusters.delete This role includes the following roles: `roles/managedkafka.viewer`
Managed Kafka Admin role `roles/managedkafka.admin`	Full access to Managed Service for Apache Kafka resources. Lowest-level resources where you can grant this role: Project Cluster ConsumerGroup Topic	This role includes the following permissions: managedkafka.operations.delete managedkafka.operations.cancel This role includes the following roles: `roles/managedkafka.client` `roles/managedkafka.topicEditor` `roles/managedkafka.consumerGroupEditor` `roles/managedkafka.clusterEditor`

Permissions associated with Managed Kafka APIs

The following tables outline the required permissions for each Managed Kafka API method related to cluster, topics, and consumer groups. It shows which IAM permission is needed to call each method, along with a description of what the method does.

Permissions for clusters

Method	Required permission(s)	Description
projects.locations.clusters.list	`managedkafka.clusters.list` on the parent location.	Lists all the Kafka clusters in a given location.
projects.locations.clusters.get	`managedkafka.clusters.get` on the requested cluster	Gets the details of a specific Kafka cluster.
projects.locations.clusters.create	`managedkafka.clusters.create` on the parent location.	Creates a new Kafka cluster in a given location.
projects.locations.clusters.update	`managedkafka.clusters.update` on the requested cluster	Updates the configuration of an existing Kafka cluster.
projects.locations.clusters.delete	`managedkafka.clusters.delete` on the requested cluster	Deletes a Kafka cluster.

Permissions for topics

Method	Required permission(s)	Description
projects.locations.clusters.topics.list	`managedkafka.topics.list` on the parent cluster	Lists all the topics in a given Kafka cluster.
projects.locations.clusters.topics.get	`managedkafka.topics.get` on the parent cluster	Gets the details of a specific topic in a Kafka cluster.
projects.locations.clusters.topics.create	`managedkafka.topics.create` on the parent cluster	Creates a new topic in a Kafka cluster.
projects.locations.clusters.topics.update	`managedkafka.topics.update` on the parent cluster	Updates the configuration of an existing topic in a Kafka cluster.
projects.locations.clusters.topics.delete	`managedkafka.topics.delete` on the parent cluster	Deletes a topic from a Kafka cluster.

Permissions for consumer groups

Method	Required permission(s)	Description
projects.locations.clusters.consumerGroups.list	`managedkafka.consumerGroups.list` on the parent cluster	Lists all the consumer groups in a given Kafka cluster.
projects.locations.clusters.consumerGroups.get	`managedkafka.consumerGroups.get` on the parent cluster	Gets the details of a specific consumer group in a Kafka cluster.
projects.locations.clusters.consumerGroups.update	`managedkafka.consumerGroups.update` on the parent cluster	Updates the configuration of an existing consumer group in a Kafka cluster.
projects.locations.clusters.consumerGroups.delete	`managedkafka.consumerGroups.delete` on the parent cluster	Deletes a consumer group from a Kafka cluster.

What's next

Apache Kafka® is a registered trademark of The Apache Software Foundation or its affiliates in the United States and/or other countries.