This document describes the access control options available to you in Apache Kafka for BigQuery. Apache Kafka for BigQuery uses Identity and Access Management (IAM) and open source Apache Kafka ACLs for access control.
Overview of IAM in Apache Kafka for BigQuery
In Apache Kafka for BigQuery, you can configure access control with IAM at the project level only. To manage access to specific topics or consumer groups, you must use open source Apache Kafka ACLs.
You have to use the Apache Kafka for BigQuery API to create and manage the cluster. The Apache Kafka for BigQuery API is managed by Google Cloud. For more information, see the Apache Kafka for BigQuery API documentation. You cannot use the open source Apache Kafka APIs for cluster management. For more information about these open source APIs, see the documentation.
Every Apache Kafka for BigQuery method requires a corresponding permission. For a list of the permissions and roles that IAM supports, see the next section.
Types of roles in Apache Kafka for BigQuery
Similar to other Google Cloud products, Apache Kafka for BigQuery supports three types of roles:
Basic roles: Basic roles are highly permissive roles that existed prior to the introduction of IAM. You can use basic roles to grant principals broad access to Google Cloud resources. In production environments, don't grant basic roles unless there is no alternative. Instead, grant the most limited predefined roles or custom roles that meet your needs. For more information about basic roles, see Basic roles.
Predefined roles: Predefined roles give granular access to specific Google Cloud resources. These roles are created and maintained by Google. Google automatically updates their permissions as necessary, such as when Google Cloud adds new features or services. For more information about predefined roles, see Predefined roles. The Apache Kafka for BigQuery predefined roles are included in a later part of this section.
Custom roles: Custom roles help you enforce the principle of least privilege, because they ensure that the principals in your organization have only the permissions that they need. Allow only a small number of highly trusted principals to edit custom roles. If a principal can edit custom roles in a project or organization, they can add any permission to any custom role in that project or organization. As a result, if you grant any custom role to the principal, they can use the custom role to get unlimited access. For more information about custom roles, see Custom roles.
Apache Kafka for BigQuery predefined roles
The following table lists the Apache Kafka for BigQuery predefined roles.
Role | Description | Permissions |
---|---|---|
Managed Kafka Viewerroles/managedkafka.viewer |
Read-only access to Apache Kafka for BigQuery
resources. Lowest-level resources where you can grant this role:
|
This role includes the following permissions:
|
Managed Kafka Client roleroles/managedkafka.client |
Provides access to connect to the Apache Kafka for BigQuery servers in a cluster. | This role includes the following permissions:
|
Managed Kafka Topic Editorroles/Managedkafka.topicEditor |
Provides read and write access to topic metadata. This role is
intended for developers who configure topics. Lowest-level
resources where you can grant this role:
|
This role includes the following permissions:
|
Managed Kafka ConsumerGroup Editorroles/managedkafka.consumerGroupEditor |
Provides read and write access to consumer group metadata. This
role is intended for developers.
Lowest-level resources where you can grant this role:
|
This role includes the following permissions:
|
Managed Kafka Cluster Editorroles/managedkafka.clusterEditor
|
Provides read and write access to Apache Kafka for BigQuery clusters. This
role is intended for organizations that separate the duties of
cluster administrators from application developers who work
with topics. Lowest-level resources where you can
grant this role:
|
This role includes the following permissions:
|
Managed Kafka Admin roleroles/managedkafka.admin |
Full access to Apache Kafka for BigQuery resources. Lowest-level resources
where you can grant this role:
|
This role includes the following permissions:
|
Permissions associated with Apache Kafka for BigQuery APIs
The following table lists the permissions required to call each REST method:
Method | Required permission(s) |
---|---|
projects.locations.clusters.list | managedkafka.clusters.list on the parent location. |
projects.locations.clusters.get | managedkafka.clusters.get on the requested cluster |
projects.locations.clusters.create | managedkafka.clusters.create on the parent location. |
projects.locations.clusters.update | managedkafka.clusters.update on the requested cluster |
projects.locations.clusters.delete | managedkafka.clusters.delete on the requested cluster |
projects.locations.clusters.topics.list | managedkafka.topics.list on the parent cluster |
projects.locations.clusters.topics.get | managedkafka.topics.get on the parent cluster |
projects.locations.clusters.topics.create | managedkafka.topics.create on the parent cluster |
projects.locations.clusters.topics.update | managedkafka.topics.update on the parent cluster |
projects.locations.clusters.topics.delete | managedkafka.topics.delete on the parent cluster |
projects.locations.clusters.consumerGroups.list | managedkafka.consumerGroups.list on the parent cluster |
projects.locations.clusters.consumerGroups.get | managedkafka.consumerGroups.get on the parent cluster |
projects.locations.clusters.consumerGroups.update | managedkafka.consumerGroups.update on the parent cluster |
projects.locations.clusters.consumerGroups.delete | managedkafka.consumerGroups.delete on the parent cluster
|
Set access control at the project level
To set access controls at the project level, see Manage access to projects, folders, and organizations.
IAM roles and permissions versus Kafka ACLs
Apache Kafka for BigQuery uses two levels of access control:
IAM roles from Google Cloud: These roles control who can manage your Apache Kafka for BigQuery cluster using Google Cloud APIs and tools. You manage these roles through the Google Cloud console or the IAM API. For more information, see the IAM documentation.
Kafka ACLs from open source Apache Kafka: These control access to Apache Kafka for BigQuery topics and operations within your cluster, enforced at the Kafka API level. You manage them using the Kafka authorization CLI.
If you're using Google Cloud to manage your cluster, IAM roles are the primary mechanism for controlling access.
You can restrict access to individual resources using Kafka ACLs in addition to IAM permissions.
For example, assume that you want to prevent a principal from editing topics. You can achieve this in two ways:
Fully through IAM: Deny the principal both the
roles/managedkafka.topicEditor
androles/managedkafka.client
roles. This completely restricts topic editing through Google Cloud APIs and prevents any Kafka API access.Using Kafka ACLs in conjunction with IAM: If the principal requires the
roles/managedkafka.client
role for actions like publishing to a topic, but you want to restrict them from editing topics, you must use Kafka ACLs. Specifically, restrict the following operations:Create
(for topic creation) at the cluster level.Alter
,AlterConfigs
,Delete
(for topic modification and deletion) at the topic level.
Therefore, you can choose the appropriate method based on whether the principal needs access to the Kafka APIs for other actions.
Configure Apache Kafka ACLs for granular access control
Apache Kafka for BigQuery enables the out-of-the-box StandardAuthorizer
,
which stores ACLs in the
KRaft-based Kafka cluster metadata. All access to a Apache Kafka for BigQuery
cluster by using open source Apache Kafka APIs is first subject to the IAM
check of the permission managedkafka.clusters.connect
, before any
Kafka ACLs are evaluated.
Apache Kafka ACLs have the format:
Principal P is [Allowed/Denied] Operation O From Host H on any
Resource R matching ResourcePattern RP.
The following is a list of important information about the format:
Principal (P): The user or service account requesting access.
Operation (O): The action performed such as Read, Write, or Create.
Host (H): The machine from which the request originates.
Resource (R): The Kafka resource being accessed such as Topic, Cluster, or Consumer Group.
ResourcePattern (RP): A pattern used to match specific resources such as "Topic:mytopic".
If RP doesn't match a specific resource R, then R has no associated ACLs.
Apache Kafka for BigQuery clusters are configured with the Apache Kafka property
allow.everyone.if.no.acl.found
set to true
by default.
So, with Apache Kafka for BigQuery clusters, if you don't explicitly
set ACLs on a resource, all principals can access that resource. If you enable
ACLs on a resource, only the authorized principals can access it.
For more information about adding, removing, and listing ACLs,
see Kafka authorization command line interface.
Apache Kafka for BigQuery gives its service agent administrative access
to your cluster. This access allows the service agent to perform essential
functions. Apache Kafka for BigQuery accomplishes this by modifying the
StandardAuthorizer
implementation. The modification grants the service agent
permissions similar to super users, except for Read and Write operations.
You cannot change this configuration.
If you want to restrict access to a topic, add ACLs using the Apache Kafka authorizer CLI.
Kafka Principals for Apache Kafka for BigQuery clusters are specified
as Google Cloud accounts, with the Kafka StandardAuthorizer
prefix "User:"
. For example, to grant access to the service account
test-kafka-client@test-project.iam.gserviceaccount.com
, use the Kafka principal
"User:test-kafka-client@test-project.iam.gserviceaccount.com"
.
Set up Kafka ACL administrator
To use Kafka ACLs, begin by setting up a non superuser ACL administrator
as the first ACL on the cluster. After the first ACL is set on the cluster and
allow.everyone.if.no.acl.found=true
, all principals not
allowed by the ACL are denied access to alter the cluster.
For example, to administer cluster ACLs with the service account
clusterAdmin@test.iam.gserviceaccount.com
, run this command on a
client that can communicate with the Apache Kafka for BigQuery cluster:
<path-to-your-kafka-installation>/bin/kafka-acls.sh \
--bootstrap-server BOOTSTRAP_ADDR \
--command-config PATH_TO_CLIENT_PROPERTIES \
--add \
--allow-principal User:clusterAdmin@test.iam.gserviceaccount.com \
--operation ALTER --cluster
The following is a list of important information about the command:
Replace
BOOTSTRAP_ADDR
with the address of your Kafka broker.PATH_TO_CLIENT_PROPERTIES
: A path to a configuration file that specifies your Kafka client configuration, including security settings. For an example of this file, see step 4 of the Local authentication server section.
After granting ALTER --cluster
to the ACL administrator, that user can
create and delete ACLs for all resources in a cluster.
The ALTER
operation on an Apache Kafka for BigQuery cluster lets a principal
to create and delete ACLs. To maintain security, it's crucial to restrict
this operation to a specific administrator.
Create a Kafka ACL to let a service account read from a topic
To create an ACL for the service account test-kafka-client@iam.gserviceaccount.com
to read from the topic topic-name
, run the following command on a
client that can communicate with the Apache Kafka for BigQuery cluster:
<path-to-your-kafka-installation>/bin/kafka-acls.sh \
--bootstrap-server BOOTSTRAP_ADDR \
--command-config PATH_TO_CLIENT_PROPERTIES \
--add \
--allow-principal "User:test-kafka-client@iam.gserviceaccount.com" \
--operation Read \
--topic topic-name
The following is a list of important information about the command:
test-kafka-client@iam.gserviceaccount.com
: The Google Cloud service account that needs read access.topic-name
: The name of the Apache Kafka for BigQuery topic to which you want to grant access.BOOTSTRAP_ADDR
: The bootstrap address of your Apache Kafka for BigQuery cluster. This address tells the ACL command how to connect to your Apache Kafka for BigQuery cluster.PATH_TO_CLIENT_PROPERTIES
: A path to a configuration file that specifies how to connect to your Apache Kafka for BigQuery securely. This file usually contains settings for authentication and encryption.
To remove read access, you can run the same command and
replace --add
with --remove
.
Create a Kafka ACL to let a service account write to a topic
To create an ACL for the service account test-kafka-client@iam.gserviceaccount.com
to write to the topic topic-name
, run the following command on a
client that can communicate with the Apache Kafka for BigQuery cluster:
<path-to-your-kafka-installation>/bin/kafka-acls.sh \
--bootstrap-server BOOTSTRAP_ADDR \
--command-config PATH_TO_CLIENT_PROPERTIES \
--add \
--allow-principal "User:test-kafka-client@iam.gserviceaccount.com" \
--operation Write \
--topic topic-name
The following is a list of important information about the command:
test-kafka-client@iam.gserviceaccount.com
: The Google Cloud service account that needs write access.topic-name
: The name of the Apache Kafka for BigQuery topic to which you want to grant access.BOOTSTRAP_ADDR
: The bootstrap address of your Apache Kafka for BigQuery cluster. This address tells the ACL command how to connect to your Apache Kafka for BigQuery cluster.PATH_TO_CLIENT_PROPERTIES
: A path to a configuration file that specifies how to connect to your Apache Kafka for BigQuery cluster securely. This file usually contains settings for authentication and encryption.
To remove write access, you can run the same command and
replace --add
with --remove
.
Create a Kafka ACL to deny topic modification
To prevent the service account test-kafka-client@iam.gserviceaccount.com
from modifying the topic Topic-Name
, run the following command on a client
that can communicate with the Apache Kafka for BigQuery cluster:
<path-to-your-kafka-installation>/bin/kafka-acls.sh \
--bootstrap-server BOOTSTRAP_ADDR \
--command-config PATH_TO_CLIENT_PROPERTIES \
--add \
--deny-principal "User:test-kafka-client@iam.gserviceaccount.com" \
--operation Alter \
--operation AlterConfigs \
--operation Delete \
--topic Topic-Name
The following is a list of important information about the command:
test-kafka-client@iam.gserviceaccount.com
: The Google Cloud service account that you want to restrict.Topic-Name
: The name of the Apache Kafka for BigQuery topic you want to protect from modification.BOOTSTRAP_ADDR
: The bootstrap address of your Google Cloud cluster.PATH_TO_CLIENT_PROPERTIES
: A path to a client configuration file containing settings for securely connecting to your Apache Kafka for BigQuery cluster.
To remove the restriction, run the same command but replace --add
with
--remove
to revoke the deny permission.