This document lists the quotas and limits that apply to Apache Kafka for BigQuery. For more information on quotas, see Virtual Private Cloud quotas.
A quota restricts how much of a shared Google Cloud resource your Google Cloud project can use, including hardware, software, and network components. Therefore, quotas are a part of a system that does the following:
- Monitors your use or consumption of Google Cloud products and services.
- Restricts your consumption of those resources, for reasons that include ensuring fairness and reducing spikes in usage.
- Maintains configurations that automatically enforce prescribed restrictions.
- Provides a means to request or make changes to the quota.
In most cases, when a quota is exceeded, the system immediately blocks access to the relevant Google resource, and the task that you're trying to perform fails. In most cases, quotas apply to each Google Cloud project and are shared across all applications and IP addresses that use that Google Cloud project.
There are also limits on Apache Kafka for BigQuery resources. These limits are unrelated to the quota system. Limits cannot be changed unless otherwise stated.
View and manage quotas
For a given project, you can use the Quotas & System Limits dashboard to view current quota limits and usage. You can also use this dashboard to do the following:
- Reduce your quota limits
- Initiate a process to apply for higher quota limits
Apache Kafka for BigQuery quotas
You can view and edit the quotas listed in the following table for each project in the Quotas & System Limits dashboard.
| Quota | Default quota limit | Description |
|---|---|---|
| Clusters per project per region | 5 | The maximum number of clusters allowed per project in a given region. |
| CPU capacity per project per region | 750 | The maximum CPU capacity allowed for brokers within a project in a given region. |
| Memory capacity per project per region | 6,000 GiB | The maximum memory capacity allowed for brokers within a project in a given region. |
| Cluster write requests per minute per project per region | 600 | The number of write requests for clusters, such as UpdateCluster and DeleteCluster, allowed per minute per project in a given region. This is not a quota on the rate at which producers can publish records. |
| Cluster create requests per minute per project per region | 1 | The number of CreateCluster requests allowed per minute per project in a given region. |
| Cluster read requests per minute per project per region. This is not a quota on the rate at which consumers can read records. | 6000 | The number of read requests for clusters, such as GetCluster and ListClusters, allowed per minute per project in a given region. |
| Operations requests per minute per project per region | 6000 | The number of requests for long-running operations allowed per minute per project in a given region. |
Note: Default quota limits can vary among projects depending on the project's history. The values quoted here are typical.
Request a quota increase
To increase or decrease most quotas, use the Google Cloud console. For more information, see Request a higher quota.
Apache Kafka for BigQuery limits
Apache Kafka for BigQuery reserves 100 GB of local SSD per CPU.