This page provides recommendations to help plan your config management architecture on Config Controller instances and keep your Google Cloud resource creation and management within service-level objectives (SLO).
This page is for Admins and architects and Operators who manage the lifecycle of the underlying tech infrastructure and plan capacity and infrastructure needs. To learn more about common roles and example tasks that we reference in Google Cloud content, see Common GKE Enterprise user roles and tasks.
Use namespaced mode
We recommended using Config Connector in namespaced mode because it can be easier to manage high numbers of resources. You can set each namespace to correspond to a single namespace, which can help manage quotas and configurations, since resources have read and write quotas per project. Starting in version 1.119.0, you can increase reconciliation rate limits by namespace. By increasing rate limits, you can allow reconciliation for more than 10,000 resources per namespace in a 10 minute interval. Both Config Connector and Config Sync support namespaced mode, which allows for mapping each namespace to a single Google Cloud project.
Scalability targets
The following table represents values we regularly test. We know that Config Connector can handle larger numbers. We have shown that 30,000 resources can be managed in a single namespace. However there is some tuning needed to make this work. We recommend you review the Namespaced mode for suggestions on these changes.
Config Controller scalability targets are groups of resources tested by Google and with the use of Config Sync GitOps. You can use these targets to help plan your config management architecture.
These targets are not hard limits. Stretching the amount of one resource kind won't necessarily make the Config Controller instance unavailable, but it could reduce the total amount of other resource kinds in the same Config Controller instance that you can deploy.
The tables on this page are intended as references and are not exhaustive.
Single namespace
The following example shows a Config Controller instance with one Config Connector namespace in the cluster. Config Connector can create and manage the following number of resources in that namespace:
Resource type |
Suggested limit |
|
450 |
|
2,250 |
|
2,500 |
|
5,000 |
|
50 |
|
200 |
|
2,500 |
|
7,500 |
Multiple namespaces
The following example shows a Config Controller instance with 50 Config Connector namespaces in a cluster. Config Connector can create and manage the following number of resources in each namespace:
Resource type |
Suggested limit |
|
9 |
SQLDatabase |
45 |
SQLUser |
45 |
StorageBucket |
100 |
ContainerCluster |
1 |
ContainerNodepool |
4 |
IAMServiceAccount |
50 |
IAMPartialPolicy |
150 |
Config Connector namespaces
Config Controller uses Config Connector namespaced mode by default. The following tables shows an example of the number of Config Connector namespaces you can have in a single Config Connector instance.
|
Number of nodes |
Number of Config Connector namespaces |
/18 |
64 |
600 |
/19 |
32 |
300 |
/20 (default and recommended) |
16 |
120 |
/21 |
8 |
60 |
Checking scalability targets
You can use the following resources to help determine if you have reached scalability targets.
Google Cloud API Quotas
You can view your Google Cloud API quotas in the Google Cloud console. When some quotas are close to their limits, consider Sharding API quota by Google Cloud projects. To learn more about monitoring and alerting on quota metrics, see Monitor and alert on quota metrics.
Config Connector memory use
You can view your Config Connector memory use in the GKE monitoring dashboard. When Config Connector's memory use is close to its limit, consider Sharding by namespace.
Scaling up Config Controller
If you have reached scalability targets, you should consider further scaling up your Config Controller instances. This section outlines different methods you can use to scale up your Config Controller instances.
Sharding by namespace
If you reach a scalability target with a single Config Connector namespace, you can Configure Config Connector to manage resources in your namespaces.
Each namespace uses its own service accounts and operator workloads, which allows Config Connector to manage your resources at scale. If you use one Config Connector instance to manage multiple Google Cloud projects, you can use one Config Connector namespace to manage each Google Cloud project.
Sharding API quota by Google Cloud projects
If you reach a scalability target due to reaching Google Cloud API Quotas, you can bind different IAM service accounts owned by different Google Cloud projects to different namespaces where Config Connector is installed in namespaced mode. You can then split your resources into different projects.
Sharding by Config Connector instances
If you reach a scalability target with multiple Config Connector namespaces, you can create and use more than one Config Controller instance. With more than one Config Controller instance, you can shard your resource config management, for example, by different development environments, application teams, or GitOps directories within your organization.
Other scalability considerations
Google Cloud API quotas
If you've run into errors indicating that you've exceeded the API quota limit, you might have created too many Config Connector resources of the same Kind under the same project. Those resources can generate too many API requests to the same API endpoint because of the reconciliation strategy in Config Connector.
To resolve this, you can either shard API quota by Google Cloud project or request a higher quota limit.
GKE limitations
Since Config Controller is built on top of GKE, there are limitations from GKE that you should consider. The following sections cover specific considerations related to Config Controller. For more information about general limits and best practices for large GKE clusters, see Plan for large GKE clusters.
Kubernetes service account limit
The number of
Kubernetes Service Accounts (KSA)
created in a single GKE cluster should not exceed 3,000 because you
might encounter a
gke-metadata-server Pod crash issue.
Whenever you add a Config Connector namespace, it also creates a Kubernetes service account.
GKE control plane performance issues
The GKE cluster's control plane can have performance issues if a Config Controller instance has too many Config Connector namespaces. You should limit the number of Config Connector namespaces to under 500 per cluster.
Whenever you add a Config Connector namespace, it also creates a controller pod.
What's next
- Learn how to shard Config Controller
- Get help with troubleshooting Config Controller