Config Controller scalability guidelines

This page provides recommendations to help plan your config management architecture on Config Controller instances and keep your Google Cloud resource creation and management within service-level objectives (SLO).

Scalability targets

Config Controller scalability targets are groups of resources tested by Google and with the use of Config Sync GitOps. You can use these targets to help plan your config management architecture.

These targets are not hard limits. Stretching the amount of one resource kind won't necessarily make the Config Controller instance unavailable, but it could reduce the total amount of other resource kinds in the same Config Controller instance that you can deploy.

The tables on this page are intended as references and are not exhaustive.

Single namespace

The following example shows a Config Controller instance with one Config Connector namespace in the cluster. Config Connector can create and manage the following number of resources in that namespace:

Resource type

Suggested limit

SQLInstance

450

SQLDatabase

2,250

SQLUser

2,500

StorageBucket

5,000

ContainerCluster

50

ContainerNodepool

200

IAMServiceAccount

2,500

IAMPartialPolicy

7,500

Multiple namespaces

The following example shows a Config Controller instance with 50 Config Connector namespaces in a cluster. Config Connector can create and manage the following number of resources in each namespace:

Resource type

Suggested limit

SQLInstance

9
SQLDatabase 45
SQLUser 45
StorageBucket 100
ContainerCluster 1
ContainerNodepool 4
IAMServiceAccount 50
IAMPartialPolicy 150

Config Connector namespaces

Config Controller uses Config Connector namespaced mode by default. The following tables shows an example of the number of Config Connector namespaces you can have in a single Config Connector instance.

--cluster-ipv4-cidr-block

Number of nodes

Number of Config Connector namespaces

/18

64

600

/19

32

300

/20 (default and recommended)

16

120

/21

8

60

Checking scalability targets

You can use the following resources to help determine if you have reached scalability targets.

Google Cloud API Quotas

You can view your Google Cloud API quotas in the Google Cloud console. When some quotas are close to their limits, consider Sharding API quota by Google Cloud projects. To learn more about monitoring and alerting on quota metrics, see Monitor and alert on quota metrics.

Config Connector memory use

You can view your Config Connector memory use in the GKE monitoring dashboard. When Config Connector's memory use is close to its limit, consider Sharding by namespace.

Scaling up Config Controller

If you have reached scalability targets, you should consider further scaling up your Config Controller instances. This section outlines different methods you can use to scale up your Config Controller instances.

Sharding by namespace

If you reach a scalability target with a single Config Connector namespace, you can Configure Config Connector to manage resources in your namespaces.

Each namespace uses its own service accounts and operator workloads, which allows Config Connector to manage your resources at scale. If you use one Config Connector instance to manage multiple Google Cloud projects, you can use one Config Connector namespace to manage each Google Cloud project.

Sharding API quota by Google Cloud projects

If you reach a scalability target due to reaching Google Cloud API Quotas, you can bind different IAM service accounts owned by different Google Cloud projects to different namespaces where Config Connector is installed in namespaced mode. You can then split your resources into different projects.

Sharding by Config Connector instances

If you reach a scalability target with multiple Config Connector namespaces, you can create and use more than one Config Controller instance. With more than one Config Controller instance, you can shard your resource config management, for example, by different development environments, application teams, or GitOps directories within your organization.

Other scalability considerations

Google Cloud API quotas

If you've run into errors indicating that you've exceeded the API quota limit, you might have created too many Config Connector resources of the same Kind under the same project. Those resources can generate too many API requests to the same API endpoint because of the reconciliation strategy in Config Connector.

To resolve this, you can either shard API quota by Google Cloud project or request a higher quota limit.

GKE limitations

Since Config Controller is built on top of GKE, there are limitations from GKE that you should consider. The following sections cover specific considerations related to Config Controller. For more information about general limits and best practices for large GKE clusters, see Plan for large GKE clusters.

Kubernetes service account limit

​​The number of Kubernetes Service Accounts (KSA) created in a single GKE cluster should not exceed 3,000 because you might encounter a gke-metadata-server Pod crash issue.

Whenever you add a Config Connector namespace, it also creates a Kubernetes service account.

GKE control plane performance issues

The GKE cluster's control plane can have performance issues if a Config Controller instance has too many Config Connector namespaces. You should limit the number of Config Connector namespaces to under 500 per cluster.

Whenever you add a Config Connector namespace, it also creates a controller pod.

What's next