This page explains configuration choices and requirements when planning your clusters for use with Config Sync.
For more information about general best practices when planning your GKE clusters, review the GKE documentation for Cluster configuration choices.
Resource requirements with Autopilot mode
GKE Autopilot mode automatically modifies resource requests to maintain workload stability. To understand how to plan for those requests, review the GKE documentation for Autopilot resource requests.
Due to the way Autopilot modifies resource requests, Config Sync makes the following adjustments:
- Adjusts user-specified resource override limits to match requests.
- Applies overrides only when there are one or more resource requests higher than the corresponding adjusted output declared in the annotation, or there are resource requests lower than the corresponding input declared in the annotation.
Supported GKE Enterprise platforms and versions
To use Config Sync, your cluster must be on a GKE Enterprise supported platform and version.
Workload Identity Federation for GKE
Workload Identity Federation for GKE is the recommended way to securely connect to Google Cloud services. Workload Identity Federation for GKE is enabled by default on Autopilot clusters.
If you want to use fleet packages (Preview) with Config Sync, Workload Identity Federation for GKE is required.
GKE release channels
If you want Config Sync to automatically manage upgrades, it's recommended to enroll your cluster in a GKE release channel. Config Sync auto-upgrades uses release channels to determine when to upgrade to a new version.
If you enable auto-upgrades without enrolling in a release channel, Config Sync manages upgrades for that cluster as if the cluster was using the Stable release channel.
Networking
The following section lists some of the changes you might need to make to your GKE cluster, depending on your networking settings.
For more information about GKE networking choices, see Network overview.
Private clusters
If you use private clusters, you should configure your clusters in one of the following ways to ensure Config Sync has access and can authenticate to your source of truth:
Configure Cloud NAT to permit egress from private GKE nodes. For details, see Example GKE setup.
Enable Private Google Access to connect to the set of external IP addresses used by Google APIs and services.
Public clusters
If you use public clusters, but have strict VPC Firewall requirements that block any unnecessary traffic, you must Create firewall rules to permit the following traffic:
- TCP: Allow ingress and egress on port 53 and 443
- UDP: Allow egress on port 53
If you don't include these rules, Config Sync doesn't sync correctly, with
nomos status reporting the following error:
Error: KNV2004: unable to sync repo Error in the git-sync container
Cloud Source Repositories with Compute Engine default service account authentication
If you are using Config Sync to connect to Cloud Source Repositories and Workload Identity Federation for GKE is not enabled, you can use the Compute Engine default service account to authenticate. You must use access scopes with read-only scopes for the nodes in the cluster.
You can add the read-only scope for Cloud Source Repositories by including
cloud-source-repos-ro in the --scopes list specified at cluster creation
time, or by using the cloud-platform scope at cluster creation time. For
example:
gcloud container clusters create CLUSTER_NAME --scopes=cloud-platform
Replace CLUSTER_NAME with the name of your cluster.
You cannot modify access scopes after you create a node pool.
However, you can create a new node pool with the proper access scope while
using the same cluster. The default gke-default scope does not include
cloud-source-repos-ro.
Arm nodes
Config Sync can run only on x86-based nodes, not Arm nodes. However, if you need to run Config Sync on a cluster with multiple architectures, take the following action, depending on your cluster type:
- GKE on AWS or GKE on Azure: add a taint to your Arm nodes, to avoid scheduling Pods onto your Arm nodes without a corresponding toleration.
- GKE: GKE adds a default taint to ensure workloads without the corresponding toleration aren't scheduled there. No additional action needed.