Installing Config Sync

Config Sync lets cluster operators manage Kubernetes resources by using files, called configs, stored in Git repositories and this page shows you how to enable and configure Config Sync.

Config Sync is available for Anthos and Google Kubernetes Engine (GKE) users.

Note: If you are using Anthos clusters on VMware, you should use kubectl commands to install Config Sync.

Before you begin

Before you install Config Sync, make sure that you have prepared your environment, clusters, and roles and enabled Anthos Config Management.

Preparing your local environment

Prepare your local environment by completing the following tasks:

1. Create, or have access to, a Git repository that can contain the configs that Config Sync syncs to.
2. Install and initialize the Cloud SDK, which provides the gcloud, and nomos commands. If you use Cloud Shell, the Cloud SDK comes pre-installed.

Cluster requirements

Create, or have access to, a cluster that meets the following requirements:

• Is on an Anthos supported platform and version.
• Uses a Kubernetes version 1.14.x or later.
• Is not an Autopilot cluster.
• Has a node pool that uses a machine type with at least 4 vCPU.
• Has Workload Identity enabled. Workload Identity is the recommended way to access Google Cloud services from applications running within Google Kubernetes Engine (GKE) due to its improved security properties and manageability.

• If you want to use a private GKE cluster, the Cloud NAT is created to permit egress from private GKE nodes. Alternatively, you can enable Private Google Access to connect to the set of external IP addresses used by Google APIs and services.

  Note: Private Google Access doesn't support SSH on port 2022. You can use Cookiefile credentials with Private Google Access.

• If you want to use Google service account when granting Config Sync access to your Git repository, then access scopes for the nodes in the cluster must include read only scope for Cloud Source Repositories.

  You can add this scope by including cloud-source-repos-ro in the -- scopes list specified at cluster creation time, or by using the cloud- platform scope at cluster creation time. For example:

  gcloud container clusters create CLUSTER_NAME --scopes=cloud-platform
  

Preparing your cluster

After you have created a suitable cluster, complete the following steps:

1. If you are using Anthos Config Management for the first time, enable Anthos Config Management.

2. Register your cluster to a fleet. Your project's fleet provides a unified way to view and manage your clusters and their workloads as part of Anthos, including clusters outside Google Cloud. Anthos charges apply only to your registered clusters.

   Note: GKE users who want to install Config Sync by using the Google Cloud Console, can register their clusters when they configure Config Sync.

Preparing permissions

Once your cluster is created, grant yourself the GKE Hub Admin role that you need to Config Sync in the Google Cloud Console. The Hub Admin role provides full access to a fleet.

Note: If you are an owner of your project, you already have this permission and don't need to complete the following steps.

To grant the Hub Admin Role, complete the following steps:

Console

1. In the Cloud Console, go to the IAM page.

   Go to the IAM page

2. Click Add.

3. In the New members field, enter an email address. You can add individuals, service accounts, or Google Groups as members.

4. In the Select a role drop-down list, search for and select GKE Hub Admin.

5. Click Save.

gcloud

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member=MEMBER \
  --role=roles/gkehub.admin

Replace the following:

• MEMBER: An identifier for the member, which usually has the following form: member-type:id For example, user:[e-mail]. For a full list of the values that member can have, see the Policy Binding reference.

• PROJECT_ID: your project ID

Note: For the member type user, the domain name in the identifier must be a Google Workspace domain or a Cloud Identity domain. To learn how to set up a Cloud Identity domain, see the overview of Cloud Identity.

Installing Config Sync

In the following sections, you grant Config Sync access to your repository. After you have granted access, you configure the installation.

Granting Config Sync read-only access to Git

Note: If you are installing Config Sync on the Google Cloud Console, you can either grant access now, or later as part of configuring Config Sync.

Config Sync needs read-only access to your Git repository so that it can read the configs committed to the repository and apply them to your clusters.

If your repository does not require authentication for read-only access, you can continue to configure Config Sync and use none as your authentication type. For example, if you can browse the repository using a web interface without logging in, or if you can use git clone to create a clone of the repository locally without providing credentials or using saved credentials, then you do not need to authenticate. In this case, you do not need to create a Secret.

However, most users need to create credentials because read access to their repository is restricted. If credentials are required, they are stored in the git-creds Secret on each enrolled cluster (unless you are using a Google service account).

Config Sync supports the following mechanisms for authentication:

• SSH key pair
• cookiefile
• Token
• Google Service Account (Google Source Repositories only)

The mechanism that you choose depends on what your repository supports. If all mechanisms are available, we recommend using an SSH key pair. GitHub, Cloud Source Repositories, and Bitbucket all support using an SSH key pair. If your organization hosts your repository and you don't know which authentication methods are supported, contact your administrator.

SSH key pair

An SSH key pair consists of two files, a public key and a private key. The public key typically has a .pub extension.

Warning: Protect the files created by this command. They grant the ability to read from your repository.

To use an SSH key pair, complete the following steps:

1. Create an SSH key pair to allow Config Sync to authenticate to your Git repository. This step is necessary if you need to authenticate to the repository to clone it or read from it. Skip this step if a security administrator provides you with a key pair. You can use a single key pair for all clusters, or a key pair per cluster, depending on your security and compliance requirements.

   The following command creates a 4096-bit RSA key. Lower values are not recommended:

   ssh-keygen -t rsa -b 4096 \
   -C "GIT_REPOSITORY_USERNAME" \
   -N '' \
   -f /path/to/KEYPAIR_FILENAME
   

   Replace the following:

   • GIT_REPOSITORY_USERNAME: the username that you want Config Sync to use to authenticate to the repository
   • /path/to/KEYPAIR_FILENAME: a path to the key pair

   If you are using a third-party Git repository host such as GitHub, or you want to use a service account with Cloud Source Repositories, we recommend that you use a separate account.

   Note: You must create the SSH key without a passphrase. Config Sync does not support SSH key passphrases.

2. Configure your repository to recognize the newly created public key. Refer to the documentation for your Git hosting provider. Instructions for some popular Git hosting providers are included for convenience:

   • Cloud Source Repositories
   • Bitbucket
   • GitHub. We recommend that you create separate deploy keys to provide read-only access to a single GitHub repository.
   • GitLab

3. Add the private key to a new Secret in the cluster:

   kubectl create ns config-management-system && \
   kubectl create secret generic git-creds \
    --namespace=config-management-system \
    --from-file=ssh=/path/to/KEYPAIR_PRIVATE_KEY_FILENAME
   

   Replace /path/to/KEYPAIR_PRIVATE_KEY_FILENAME with the name of the private key (the one without the .pub suffix).

4. Delete the private key from the local disk or otherwise protect it.

cookiefile

Note: Using cookiefile credentials is only supported with Cloud Source Repositories. If you would like to use GitHub with cookiefile authentication, you can use Cloud Source Repositories to mirror your GitHub repository.

The process for acquiring a cookiefile depends on the configuration of your repository. For an example, see Generate static credentials in the Cloud Source Repositories documentation. The credentials are usually stored in the .gitcookies file in your home directory, or they might be provided to you by a security administrator.

To use a cookiefile, complete the following steps:

1. After you create and obtain the cookiefile, add it to a new Secret in the cluster.

   kubectl create ns config-management-system && \
   kubectl create secret generic git-creds \
    --namespace=config-management-system \
    --from-file=cookie_file=/path/to/COOKIEFILE
   

   Replace /path/to/COOKIEFILE with the appropriate path and filename.

2. Protect the contents of the cookiefile if you still need it locally. Otherwise, delete it.

Token

If your organization does not permit the use of SSH keys, you might prefer to use a token. With Config Sync, you can use GitHub's personal access tokens (PATs), GiLab's PATs or deploy keys, or Bitbucket's app password as your token.

To create a Secret using your token, complete the following steps:

1. Create a token using GitHub, GitLab, or Bitbucket:

   • GitHub: Create a PAT. Grant the token the repo scope so that it can read from private repositories. Because you bind a PAT to a GitHub account, we also recommend that you create a machine user and bind your PAT to the machine user.
   • GitLab: Create a PAT or create a deploy token
   • Bitbucket: Create an app password.

2. After you create and obtain the token, add it to a new Secret in the cluster:

   kubectl create ns config-management-system && \
   kubectl create secret generic git-creds \
     --namespace="config-management-system" \
     --from-literal=username=USERNAME \
     --from-literal=token=TOKEN
   

   Replace the following:

   • USERNAME: the username that you want to use.
   • TOKEN: the token that you created in the previous step.

3. Protect the token if you still need it locally. Otherwise, delete it.

Google service account

Note: This method only works on GKE clusters.

If your repository is in Cloud Source Repositories, you can give Config Sync access to a repository in the same project as your managed cluster by using a Google service account.

Caution: If you have Workload Identity enabled on your cluster and want to use a Google service account as your method of authentication, you cannot use the Google Cloud Console to configure Config Sync. Instead, you need to use the gcloud command-line tool and select gcpServiceAccount as your authentication method.

To use a repository in Cloud Source Repositories as your Config Sync repository, complete the following steps:

1. List all repositories:

   gcloud source repos list
   

2. From the output, copy the URL from the repository that you want to use. For example:

   REPO_NAME  PROJECT_ID  URL
   my-repo    my-project  https://source.developers.google.com/p/my-project/r/my-repo-csr
   

3. When you configure Config Sync in the following section, add the URL to your configuration. If you are configuring Config Sync using the Google Cloud Console, add the value that you copied in the URL field. If you are configuring Config Sync using the gcloud command-line tool, add the value that you copied to the repo field of your ConfigManagement object.

4. When you configure Config Sync, select the appropriate authentication type.

   If you are configuring Config Sync using the Google Cloud Console, select Google Cloud Repository as your Authentication type when you configure Config Sync.

   If you are configuring Config Sync using the gcloud command-line tool, select gcpServiceAccount or gcenode as your secretType:

   • gcpServiceAccount: If Workload Identity is enabled on your cluster add gcpServiceAccount as your authentication type.

     If necessary, create a service account that you can use when you configure Config Sync. Ensure that the service account has read access to Cloud Source Repositories by granting it the source.reader role.

     When you select gcpServiceAccount as your method of authentication, you also need to add a gcpServiceAccountEmail when you configure Config Sync. In addition to adding the gcpServiceAccountEmail, you need to create a role binding after configuring Config Sync. For information on these steps, see the Using Cloud Source Repositories with Workload Identity section.

   • gcenode: If Workload Identity is not enabled on your cluster, add gcenode to use the Compute Engine default service account. By default, the Compute Engine default service account, [e-mail], has source.reader access to the repository for the same project. However, if your Cloud Source Repositories is located in a project different from your cluster's project, you have to grant default Compute Engine service account from the cluster's project, the source.reader in the Cloud Source Repositories's project.

     If needed, you can add the source.reader` role with the following command:

     gcloud projects add-iam-policy-binding PROJECT_ID \
       --member serviceAccount:[e-mail] \
       --role roles/source.reader
     

     Replace the following:

     • PROJECT_ID: your organization's project ID
     • PROJECT_NUMBER: your organization's project number

     You cannot modify access scopes after you create a node pool. However, you can create a new node pool with the proper access scope while using the same cluster. The default gke-default scope does not include cloud-source-repos-ro.

Configuring Config Sync

The Google Cloud Console guides you through the installation process and automates many of the steps. Console commands are slightly different for Anthos and GKE customers. Click on the relevant tab to view the appropriate instructions.You can also use the gcloud command-line tool to complete the installation.

Console - Anthos

Before continuing, make sure you've registered your clusters to a fleet.

If you use Anthos, complete the following steps to configure Config Sync:

1. In the Cloud Console, go to the Anthos Config Management page.

   Go to Anthos Config Management

2. Select your registered clusters and click Configure.

3. In the Git Repository Authentication for ACM section, select one of the following options:

   • None: Use no authentication
   • SSH: Use a SSH key pair
   • Cookiefile: Use a cookiefile
   • Token: Use a token
   • Google Cloud Repository: Use a Google service account to access a Cloud Source Repositories. Only select this option if Workload Identity is not enabled in your cluster.

4. If you have not done so already, follow the instructions in the Google Cloud Console to grant Config Sync read-only access to Git.

5. Click Continue.

6. In the ACM settings for your clusters section, complete the following:

   1. In the Version field, select version 1.7.0 or later for Anthos Config Management, which enables syncing from multiple repositories by default.
   2. Select the Enable Config Sync checkbox.
      1. In the URL field, add the URL of the Git repository to use as the source of truth. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/anthos-config-management-samples uses the HTTPS protocol. If you don't enter a protocol, the URL is treated as an HTTPS URL. This field is required.
      2. Optional: In the Branch field, add the branch of the repository to sync from. The default is the main (master) branch.
      3. Optional: In the Tag/Commit field, add the Git revision (tag or hash) to check out. The default is HEAD.
      4. Optional: In the Policy directory field, add the path within the repository to the top of the policy hierarchy to sync. The default is the root directory of the repository.
      5. Optional: In the Sync wait field, add the period in seconds between consecutive syncs. The default is 15 seconds.
      6. Optional: In the Git proxy field, enter the URL for the HTTPS proxy to be used when communicating with the Git repository. This is an optional field, and if it's left blank, no proxy is used.
      7. Optional: In the Source format field, choose either hierarchy (default) or unstructured (recommended). We recommend that you select unstructured as this format lets you organize your configs in the way that is most convenient to you.

7. Click Done. You are taken back to the Anthos Config Management page. After a few minutes, you should see Synced in the status column next to the clusters that you configured. If you see Error, click on the word Error for more information.

   Note: When you use the Google Cloud Console to install Config Sync and use a version of 1.7.0 or later, multi-repo mode is enabled by default.

Console - GKE

If you use GKE, complete the following steps to configure Config Sync:

Registering your clusters

To use Config Management with GKE, you must first register the clusters. Registering your clusters lets them share a common set of configurations and policies.

To register your clusters, complete the following tasks:

1. In the Cloud Console, go to the Config Management page.

   Go to Config Management

   This page shows you which clusters are currently registered and configured and lets you begin registering new clusters.

2. To begin the registration process, click Set up Config Management.

3. To enable the Config Management API, click Next.

4. In the Select registered clusters for Config Management page, locate the Unregistered clusters from this project table, and find the cluster that you want to register.

5. Click Register next to the cluster that you want to register.

   Once the cluster is successfully registered, it appears in the Select registered clusters for Config Management table.

Installing Config Sync

Once you have registered your clusters, you can continue on to install and configure Config Sync.

To install Config Sync, complete the following steps:

1. To install Config Sync, select the clusters that you want to configure, and select Next. Selecting multiple clusters lets you apply the same configurations to whichever clusters you configure.
2. In the Config Sync page that appears, select the Anthos Config Management Version that you want to use. The default is the current version.
3. In the Configurations section, leave the Enable Config Sync checkbox selected.
4. In the URL field, add the URL of the Git repository to use as the source of truth. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/anthos-config-management-samples uses the HTTPS protocol. If you don't enter a protocol, the URL is treated as an HTTPS URL.

5. In the Authentication type drop-down list, select the one of the following options:

   • None: Use no authentication
   • SSH: Use a SSH key pair
   • Cookiefile: Use a cookiefile
   • Token: Use a token
   • Google Cloud Repository: Use a Google service account to access a Cloud Source Repositories. Only select this option if Workload Identity is not enabled in your cluster.

6. Follow the instructions in the Google Cloud Console to grant Config Sync read-only access to Git and click Continue.

7. Optional: In the Branch field, add the branch of the repository to sync from. The default is the master branch.

8. Optional: In the Tag/Commit field, add the Git revision (tag or hash) to check out. The default is HEAD.

9. Optional: In the Policy directory field, add the path within the repository to the top of the policy hierarchy to sync. The default is the root directory of the repository.

10. Optional: In the Sync wait field, add the period in seconds between consecutive syncs. The default is 15 seconds.

11. Optional: In the Git proxy field, enter the URL for the HTTPS proxy to be used when communicating with the Git repository. This is an optional field, and if it's left blank, no proxy is used.

12. Optional: In the Source format field, choose either hierarchy (default) or unstructured (recommended). We recommend that you select unstructured as this format lets you organize your configs in the way that is most convenient to you.

13. If you don't want to install Policy Controller, click Next and clear the **Enable Policy Controller checkbox.

14. Click Complete to complete the Config Sync installation.

gcloud

Note: The file format for the gcloud tool apply command changed for version 1.8.0, the gcloud command-line tool no longer accepts the same config file used by kubectl. Please make sure you update your gcloud tool to the latest version.

Before continuing, make sure you've registered your clusters to a fleet.

If you use Anthos or GKE, complete the following steps to configure Config Sync with the gcloud command-line tool.

1. Prepare the configuration by either creating a new apply-spec.yaml manifest or by using an existing manifest. Using an existing manifest lets you configure your cluster with the same settings used by another cluster.

Create new manifest

To configure Config Sync with new settings for your cluster, create a file named apply-spec.yaml and copy the following YAML file into it:

# apply-spec.yaml

applySpecVersion: 1
spec:
  configSync:
    # Set to true to install and enable Config Sync
    enabled: true
    sourceFormat: FORMAT
    syncRepo: REPO
    syncBranch: BRANCH
    secretType: TYPE
    gcpServiceAccountEmail: EMAIL
    policyDir: DIRECTORY

Replace the following:

• FORMAT: add unstructured to use an unstructured repository or add hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default value is hierarchy. We recommend that you add unstructured as this format lets you organize your configs in the way that is most convenient to you.
• REPO: add the URL of the Git repository to use as the source of truth. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/anthos-config-management-samples uses the HTTPS protocol. If you don't enter a protocol, the URL is treated as an HTTPS URL. This field is required.
• BRANCH: the branch of the repository to sync from. The default is the main (master) branch.

• TYPE: one of the following secretTypes:

  • none: Use no authentication
  • ssh: Use a SSH key pair
  • cookiefile: Use a cookiefile
  • token: Use a token
  • gcpserviceaccount: Use a Google service account to access a Cloud Source Repositories.
  • gcenode: Use a Google service account to access a Cloud Source Repositories. Only select this option if Workload Identity is not enabled in your cluster.

  For more information on these authentication types, see Granting Config Sync read-only access to Git.

• EMAIL: If you added gcpserviceaccount as your secretType, add your Google service account email address. For example, acm@PROJECT_ID.iam.gserviceaccount.com.

• DIRECTORY: the path in the Git repository to the root directory that contains the configuration that you want to sync to. The default is the root directory of the repository.

For a complete list of fields that you can add to the spec field, see gcloud fields.

Note: When you use the gcloud command-line tool to install Config Sync and use a version of 1.7.0 or later, multi-repo mode is enabled by default.

Use existing manifest

To configure your cluster with the same settings used by another cluster, fetch the settings from a registered cluster:

 gcloud alpha container hub config-management fetch-for-apply \
     --membership=MEMBERSHIP_NAME \
     --project=PROJECT_ID \
     > CONFIG_YAML_PATH

Replace the following:

• MEMBERSHIP_NAME: the membership name of the registered cluster that has the Config Sync settings you want to use
• PROJECT_ID: your project ID
• CONFIG_YAML_PATH: the path to the apply-spec.yaml file

2. Apply the apply-spec.yaml file:

  gcloud beta container hub config-management apply \
      --membership=MEMBERSHIP_NAME \
      --config=CONFIG_YAML_PATH \
      --project=PROJECT_ID

Replace the following:

• MEMBERSHIP_NAME: the membership name that you chose when you registered your cluster, you can find the name with gcloud container hub memberships list.
• CONFIG_YAML_PATH: the path to your apply-spec.yaml file
• PROJECT_ID: your project ID

Using Cloud Source Repositories with Workload Identity

If Workload Identity is enabled on your cluster and you used a Google service account as your authentication type, you need to complete additional steps.

After completing the preceding steps, create an IAM policy binding between the Kubernetes service account and the Google service account. The Kubernetes service account is not created until you configure Config Sync for the first time.

This binding lets the Config Sync Kubernetes service account act as the Google service account:

gcloud iam service-accounts add-iam-policy-binding \
   --role roles/iam.workloadIdentityUser \
   --member "serviceAccount:PROJECT_ID.svc.id.goog[config-management-system/root-reconciler]" \
   GSA_NAME@PROJECT_ID.iam.gserviceaccount.com

Replace the following:

• PROJECT_ID: your organization's project ID
• GSA_NAME: the custom Google service account that you want to use to connect to Cloud Source Repositories. Make sure that the Google service account that you select has the source.reader role.

Note: Creating this policy binding requires the iam.serviceAccounts.setIamPolicy permission.

Verifying the installation

After you have installed and configured Config Sync, you can verify that the installation completed successfully.

Console - Anthos

Complete the following steps:

1. In the Cloud Console, go to the Anthos Config Management page.

   Go to Anthos Config Management

2. View the Status column. A successful installation has a status of Synced.

For a detailed view of your cluster status:

• Select the cluster that you want to explore. The Cluster details page appears. On this page, you can see details for your cluster and details for your Config Sync installation.

Console - GKE

Complete the following steps:

1. In the Cloud Console, go to the Config Management page.

   Go to Config Management

2. In the cluster table, view the Config sync status column. A successful installation has a status of Synced.

gcloud

Run the following command:

gcloud beta container hub config-management status \
    --project=PROJECT_ID

Replace PROJECT_ID with your project's ID.

A successful installation has a status of SYNCED. If you see an error after running the preceding command, make sure that you created the git-creds Secret. If you have created the Secret, try rerunning the following command:

gcloud beta container hub config-management apply

You can also use the nomos status command to check if the Config Sync is installed successfully. A valid installation with no problems has a status of PENDING or SYNCED. An invalid or incomplete installation has a status of NOT INSTALLED OR NOT CONFIGURED. The output also includes any reported errors.

It's possible to have an invalid configuration that isn't detected right away, such as a missing or invalid git-creds Secret. For troubleshooting steps, see Valid but incorrect ConfigManagement object in the Troubleshooting page.

Resource requests

Summary of total resource requests

The following table lists the combined amount of resource requests for Config Sync, depending on what features you are using.

Feature	CPU	Memory
Default (multi-repo) mode	710 m + 260 m * (number of RootSync and RepoSync objects)	620 Mi + 210 Mi * (number of RootSync and RepoSync objects)
Hierarchy Controller	220 m	220 Mi
Single repository mode	460 m	310 Mi

Detailed resource requests

The following table lists Kubernetes resource requirements for Config Sync components. For more information, see Managing Resources for Containers in the Kubernetes documentation.

Deployment name	CPU request (m) per replica	Memory request (Mi) per replica	Single or multiple repositories
git-importer	450	300	Single
monitor	Default^1	Default	Single
resource-group-controller-manager	100+default	50+default	Multi
admission-webhook^2	100	20	Multi
otel-collector	200	400	Multi
reconciler-manager	200	120	Multi
reconciler (one per RootSync and RepoSync)	250+default	200+default	Multi
hnc-controller-manager	100+default	150+default	Hierarchy Controller
gke-hc-controller-manager	100+default	50+default	Hierarchy Controller

¹The default resource request uses a CPU request of 10 and a memory request of 10 Mi.

²The admission webhook has two replicas, so when calculating the total resource requests, you need to double the value.

What's next

• Learn more about the gcloud commands for configuring Config Sync with Anthos Config Management.
• Learn about creating configs.
• Use the nomos command.
• Learn how to upgrade your Config Sync installation.
• Learn more about troubleshooting Config Sync.
• Learn how to uninstall Config Sync.