Install Config Sync manually using kubectl (not recommended)

This page shows you how to install Config Sync using kubectl commands. The ConfigManagement Operator is a controller that manages Config Sync in a Kubernetes cluster. Follow these steps to install and configure the Operator in each cluster you want to manage using Config Sync.

Caution: The installation methods used on this page are not recommended because they are not compatible with the Google Cloud console or the Google Cloud CLI.

Before you begin

This section describes prerequisites you must meet before installing Config Sync using kubectl.

Prepare your local environment

Before you install the Operator, make sure you have prepared your local environment by completing the following tasks:

• Create, or have access to a source of truth.

• Install and initialize the Google Cloud CLI, which provides the gcloud, kubectl, and nomos commands used in these instructions. If you use Cloud Shell, Google Cloud CLI comes pre-installed.

• kubectl is not installed by default by Google Cloud CLI. To install kubectl, use the following command:

  gcloud components install kubectl
  

• Authenticate to Google Cloud using the gcloud auth login command so that you can download components of Config Sync.

Prepare your clusters

Create, or have access to, a Google Kubernetes Engine (GKE) Enterprise edition cluster that meets the requirements for Config Sync.

Prepare permissions

The Google Cloud user installing Config Sync needs IAM permissions to create new roles in your cluster. If needed, grant these roles with the following commands:

gcloud container clusters get-credentials CLUSTER_NAME

kubectl create clusterrolebinding cluster-admin-binding \
    --clusterrole cluster-admin --user USER_ACCOUNT

Replace the following:

• CLUSTER_NAME: your cluster name
• USER_ACCOUNT: your Google Cloud account's email address

Depending on how you configured the Google Cloud CLI on your local system, you might need to add the --project and --zone fields.

If you need to grant the Operator access to OCI using gcpserviceaccount as your authentication type, to create a policy binding, you must also have the iam.serviceAccounts.setIamPolicy permission. You can get this permission by granting the Service Account Admin (roles/iam.serviceAccountAdmin) IAM role. You might also be able to get this permission with custom roles or other predefined roles.

For more information about granting roles, see Manage access.

Enroll a cluster

To enroll a cluster in Config Sync, complete the following steps:

1. Deploy the Operator
2. Grant the Operator read-only access to one of the following:
   • Git
   • OCI
   • Helm
3. Configure the Operator

Deploy the Operator

After ensuring that you meet all the prerequisites, you can deploy the Operator by downloading and applying a YAML manifest.

1. Download the latest version of the Operator manifests using the following command. To download a specific version instead, see Downloads.

   gcloud storage cp gs://config-management-release/released/latest/config-management-operator.yaml config-management-operator.yaml
   

2. Apply the manifests:

   kubectl apply -f config-management-operator.yaml

If this fails because of a problem with the ConfigManagement object that is not due to a YAML or JSON syntax error, the object might be instantiated in the cluster, but might not work correctly. In this situation, you can use the nomos status command to check for errors in the object.

A valid installation with no problems has a status of PENDING or SYNCED.

An invalid installation has a status of NOT CONFIGURED and lists one of the following errors:

• missing git-creds Secret
• missing required syncRepo field
• git-creds Secret is missing the key specified by secretType

To fix the problem, correct the configuration error. Depending on the type of error, you might need to re-apply the ConfigManagement manifest to the cluster.

If the problem is that you forgot to create the git-creds Secret, Config Sync detects the Secret as soon as you create it, and you don't need to re-apply the configuration.

Grant the Operator read-only access

If you store your configs in Git, you must grant the Operator read-only access to Git. If you store your configs as OCI images, you must grant the Operator read-only access to OCI. If you store your configs in Helm, you must grant the Operator read-only access to Helm.

Grant Operator read-only access to Git

Config Sync needs read-only access to your Git repository so that it can read the configs committed to the repository and apply them to your clusters.

If your repository does not require authentication for read-only access, you can continue to configure Config Sync and use none as your authentication type. For example, if you can browse the repository using a web interface without logging in, or if you can use git clone to create a clone of the repository locally without providing credentials or using saved credentials, then you don't need to authenticate. In this case, you don't need to create a Secret.

However, most users need to create credentials because read access to their repository is restricted. If credentials are required, they are stored in the git-creds Secret on each enrolled cluster (unless you are using a Google service account). The Secret must be named git-creds because this is a fixed value.

Config Sync supports the following mechanisms for authentication:

• SSH key pair (ssh)
• Cookiefile (cookiefile)
• Token (token)
• Google service account (gcpserviceaccount)
• Compute Engine default service account (gcenode)
• GitHub App (githubapp)

The mechanism that you choose depends on what your repository supports. Generally, we recommend using an SSH key pair. GitHub and Bitbucket both support using an SSH key pair. However, if you are using a repository in Cloud Source Repositories, we recommend that you use a Google service account instead as the process is simpler. If your organization hosts your repository and you don't know which authentication methods are supported, contact your administrator.

To use a repository in Cloud Source Repositories as your Config Sync repository, complete the following steps to retrieve your Cloud Source Repositories URL:

1. List all repositories:

   gcloud source repos list
   

2. From the output, copy the URL from the repository that you want to use. For example:

   REPO_NAME  PROJECT_ID  URL
   my-repo    my-project  https://source.developers.google.com/p/my-project/r/my-repo-csr
   

   You need to use this URL when you configure Config Sync in the following section. If you configure Config Sync using the Google Cloud console, you add the URL in the URL field. If you configure Config Sync using the Google Cloud CLI, you add the URL to the syncRepo field of your configuration file.

SSH key pair

An SSH key pair consists of two files, a public key and a private key. The public key typically has a .pub extension.

Warning: Protect the files created by this command. They grant the ability to read from your repository.

To use an SSH key pair, complete the following steps:

1. Create an SSH key pair to allow Config Sync to authenticate to your Git repository. This step is necessary if you need to authenticate to the repository to clone it or read from it. Skip this step if a security administrator provides you with a key pair. You can use a single key pair for all clusters, or a key pair per cluster, depending on your security and compliance requirements.

   The following command creates a 4096-bit RSA key. Lower values are not recommended:

   ssh-keygen -t rsa -b 4096 \
   -C "GIT_REPOSITORY_USERNAME" \
   -N '' \
   -f /path/to/KEYPAIR_FILENAME
   

   Replace the following:

   • GIT_REPOSITORY_USERNAME: the username that you want Config Sync to use to authenticate to the repository
   • /path/to/KEYPAIR_FILENAME: a path to the key pair

   If you are using a third-party Git repository host such as GitHub, or you want to use a service account with Cloud Source Repositories, we recommend that you use a separate account.

   Note: You must create the SSH key without a passphrase. Config Sync does not support SSH key passphrases.

2. Configure your repository to recognize the newly created public key. Refer to the documentation for your Git hosting provider. Instructions for some popular Git hosting providers are included for convenience:

   • Cloud Source Repositories
   • Bitbucket
   • GitHub We recommend that you create separate deploy keys to provide read-only access to a single GitHub repository.
   • GitLab

3. Add the private key to a new Secret in the cluster:

   kubectl create ns config-management-system && \
   kubectl create secret generic git-creds \
    --namespace=config-management-system \
    --from-file=ssh=/path/to/KEYPAIR_PRIVATE_KEY_FILENAME
   

   Replace /path/to/KEYPAIR_PRIVATE_KEY_FILENAME with the name of the private key (the one without the .pub suffix).

4. (Recommended) To configure known hosts checking using SSH authentication, you can add the known hosts key to the data.known_hosts field in the git_creds secret. To disable known_hosts checking, you can remove the known_hosts field from the secret. To add the known hosts key, run:

   kubectl edit secret git-creds \
    --namespace=config-management-system
   

   Then, under data, add the known hosts entry:

   known_hosts: KNOWN_HOSTS_KEY
   

5. Delete the private key from the local disk or otherwise protect it.

6. When you configure Config Sync and add the URL for your Git repository, use the SSH protocol. If you are using a repository in Cloud Source Repositories, you must use the following format when you enter your URL:

   ssh://[e-mail]:2022/p/PROJECT_ID/r/REPO_NAME
   

   Replace the following:

   • EMAIL: your Google Cloud username
   • PROJECT_ID: the ID of the Google Cloud project where the repository is located
   • REPO_NAME: the name of the repository

Cookiefile

Note: Using cookiefile credentials is only supported with Cloud Source Repositories. If you would like to use GitHub with cookiefile authentication, you can use Cloud Source Repositories to mirror your GitHub repository.

The process for acquiring a cookiefile depends on the configuration of your repository. For an example, see Generate static credentials in the Cloud Source Repositories documentation. The credentials are usually stored in the .gitcookies file in your home directory, or they might be provided to you by a security administrator.

To use a cookiefile, complete the following steps:

1. After you create and obtain the cookiefile, add it to a new Secret in the cluster.

   If you don't use an HTTPS proxy, create the Secret with the following command:

   kubectl create ns config-management-system && \
   kubectl create secret generic git-creds \
    --namespace=config-management-system \
    --from-file=cookie_file=/path/to/COOKIEFILE
   

   If you need to use an HTTPS proxy, add it to the Secret together with cookiefile by running the following command:

   kubectl create ns config-management-system && \
   kubectl create secret generic git-creds \
    --namespace=config-management-system \
    --from-file=cookie_file=/path/to/COOKIEFILE \
    --from-literal=https_proxy=HTTPS_PROXY_URL
   

   Replace the following:

   • /path/to/COOKIEFILE: the appropriate path and filename
   • HTTPS_PROXY_URL: the URL for the HTTPS proxy that you use when communicating with the Git repository
   Note: HTTP proxy is not supported for security reasons.

2. Protect the contents of the cookiefile if you still need it locally. Otherwise, delete it.

Token

If your organization does not permit the use of SSH keys, you might prefer to use a token. With Config Sync, you can use GitHub's personal access tokens (PATs), GiLab's PATs or deploy keys, or Bitbucket's app password as your token.

To create a Secret using your token, complete the following steps:

1. Create a token using GitHub, GitLab, or Bitbucket:

   • GitHub: Create a PAT. Grant the token the repo scope so that it can read from private repositories. Because you bind a PAT to a GitHub account, we also recommend that you create a machine user and bind your PAT to the machine user.
   • GitLab: Create a PAT or create a deploy token
   • Bitbucket: Create an app password.

2. After you create and obtain the token, add it to a new Secret in the cluster.

   If you don't use an HTTPS proxy, create the Secret with the following command:

   kubectl create ns config-management-system && \
   kubectl create secret generic git-creds \
     --namespace="config-management-system" \
     --from-literal=username=USERNAME \
     --from-literal=token=TOKEN
   

   Replace the following:

   • USERNAME: the username that you want to use.
   • TOKEN: the token that you created in the previous step.

   If you need to use an HTTPS proxy, add it to the Secret together with username and token by running the following command:

   kubectl create ns config-management-system && \
   kubectl create secret generic git-creds \
    --namespace=config-management-system \
    --from-literal=username=USERNAME \
     --from-literal=token=TOKEN \
    --from-literal=https_proxy=HTTPS_PROXY_URL
   

   Replace the following:

   • USERNAME: the username that you want to use.
   • TOKEN: the token that you created in the previous step.
   • HTTPS_PROXY_URL: the URL for the HTTPS proxy that you use when communicating with the Git repository.
   Note: HTTP proxy is not supported for security reasons.

3. Protect the token if you still need it locally. Otherwise, delete it.

Google service account

If your repository is in Cloud Source Repositories, and your cluster uses GKE Workload Identity Federation for GKE or fleet Workload Identity Federation for GKE, you can give Config Sync access to a repository in the same project as your managed cluster by using a Google service account.

1. If you don't already have a service account, create a service account.

2. Grant the Cloud Source Repositories Reader (roles/source.reader) IAM role to the Google service account. For more information about Cloud Source Repositories roles and permissions, see Grant permissions to view repositories.

   • Grant project-wide permission if the same permissions apply to all repositories in the project.

     gcloud projects add-iam-policy-binding PROJECT_ID \
       --role=roles/source.reader \
       --member="serviceAccount:GSA_NAME@PROJECT_ID.iam.gserviceaccount.com"
     

   • Grant repository-specific permission when you want service accounts to have different levels of access for each repository in your project.

     gcloud source repos set-iam-policy REPOSITORY POLICY_FILE --project=PROJECT_ID
     

3. If you configure Config Sync using the Google Cloud console, select Workload Identity Federation for GKE as the Authentication Type and then add your service account email.

   If you configure Config Sync using the Google Cloud CLI, add gcpserviceaccount as the secretType and then add your service account email to gcpServiceAccountEmail.

4. After configuring Config Sync, create an IAM policy binding between the Kubernetes service account and the Google service account. The Kubernetes service account is not created until you configure Config Sync for the first time.

   If you are using clusters that are registered to a fleet, you only have to create the policy binding once per fleet. All clusters registered in a fleet share the same Workload Identity Federation for GKEpool. With fleet's concept of sameness, if you add the IAM policy to your Kubernetes service account in one cluster, then the Kubernetes service account from the same namespace on other clusters in the same fleet also get the same IAM policy.

   This binding lets the Config Sync Kubernetes service account act as the Google service account:

   gcloud iam service-accounts add-iam-policy-binding \
       GSA_NAME@PROJECT_ID.iam.gserviceaccount.com \
       --role=roles/iam.workloadIdentityUser \
       --member="serviceAccount:FLEET_HOST_PROJECT_ID.svc.id.goog[config-management-system/KSA_NAME]" \
       --project=PROJECT_ID
   

Replace the following:

• PROJECT_ID: the organization's project ID.
• FLEET_HOST_PROJECT_ID: if you're using GKE Workload Identity Federation for GKE, this is the same as PROJECT_ID. If you're using fleet Workload Identity Federation for GKE, this is the project ID of the fleet that your cluster is registered to.
• GSA_NAME: the custom Google service account that you want to use to connect to Artifact Registry. The service account must have the Artifact Registry Reader (roles/artifactregistry.reader) IAM role.
• KSA_NAME: the Kubernetes service account for the reconciler.
  • For root repositories, if the RootSync name is root-sync, use root-reconciler. Otherwise, use root-reconciler-ROOT_SYNC_NAME. If you install Config Sync using the Google Cloud console or the Google Cloud CLI, Config Sync automatically creates a RootSync object named root-sync.
• REPOSITORY: the name of the repository.
• POLICY_FILE: the JSON or YAML file with the Identity and Access Management policy.

Note: Creating this policy binding requires the iam.serviceAccounts.setIamPolicy permission.

Compute Engine default service account

If your repository is in Cloud Source Repositories, and your cluster is GKE with Workload Identity Federation for GKE disabled, you can use gcenode as your authentication type.

If you configure Config Sync using the Google Cloud console, select Google Cloud Repository as the Authentication Type.

If you configure Config Sync using the Google Cloud CLI, add gcenode as the secretType.

Selecting either Google Cloud Repository or gcenode lets you use the Compute Engine default service account. You must grant the Cloud Source Repositories Reader (roles/source.reader) IAM role to the Compute Engine default service account. For more information about Cloud Source Repositories roles and permissions, see Grant permissions to view repositories.

gcloud projects add-iam-policy-binding PROJECT_ID \
  --role=roles/source.reader \
  --member="serviceAccount:[e-mail]"

Replace PROJECT_ID with your organization's project ID, and replace PROJECT_NUMBER with your organization's project number.

GitHub App

Note: This authentication type is supported only in version 1.19.1 and later.

If your repository is in GitHub, you can use githubapp as your authentication type.

To use a GitHub App, complete the following steps:

1. Follow the instructions on GitHub to provision a GitHub App and give it permission to read from your repository.

2. Add the GitHub App configuration to a new Secret in the cluster:

   Using Client ID

   kubectl create ns config-management-system && \
   kubectl create secret generic git-creds \
     --namespace=config-management-system \
     --from-literal=github-app-client-id=CLIENT_ID \
     --from-literal=github-app-installation-id=INSTALLATION_ID \
     --from-file=github-app-private-key=/path/to/GITHUB_PRIVATE_KEY \
     --from-literal=github-app-base-url=BASE_URL
   

   • Replace CLIENT_ID with the client ID for the GitHub App.
   • Replace INSTALLATION_ID with the installation ID for the GitHub App.
   • Replace /path/to/GITHUB_PRIVATE_KEY with the name of the file containing the private key.
   • Replace BASE_URL with the base URL for the GitHub API endpoint. This is only needed when the repository is not hosted at www.github.com. The argument can otherwise be omitted and will default to https://api.github.com/.

   Using Application ID

   kubectl create ns config-management-system && \
   kubectl create secret generic git-creds \
     --namespace=config-management-system \
     --from-literal=github-app-application-id=APPLICATION_ID \
     --from-literal=github-app-installation-id=INSTALLATION_ID \
     --from-file=github-app-private-key=/path/to/GITHUB_PRIVATE_KEY \
     --from-literal=github-app-base-url=BASE_URL
   

   • Replace APPLICATION_ID with the application ID for the GitHub App.
   • Replace INSTALLATION_ID with the installation ID for the GitHub App.
   • Replace /path/to/GITHUB_PRIVATE_KEY with the name of the file containing the private key.
   • Replace BASE_URL with the base URL for the GitHub API endpoint. This is only needed when the repository is not hosted at www.github.com. The argument can otherwise be omitted and will default to https://api.github.com/.

3. Delete the private key from the local disk or otherwise protect it.

4. When you configure Config Sync and add the URL for your Git repository, use the githubapp auth type.

Grant Operator read-only access to OCI

Config Sync needs read-only access to your OCI image stored in Artifact Registry so that it can read the configs included in the image and apply them to your clusters.

If your image does not require authentication for read-only access, you can continue to configure Config Sync and use none as your authentication type. For example, if your image is public and can be accessed by anyone on the internet, then you don't need to authenticate.

However, most users need to create credentials to access restricted images. Config Sync supports the following mechanisms for authentication:

• Kubernetes service account (k8sserviceaccount)
• Google service account (gcpserviceaccount)

• Compute Engine default service account (gcenode)

Kubernetes service account

If you store your OCI image in Artifact Registry and your cluster uses GKE Workload Identity Federation for GKE or fleet Workload Identity Federation for GKE, you can use k8sserviceaccount as your authentication type in version 1.17.2 and later. This option is recommended over gcpserviceaccount due to its simplified configuration process.

1. Grant the Artifact Registry Reader (roles/artifactregistry.reader) IAM role to the Kubernetes service account with the Workload Identity Federation for GKE pool. For more information about Artifact Registry roles and permissions, see Configure roles and permissions for Artifact Registry.

   • Grant project-wide permission if the same permissions apply to all repositories in the project.

     gcloud projects add-iam-policy-binding PROJECT_ID \
           --role=roles/artifactregistry.reader \
           --member="serviceAccount:FLEET_HOST_PROJECT_ID.svc.id.goog[config-management-system/KSA_NAME]"
     

   • Grant repository-specific permission when you want service accounts to have different levels of access for each repository in your project.

     gcloud artifacts repositories add-iam-policy-binding REPOSITORY \
        --location=LOCATION \
        --role=roles/artifactregistry.reader \
        --member="serviceAccount:FLEET_HOST_PROJECT_ID.svc.id.goog[config-management-system/KSA_NAME]" \
        --project=PROJECT_ID
     

   Replace the following:

   • PROJECT_ID: the organization's project ID.
   • FLEET_HOST_PROJECT_ID: if you're using GKE Workload Identity Federation for GKE, this is the same as PROJECT_ID. If you're using fleet Workload Identity Federation for GKE, this is the project ID of the fleet that your cluster is registered to.
   • KSA_NAME: the Kubernetes service account for the reconciler.
     • For root repositories, if the RootSync name is root-sync, use root-reconciler. Otherwise, use root-reconciler-ROOT_SYNC_NAME. If you install Config Sync using the Google Cloud console or the Google Cloud CLI, Config Sync automatically creates a RootSync object named root-sync.
   • REPOSITORY: the ID of the repository.
   • LOCATION: the regional or multi-regional location of the repository.
   Note: Creating this policy binding requires the iam.serviceAccounts.setIamPolicy permission.

Google service account

If you store your OCI image in Artifact Registry and your cluster uses GKE Workload Identity Federation for GKE or fleet Workload Identity Federation for GKE, you can use gcpserviceaccount as your authentication type. Starting from version 1.17.2, it's recommended to use k8sserviceaccount instead. This option eliminates the extra steps of creating a Google service account and the associated IAM policy binding.

1. If you don't already have a service account, create a service account.

2. Grant the Artifact Registry Reader (roles/artifactregistry.reader) IAM role to the Google service account. For more information about Artifact Registry roles and permissions, see Configure roles and permissions for Artifact Registry.

   • Grant project-wide permission if the same permissions apply to all repositories in the project.

     gcloud projects add-iam-policy-binding PROJECT_ID  \
        --role=roles/artifactregistry.reader \
        --member="serviceAccount:GSA_NAME@PROJECT_ID.iam.gserviceaccount.com"
     

   • Grant repository-specific permission when you want service accounts to have different levels of access for each repository in your project.

     gcloud artifacts repositories add-iam-policy-binding REPOSITORY \
        --location=LOCATION \
        --role=roles/artifactregistry.reader \
        --member="serviceAccount:GSA_NAME@PROJECT_ID.iam.gserviceaccount.com" \
        --project=PROJECT_ID
     

3. Create an IAM policy binding between the Kubernetes service account and the Google service account by running the following command:

   gcloud iam service-accounts add-iam-policy-binding
     GSA_NAME@PROJECT_ID.iam.gserviceaccount.com \
       --role=roles/iam.workloadIdentityUser \
       --member="serviceAccount:FLEET_HOST_PROJECT_ID.svc.id.goog[config-management-system/KSA_NAME]" \
       --project=PROJECT_ID
   

Replace the following:

• PROJECT_ID: the organization's project ID.
• FLEET_HOST_PROJECT_ID: if you're using GKE Workload Identity Federation for GKE, this is the same as PROJECT_ID. If you're using fleet Workload Identity Federation for GKE, this is the project ID of the fleet that your cluster is registered to.
• GSA_NAME: the custom Google service account that you want to use to connect to Artifact Registry. The service account must have the Artifact Registry Reader (roles/artifactregistry.reader) IAM role.
• KSA_NAME: the Kubernetes service account for the reconciler.
  • For root repositories, if the RootSync name is root-sync, use root-reconciler. Otherwise, use root-reconciler-ROOT_SYNC_NAME. If you install Config Sync using the Google Cloud console or the Google Cloud CLI, Config Sync automatically creates a RootSync object named root-sync.
• REPOSITORY: the ID of the repository.
• LOCATION: the regional or multi-regional location of the repository.

Note: Creating this policy binding requires the iam.serviceAccounts.setIamPolicy permission.

Compute Engine default service account

If you store your Helm chart in Artifact Registry and your cluster is GKE with Workload Identity Federation for GKE disabled, you can use gcenode as your authentication type. Config Sync uses the Compute Engine default service account. You must grant your Compute Engine default service account reader access to Artifact Registry.

1. Grant the Compute Engine service account read permission to Artifact Registry by running the following command:

   gcloud projects add-iam-policy-binding PROJECT_ID \
      --member=serviceAccount:[e-mail] \
      --role=roles/artifactregistry.reader
   

   Replace PROJECT_ID with your organization's project ID, and replace PROJECT_NUMBER with your organization's project number.

Configure the Operator for a Certificate Authority

For servers configured with certificates from a Certificate Authority (CA) that is not already trusted, Config Sync can be configured to use a CA certificate to verify HTTPS connections to the server. This is supported for Git, Helm, or OCI servers. The CA certificate must include full SSL certificates (Root/Intermediate/Leaf). If your server is already using a trusted CA or you are not connecting over HTTPS, you can skip this step and leave caCertSecretRef unset.

RootSync

1. Fetch the CA certificate which was used to issue the certificate for your Git server and save it to a file.

2. For RootSync objects, the Secret must be created in the config-management-system namespace. For example:

   kubectl create ns config-management-system && 
   
   kubectl create secret generic ROOT_CA_CERT_SECRET_NAME 
   
    --namespace=config-management-system 
   
    --from-file=cert=/path/to/CA_CERT_FILE
   

3. When you configure Operator, set the value of the caCertSecretRef.name field in the RootSync object to ROOT_CA_CERT_SECRET_NAME.

RepoSync

1. Fetch the CA certificate which was used to issue the certificate for your Git server and save it to a file.

2. For RepoSync objects, the Secret must be created in the same namespace as the RepoSync. For example:

   kubectl create ns REPO_SYNC_NAMESPACE && 
   
   kubectl create secret generic NAMESPACE_CA_CERT_SECRET_NAME 
   
    --namespace=REPO_SYNC_NAMESPACE 
   
    --from-file=cert=/path/to/CA_CERT_FILE
   

3. When you configure the RepoSync, set the value of the caCertSecretRef.name field in the RepoSync object to NAMESPACE_CA_CERT_SECRET_NAME.

Grant Operator read-only access to Helm

Config Sync needs read-only access to your Helm repository so that it can read the Helm charts in your repository and install them in your clusters.

If your repository does not require authentication for read-only access, you can continue to configure Config Sync and use none as your authentication type. For example, if your Helm repository is public and can be accessed by anyone on the internet, then you don't need to authenticate.

However, most users need to create credentials to access private Helm repositories. Config Sync supports the following mechanisms for authentication:

• Token (token)
• Kubernetes service account (k8sserviceaccount)
• Google service account (gcpserviceaccount)
• Compute Engine default service account (gcenode)

Note: Config Sync doesn't support configuring helm as the source type using the Google Cloud console or the Google Cloud CLI. You can configure your RootSync or RepoSync object to sync from a Helm repository using the Kubernetes API, or declare it in another source of truth. See Configuration for the Helm repository for more information.

Token

Create a Secret with a Helm repository username and password:

kubectl create secret generic SECRET_NAME \
    --namespace=config-management-system \
    --from-literal=username=USERNAME \
    --from-literal=password=PASSWORD

Replace the following:

• SECRET_NAME: the name that you want to give your Secret.
• USERNAME: the Helm repository username.
• PASSWORD: the Helm repository password.

When you Configure the ConfigManagement Operator, you will use the Secret name you chose for spec.helm.secretRef.name.

Kubernetes service account

If you store your Helm chart in Artifact Registry and your cluster uses GKE Workload Identity Federation for GKE or fleet Workload Identity Federation for GKE, you can use k8sserviceaccount as your authentication type in version 1.17.2 and later. This option is recommended over gcpserviceaccount due to its simplified configuration process.

1. Grant the Artifact Registry Reader (roles/artifactregistry.reader) IAM role to the Kubernetes service account with the Workload Identity Federation for GKE pool. For more information about Artifact Registry roles and permissions, see Configure roles and permissions for Artifact Registry.

   • Grant project-wide permission if the same permissions apply to all repositories in the project.

     gcloud projects add-iam-policy-binding PROJECT_ID \
        --role=roles/artifactregistry.reader \
        --member="serviceAccount:FLEET_HOST_PROJECT_ID.svc.id.goog[config-management-system/KSA_NAME]"
     

   • Grant repository-specific permission when you want service accounts to have different levels of access for each repository in your project.

     gcloud artifacts repositories add-iam-policy-binding REPOSITORY \
        --location=LOCATION \
        --role=roles/artifactregistry.reader \
        --member="serviceAccount:FLEET_HOST_PROJECT_ID.svc.id.goog[config-management-system/KSA_NAME]" \
        --project=PROJECT_ID
     

   Replace the following:

   • PROJECT_ID: the organization's project ID.
   • FLEET_HOST_PROJECT_ID: if you're using GKE Workload Identity Federation for GKE, this is the same as PROJECT_ID. If you're using fleet Workload Identity Federation for GKE, this is the project ID of the fleet that your cluster is registered to.
   • KSA_NAME: the Kubernetes service account for the reconciler.
     • For root repositories, if the RootSync name is root-sync, use root-reconciler. Otherwise, use root-reconciler-ROOT_SYNC_NAME.
   • REPOSITORY: the ID of the repository.
   • LOCATION: the regional or multi-regional location of the repository.

Note: Creating this policy binding requires the iam.serviceAccounts.setIamPolicy permission.

Google service account

If you store your Helm chart in Artifact Registry and your cluster uses GKE Workload Identity Federation for GKE or fleet Workload Identity Federation for GKE, you can use gcpserviceaccount as your authentication type. Starting from version 1.17.2, it's recommended to use k8sserviceaccount instead. This option eliminates the extra steps of creating a Google service account and the associated IAM policy binding.

1. If you don't already have a service account, create a service account.

2. Grant the Artifact Registry Reader (roles/artifactregistry.reader) IAM role to the Google service account. For more information about Artifact Registry roles and permissions, see Configure roles and permissions for Artifact Registry.

   • Grant project-wide permission if the same permissions apply to all repositories in the project.

     gcloud projects add-iam-policy-binding PROJECT_ID  \
           --role=roles/artifactregistry.reader \
           --member="serviceAccount:GSA_NAME@PROJECT_ID.iam.gserviceaccount.com"
     

   • Grant repository-specific permission when you want service accounts to have different levels of access for each repository in your project.

     gcloud artifacts repositories add-iam-policy-binding REPOSITORY \
        --location=LOCATION \
        --role=roles/artifactregistry.reader \
        --member="serviceAccount:GSA_NAME@PROJECT_ID.iam.gserviceaccount.com" \
        --project=PROJECT_ID
     

3. Create an IAM policy binding between the Kubernetes service account and the Google service account by running the following command:

   gcloud iam service-accounts add-iam-policy-binding
     GSA_NAME@PROJECT_ID.iam.gserviceaccount.com \
       --role=roles/iam.workloadIdentityUser \
       --member="serviceAccount:FLEET_HOST_PROJECT_ID.svc.id.goog[config-management-system/KSA_NAME]"
       --project=PROJECT_ID
   

Replace the following:

• PROJECT_ID: the organization's project ID.
• FLEET_HOST_PROJECT_ID: if you're using GKE Workload Identity Federation for GKE, this is the same as PROJECT_ID. If you're using fleet Workload Identity Federation for GKE, this is the project ID of the fleet that your cluster is registered to.
• GSA_NAME: the custom Google service account that you want to use to connect to Artifact Registry. The service account must have the Artifact Registry Reader (roles/artifactregistry.reader) IAM role.
• KSA_NAME: the Kubernetes service account for the reconciler.
  • For root repositories, if the RootSync name is root-sync, use root-reconciler. Otherwise, use root-reconciler-ROOT_SYNC_NAME.
• REPOSITORY: the ID of the repository.
• LOCATION: the regional or multi-regional location of the repository.

Note: Creating this policy binding requires the iam.serviceAccounts.setIamPolicy permission.

Compute Engine default service account

If you store your Helm chart in Artifact Registry and your cluster is GKE with Workload Identity Federation for GKE disabled, you can use gcenode as your authentication type. Config Sync uses the Compute Engine default service account. You must grant your Compute Engine default service account reader access to Artifact Registry. You might need to grant the storage-ro access scope to grant read-only permission to pull images.

1. Grant the Compute Engine service account read permission to Artifact Registry:

   gcloud projects add-iam-policy-binding PROJECT_ID \
       --member=serviceAccount:[e-mail] \
       --role=roles/artifactregistry.reader
   

   Replace PROJECT_ID with your organization's project ID, and replace PROJECT_NUMBER with your organization's project number.

Configure the Operator

To configure syncing from the root repository, you need to enable multi-repo mode in your ConfigManagement object and create a RootSync object that syncs your root repository to the cluster. You can only create one root repository per cluster and the root repository can be either an unstructured repository or an hierarchical repository.

1. If you're using the Config Sync admission webhook (the admission webhook is disabled by default) and are installing Config Sync in a private cluster, add a firewall rule to allow port 10250. The Config Sync admission webhook uses port 10250 for drift prevention.

2. Create a file named config-management.yaml and copy the following YAML file into it:

   # config-management.yaml
   apiVersion: configmanagement.gke.io/v1
   kind: ConfigManagement
   metadata:
     name: config-management
   spec:
     # The `enableMultiRepo` field is set to true to enable RootSync and RepoSync APIs.
     enableMultiRepo: true
     preventDrift: PREVENT_DRIFT
   

   Replace the following:

   • PREVENT_DRIFT: If set to true, enables the Config Sync admission webhook to prevent drifts by rejecting conflicting changes from being pushed to live clusters. The default setting is false. Config Sync always remediates drifts no matter the value of this field.

3. Apply the changes:

   kubectl apply -f config-management.yaml
   

4. Wait for the RootSync and RepoSync CRDs to be available:

   until kubectl get customresourcedefinitions rootsyncs.configsync.gke.io reposyncs.configsync.gke.io; do date; sleep 1; echo ""; done
   

5. Save one of the following manifests as root-sync.yaml. Use the manifest version that corresponds to the source type for your configs.

   Git

   # root-sync.yaml
   apiVersion: configsync.gke.io/v1beta1
   kind: RootSync
   metadata:
     name: ROOT_SYNC_NAME
     namespace: config-management-system
   spec:
     sourceType: git
     sourceFormat: ROOT_FORMAT
     git:
       repo: ROOT_REPOSITORY
       revision: ROOT_REVISION
       branch: ROOT_BRANCH
       dir: ROOT_DIRECTORY
       auth: ROOT_AUTH_TYPE
       gcpServiceAccountEmail: ROOT_EMAIL
       secretRef:
         name: ROOT_SECRET_NAME
       noSSLVerify: ROOT_NO_SSL_VERIFY
       caCertSecretRef:
         name: ROOT_CA_CERT_SECRET_NAME
   

   Replace the following:

   • ROOT_SYNC_NAME: add the name of your RootSync object.
   • ROOT_FORMAT: add unstructured to use an unstructured repository or add hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default value is hierarchy. We recommend that you add unstructured as this format lets you organize your configs in the way that is most convenient to you.
   • ROOT_REPOSITORY: add the URL of the Git repository to use as the root repository. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/anthos-config-management-samples uses the HTTPS protocol. This field is required.
   • ROOT_REVISION: add the Git revision (tag or hash) to sync from. This field is optional and the default value is HEAD. Starting from Config Sync version 1.17.0, you can also specify a branch name in the revision field. When using a hash in version 1.17.0 or later, it must be a full hash, and not an abbreviated form.
   • ROOT_BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master. Starting from Config Sync version 1.17.0, it's recommended to use the revision field to specify a branch name for simplicity. If both the revision field and the branch field are specified, revision takes precedence over branch.
   • ROOT_DIRECTORY: add the path in the Git repository to the root directory that contains the configuration that you want to sync to. This field is optional and the default is the root directory (/) of the repository.

   • ROOT_AUTH_TYPE: add one of the following authentication types:

     • none: Use no authentication
     • ssh: Use a SSH key pair
     • cookiefile: Use a cookiefile
     • token: Use a token
     • gcpserviceaccount: Use a Google service account to access a Cloud Source Repositories.
     • gcenode: Use a Google service account to access a Cloud Source Repositories. Only select this option if Workload Identity Federation for GKE is not enabled in your cluster.

     For more information on these authentication types, see Granting Config Sync read-only access to Git.

     This field is required.

   • ROOT_EMAIL: If you added gcpserviceaccount as your ROOT_AUTH_TYPE, add your Google service account email address. For example, acm@PROJECT_ID.iam.gserviceaccount.com.

   • ROOT_SECRET_NAME: add the name of your Secret. If this field is set, you must add the Secret's public key to the Git provider. This field is optional.

   • ROOT_NO_SSL_VERIFY: To disable the SSL certificate verification, set this field to true. The default value is false.

   • ROOT_CA_CERT_SECRET_NAME: add the name of your Secret. If this field is set, your Git provider must be using a certificate issued by this certificate authority (CA). The Secret must contain the CA certificate under a key named cert. This field is optional.

     To learn more about how to configure the Secret object for the CA certificate, see Configure Certificate Authority

   For an explanation of the fields and a complete list of fields that you can add to the spec field, see RootSync fields.

   This manifest creates a RootSync object that uses Git as the source.

   OCI

   # root-sync.yaml
   apiVersion: configsync.gke.io/v1beta1
   kind: RootSync
   metadata:
     name: ROOT_SYNC_NAME
     namespace: config-management-system
   spec:
     sourceType: oci
     sourceFormat: ROOT_FORMAT
     oci:
       image: ROOT_IMAGE
       dir: ROOT_DIRECTORY
       auth: ROOT_AUTH_TYPE
       gcpServiceAccountEmail: ROOT_EMAIL
       caCertSecretRef:
         name: ROOT_CA_CERT_SECRET_NAME
   

   Replace the following:

   • ROOT_SYNC_NAME: add the name of your RootSync object.
   • ROOT_FORMAT: add unstructured to use an unstructured repository or add hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default value is hierarchy. We recommend that you add unstructured as this format lets you organize your configs in the way that is most convenient to you.
   • ROOT_IMAGE: the URL of the OCI image to use as the root repository, for example LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY_NAME/PACKAGE_NAME. By default, the image is pulled from the latest tag, but you can pull in images by TAG or DIGEST instead. Specify TAG or DIGEST in the PACKAGE_NAME:
     • To pull by TAG: LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY_NAME/PACKAGE_NAME:TAG
     • To pull by DIGEST: LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY_NAME/PACKAGE_NAME@sha256:DIGEST

   Caution: Config Sync requires that the OCI layer is compressed in the tar or tar+gzip. Other formats (for example, tar+bz2) will not be recognized by Config Sync. Switching from a valid REPO to an OCI image with an unsupported format will cause managed resources to be pruned without an error.

   • ROOT_DIRECTORY: add the path in the repository to the root directory that contains the configuration that you want to sync to. This field is optional and the default is the root directory (/) of the repository.

   • ROOT_AUTH_TYPE: add one of the following authentication types:

     • none: Use no authentication
     • gcenode: Use the Compute Engine default service account to access an image in Artifact Registry. Only select this option if Workload Identity Federation for GKE is not enabled in your cluster.
     • gcpserviceaccount: Use a Google service account to access an image.

     This field is required.

   • ROOT_EMAIL: If you added gcpserviceaccount as your ROOT_AUTH_TYPE, add your Google service account email address. For example, acm@PROJECT_ID.iam.gserviceaccount.com.

   • ROOT_CA_CERT_SECRET_NAME: add the name of your Secret. If this field is set, your OCI provider must be using a certificate issued by this certificate authority (CA). The Secret must contain the CA certificate under a key named cert. This field is optional.

   To learn more about how to configure the Secret object for the CA certificate, see Configure Certificate Authority

   For an explanation of the fields and a complete list of fields that you can add to the spec field, see RootSync fields.

   This manifest creates a RootSync object that uses an OCI image as the source.

   Helm

   # root-sync.yaml
   apiVersion: configsync.gke.io/v1beta1
   kind: RootSync
   metadata:
     name: ROOT_SYNC_NAME
     namespace: config-management-system
   spec:
     sourceType: helm
     sourceFormat: ROOT_FORMAT
     helm:
       repo: ROOT_HELM_REPOSITORY
       chart: HELM_CHART_NAME
       version: HELM_CHART_VERSION
       releaseName: HELM_RELEASE_NAME
       namespace: HELM_RELEASE_NAMESPACE
       values:
         foo:
           bar: VALUE_1
         baz:
         - qux: VALUE_2
           xyz: VALUE_3
       includeCRDs: HELM_INCLUDE_CRDS
       auth: ROOT_AUTH_TYPE
         gcpServiceAccountEmail: ROOT_EMAIL
         secretRef:
           name: ROOT_SECRET_NAME
       caCertSecretRef:
         name: ROOT_CA_CERT_SECRET_NAME
   

   Replace the following:

   • ROOT_SYNC_NAME: add the name of your RootSync object.
   • ROOT_FORMAT: add unstructured to use an unstructured repository or add hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default value is hierarchy. We recommend that you add unstructured as this format lets you organize your configs in the way that is most convenient to you.
   • ROOT_HELM_REPOSITORY: the URL of the Helm repository to use as the root repository. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/anthos-config-management-samples uses the HTTPS protocol. This field is required.
   • HELM_CHART_NAME: add the name of your Helm chart. This field is required.
   • HELM_CHART_VERSION: the version of your chart. This field is optional. If no value is specified, the latest version is used.
   • HELM_RELEASE_NAME: the name of the Helm release. This field is optional.
   • HELM_RELEASE_NAMESPACE: the target namespace for a release. It only sets a namespace for resources that contain namespace: {{ .Release.Namespace }} in their templates. This field is optional. If no value is specified, the default namespace config-management-system is used.
   • HELM_INCLUDE_CRDS: set to true if you want the Helm template to also generate a CustomResourceDefinition. This field is optional. If no value is specified, the default is false and a CRD won't be generated.
   • VALUE: values to use instead of default values that accompany the Helm chart. Format this field the same way as helm chart's values.yaml file. This field is optional.

   • ROOT_AUTH_TYPE: add one of the following authentication types:

     • none: Use no authentication
     • token: Use a username and password to access a private Helm repository.
     • gcenode: Use the Compute Engine default service account to access an image in Artifact Registry. Only select this option if Workload Identity Federation for GKE is not enabled in your cluster.
     • gcpserviceaccount: Use a Google service account to access an image.

     This field is required.

   • ROOT_EMAIL: If you added gcpserviceaccount as your ROOT_AUTH_TYPE, add your Google service account email address. For example, acm@PROJECT_ID.iam.gserviceaccount.com.

   • ROOT_SECRET_NAME: add the name of your Secret if token is the ROOT_AUTH_TYPE. This field is optional.

   • ROOT_CA_CERT_SECRET_NAME: add the name of your Secret. If this field is set, your Helm provider must be using a certificate issued by this certificate authority (CA). The Secret must contain the CA certificate under a key named cert. This field is optional.

   To learn more about how to configure the Secret object for the CA certificate, see Configure Certificate Authority

   For an explanation of the fields and a complete list of fields that you can add to the spec field, see RootSync fields.

   This manifest creates a RootSync object that uses Helm as the source.

6. Apply the changes:

   kubectl apply -f root-sync.yaml
   

Verify the sync status of the root repository

You can use the nomos status command to inspect the sync status of the root repository:

nomos status

You should see output similar to the following example:

my_managed_cluster-1
  --------------------
  <root>   [e-mail]:foo-corp/acme/admin@main
  SYNCED   f52a11e4

Verify the RootSync installation

When you create a RootSync object, Config Sync creates a reconciler with the root-reconciler prefix. A reconciler is a Pod that is deployed as a Deployment. It syncs manifests from a Git repository to a cluster.

You can verify that the RootSync object is working correctly by checking the status of the root-reconciler Deployment:

kubectl get -n config-management-system deployment \
    -l configsync.gke.io/sync-name=ROOT_SYNC_NAME

Replace ROOT_SYNC_NAME with the name of RootSync.

You should see output similar to the following example:

NAME              READY   UP-TO-DATE   AVAILABLE   AGE
root-reconciler   1/1     1            1           3h42m

For further ways to explore the status of your RootSync object, see Monitoring RootSync and RepoSync objects.

After you have finished configuring your root repository, you can optionally choose to configure syncing from multiple repositories. These repositories are helpful if you want a repository that contains namespace-scoped configs synced to a particular namespace across clusters.

Upgrade Config Sync

Note: Upgrading from a version earlier than 1.10.0 would remove all the Config Sync admission webhook resources.

Note: If upgrading from an unsupported version, you should perform a step-by-step upgrade with increments to no more than three minor versions at a time. For example, if the current Config Sync version is 1.11.0, first upgrade to version 1.14.0, and then to version 1.17.0.

To upgrade Config Sync, run these commands for each enrolled cluster:

1. Download the Config Sync manifest and nomos commands for the new version.

2. Apply the Config Sync manifest:

   kubectl apply -f config-management-operator.yaml
   

   This command updates the ConfigManagement Operator image. Kubernetes retrieves the new version and restarts the Config Sync Pod using the new version. When Config Sync starts, it runs a reconcile loop that applies the set of manifests bundled in the new image. This updates and restarts each component Pod.

3. Replace the nomos command on all clients with the new version. This change ensures that the nomos command can always get the status of all enrolled clusters and can validate configs for them.

Uninstall Config Sync

Caution: Deleting a RootSync or RepoSync object does not clean up Config Sync annotations and labels, and the Config Sync admission webhook denies requests trying to delete these resources if Config Sync is still enabled in the cluster. The workaround is to unmanage or delete these managed resources before deleting the corresponding RootSync or RepoSync object.

To uninstall Config Sync, complete the following steps:

1. A central administrator should remove the root repo:

   1. If you've enabled the webhook and want to keep your resources, disable drift prevention for abandoned resources. If you haven't enabled the webhook, you don't have to take any additional steps to keep your resources.

   2. Delete the RootSync object by running the following command:

      kubectl delete -f root-sync.yaml
      

2. Remove any repositories.

3. Remove the spec.enableMultiRepo field in your config-management.yaml file.

4. Apply your config-management.yaml file to your cluster.

Note: Disabling this feature does not affect any configurations that Config Sync previously applied to your clusters.

If you want to fully uninstall Config Sync, see Removing the ConfigManagement Operator.

What's next

• Discover how to configure syncing from multiple repositories.