Install Config Sync manually using kubectl (not recommended)

This page shows you how to install Config Sync using kubectl commands.

Before you begin

This section describes prerequisites you must meet before installing Config Sync using kubectl.

Prepare your local environment

Before you install Config Sync, make sure you have prepared your local environment by completing the following tasks:

  • Create, or have access to a source of truth.

  • Install and initialize the Google Cloud CLI, which provides the gcloud, kubectl, and nomos commands used in these instructions. If you use Cloud Shell, Google Cloud CLI comes pre-installed.

  • kubectl is not installed by default by Google Cloud CLI. To install kubectl, use the following command:

    gcloud components install kubectl
    
  • Authenticate to Google Cloud using the gcloud auth login command so that you can download components of Config Sync.

Prepare your clusters

Create, or have access to, a Google Kubernetes Engine (GKE) Enterprise edition cluster that meets the requirements for Config Sync.

Prepare permissions

The Google Cloud user installing Config Sync needs IAM permissions to create new roles in your cluster. If needed, grant these roles with the following commands:

gcloud container clusters get-credentials CLUSTER_NAME

kubectl create clusterrolebinding cluster-admin-binding \
    --clusterrole cluster-admin --user USER_ACCOUNT

Replace the following:

  • CLUSTER_NAME: your cluster name
  • USER_ACCOUNT: your Google Cloud account's email address

Depending on how you configured the Google Cloud CLI on your local system, you might need to add the --project and --zone fields.

If you need to grant Config Sync access to OCI using gcpserviceaccount as your authentication type, to create a policy binding, you must also have the iam.serviceAccounts.setIamPolicy permission. You can get this permission by granting the Service Account Admin (roles/iam.serviceAccountAdmin) IAM role. You might also be able to get this permission with custom roles or other predefined roles.

For more information about granting roles, see Manage access.

Enroll a cluster

To enroll a cluster in Config Sync, complete the following steps:

  1. Deploy Config Sync
  2. Grant Config Sync read-only access to one of the following:
  3. Configure Config Sync

Deploy Config Sync

After ensuring that you meet all the prerequisites, you can deploy Config Sync by downloading and applying a YAML manifest.

1.20.0 or later

  1. Download the latest version of the Config Sync manifests using the following command. To download a specific version instead, see Downloads.

    gcloud storage cp gs://config-management-release/released/latest/config-sync.tar.gz config-sync.tar.gz
    
  2. Extract the archive:

    tar -xzvf config-sync.tar.gz
  3. In the archive you extracted in the previous step, follow the instructions in the provided README to edit the kustomization.

  4. To update the Config Sync installation, apply the rendered manifest that you built following the README instructions:

    kubectl apply -f CONFIG_SYNC_MANIFEST
    

    Replace CONFIG_SYNC_MANIFEST with the name of the rendered manifest.

  5. Replace the nomos command on all clients with the new version. This change ensures that the nomos command can always get the status of all enrolled clusters and can validate configs for them.

1.19.2 or earlier

  1. Download the Config Sync manifest and nomos commands for the new version.

  2. Apply the Config Sync manifest:

    kubectl apply -f config-management-operator.yaml
    
  3. Create a file named config-management.yaml and copy the following YAML file into it:

    # config-management.yaml
    apiVersion: configmanagement.gke.io/v1
    kind: ConfigManagement
    metadata:
      name: config-management
    spec:
      # The `enableMultiRepo` field is set to true to enable RootSync and RepoSync APIs.
      enableMultiRepo: true
      preventDrift: PREVENT_DRIFT
    

    Replace the following:

    • PREVENT_DRIFT: If set to true, enables the Config Sync admission webhook to prevent drifts by rejecting conflicting changes from being pushed to live clusters. The default setting is false. Config Sync always remediates drifts no matter the value of this field.
  4. Apply the changes:

    kubectl apply -f config-management.yaml
    
  5. Replace the nomos command on all clients with the new version. This change ensures that the nomos command can always get the status of all enrolled clusters and can validate configs for them.

If this fails because of a problem with Config Sync that is not due to a YAML or JSON syntax error, the object might be instantiated in the cluster, but might not work correctly. In this situation, you can use the nomos status command to check for errors in the object.

A valid installation with no problems has a status of PENDING or SYNCED.

An invalid installation has a status of NOT CONFIGURED and lists one of the following errors:

  • missing git-creds Secret
  • git-creds Secret is missing the key specified by secretType

To fix the problem, correct the configuration error. Depending on the type of error, you might need to re-apply the Config Sync manifest to the cluster.

If the problem is that you forgot to create the git-creds Secret, Config Sync detects the Secret as soon as you create it, and you don't need to re-apply the configuration.

Grant Config Sync read-only access

If you store your configs in Git, you must grant Config Sync read-only access to Git. If you store your configs as OCI images, you must grant Config Sync read-only access to OCI. If you store your configs in Helm, you must grant Config Sync read-only access to Helm.

Grant Config Sync read-only access to Git

Config Sync needs read-only access to your Git repository so that it can read the configs committed to the repository and apply them to your clusters.

If your repository does not require authentication for read-only access, you can continue to configure Config Sync and use none as your authentication type. For example, if you can browse the repository using a web interface without logging in, or if you can use git clone to create a clone of the repository locally without providing credentials or using saved credentials, then you don't need to authenticate. In this case, you don't need to create a Secret.

However, most users need to create credentials because read access to their repository is restricted. If credentials are required, they are stored in the git-creds Secret on each enrolled cluster (unless you are using a Google service account). The Secret must be named git-creds because this is a fixed value.

Config Sync supports the following mechanisms for authentication:

  • SSH key pair (ssh)
  • Cookiefile (cookiefile)
  • Token (token)
  • Google service account (gcpserviceaccount)
  • Compute Engine default service account (gcenode)
  • GitHub App (githubapp)

The mechanism that you choose depends on what your repository supports. Generally, we recommend using an SSH key pair. GitHub and Bitbucket both support using an SSH key pair. However, if you are using a repository in Cloud Source Repositories, we recommend that you use a Google service account instead as the process is simpler. If your organization hosts your repository and you don't know which authentication methods are supported, contact your administrator.

To use a repository in Cloud Source Repositories as your Config Sync repository, complete the following steps to retrieve your Cloud Source Repositories URL:

  1. List all repositories:

    gcloud source repos list
    
  2. From the output, copy the URL from the repository that you want to use. For example:

    REPO_NAME  PROJECT_ID  URL
    my-repo    my-project  https://source.developers.google.com/p/my-project/r/my-repo-csr
    

    You need to use this URL when you configure Config Sync in the following section. If you configure Config Sync using the Google Cloud console, you add the URL in the URL field. If you configure Config Sync using the Google Cloud CLI, you add the URL to the syncRepo field of your configuration file.

SSH key pair

An SSH key pair consists of two files, a public key and a private key. The public key typically has a .pub extension.

To use an SSH key pair, complete the following steps:

  1. Create an SSH key pair to allow Config Sync to authenticate to your Git repository. This step is necessary if you need to authenticate to the repository to clone it or read from it. Skip this step if a security administrator provides you with a key pair. You can use a single key pair for all clusters, or a key pair per cluster, depending on your security and compliance requirements.

    The following command creates a 4096-bit RSA key. Lower values are not recommended:

    ssh-keygen -t rsa -b 4096 \
    -C "GIT_REPOSITORY_USERNAME" \
    -N '' \
    -f /path/to/KEYPAIR_FILENAME
    

    Replace the following:

    • GIT_REPOSITORY_USERNAME: the username that you want Config Sync to use to authenticate to the repository
    • /path/to/KEYPAIR_FILENAME: a path to the key pair

    If you are using a third-party Git repository host such as GitHub, or you want to use a service account with Cloud Source Repositories, we recommend that you use a separate account.

  2. Configure your repository to recognize the newly created public key. Refer to the documentation for your Git hosting provider. Instructions for some popular Git hosting providers are included for convenience:

  3. Add the private key to a new Secret in the cluster:

    kubectl create ns config-management-system && \
    kubectl create secret generic git-creds \
     --namespace=config-management-system \
     --from-file=ssh=/path/to/KEYPAIR_PRIVATE_KEY_FILENAME
    

    Replace /path/to/KEYPAIR_PRIVATE_KEY_FILENAME with the name of the private key (the one without the .pub suffix).

  4. (Recommended) To configure known hosts checking using SSH authentication, you can add the known hosts key to the data.known_hosts field in the git_creds secret. To disable known_hosts checking, you can remove the known_hosts field from the secret. To add the known hosts key, run:

    kubectl edit secret git-creds \
     --namespace=config-management-system
    

    Then, under data, add the known hosts entry:

    known_hosts: KNOWN_HOSTS_KEY
    
  5. Delete the private key from the local disk or otherwise protect it.

  6. When you configure Config Sync and add the URL for your Git repository, use the SSH protocol. If you are using a repository in Cloud Source Repositories, you must use the following format when you enter your URL:

    ssh://EMAIL@source.developers.google.com:2022/p/PROJECT_ID/r/REPO_NAME
    

    Replace the following:

    • EMAIL: your Google Cloud username
    • PROJECT_ID: the ID of the Google Cloud project where the repository is located
    • REPO_NAME: the name of the repository

Cookiefile

The process for acquiring a cookiefile depends on the configuration of your repository. For an example, see Generate static credentials in the Cloud Source Repositories documentation. The credentials are usually stored in the .gitcookies file in your home directory, or they might be provided to you by a security administrator.

To use a cookiefile, complete the following steps:

  1. After you create and obtain the cookiefile, add it to a new Secret in the cluster.

    If you don't use an HTTPS proxy, create the Secret with the following command:

    kubectl create ns config-management-system && \
    kubectl create secret generic git-creds \
     --namespace=config-management-system \
     --from-file=cookie_file=/path/to/COOKIEFILE
    

    If you need to use an HTTPS proxy, add it to the Secret together with cookiefile by running the following command:

    kubectl create ns config-management-system && \
    kubectl create secret generic git-creds \
     --namespace=config-management-system \
     --from-file=cookie_file=/path/to/COOKIEFILE \
     --from-literal=https_proxy=HTTPS_PROXY_URL
    

    Replace the following:

    • /path/to/COOKIEFILE: the appropriate path and filename
    • HTTPS_PROXY_URL: the URL for the HTTPS proxy that you use when communicating with the Git repository
  2. Protect the contents of the cookiefile if you still need it locally. Otherwise, delete it.

Token

If your organization does not permit the use of SSH keys, you might prefer to use a token. With Config Sync, you can use GitHub's personal access tokens (PATs), GiLab's PATs or deploy keys, or Bitbucket's app password as your token.

To create a Secret using your token, complete the following steps:

  1. Create a token using GitHub, GitLab, or Bitbucket:

  2. After you create and obtain the token, add it to a new Secret in the cluster.

    If you don't use an HTTPS proxy, create the Secret with the following command:

    kubectl create ns config-management-system && \
    kubectl create secret generic git-creds \
      --namespace="config-management-system" \
      --from-literal=username=USERNAME \
      --from-literal=token=TOKEN
    

    Replace the following:

    • USERNAME: the username that you want to use.
    • TOKEN: the token that you created in the previous step.

    If you need to use an HTTPS proxy, add it to the Secret together with username and token by running the following command:

    kubectl create ns config-management-system && \
    kubectl create secret generic git-creds \
     --namespace=config-management-system \
     --from-literal=username=USERNAME \
      --from-literal=token=TOKEN \
     --from-literal=https_proxy=HTTPS_PROXY_URL
    

    Replace the following:

    • USERNAME: the username that you want to use.
    • TOKEN: the token that you created in the previous step.
    • HTTPS_PROXY_URL: the URL for the HTTPS proxy that you use when communicating with the Git repository.
  3. Protect the token if you still need it locally. Otherwise, delete it.

Google service account

If your repository is in Cloud Source Repositories, and your cluster uses GKE Workload Identity Federation for GKE or fleet Workload Identity Federation for GKE, you can give Config Sync access to a repository in the same project as your managed cluster by using a Google service account.

  1. If you don't already have a service account, create a service account.

  2. Grant the Cloud Source Repositories Reader (roles/source.reader) IAM role to the Google service account. For more information about Cloud Source Repositories roles and permissions, see Grant permissions to view repositories.

    • Grant project-wide permission if the same permissions apply to all repositories in the project.

      gcloud projects add-iam-policy-binding PROJECT_ID \
        --role=roles/source.reader \
        --member="serviceAccount:GSA_NAME@PROJECT_ID.iam.gserviceaccount.com"
      
    • Grant repository-specific permission when you want service accounts to have different levels of access for each repository in your project.

      gcloud source repos set-iam-policy REPOSITORY POLICY_FILE --project=PROJECT_ID
      
  3. If you configure Config Sync using the Google Cloud console, select Workload Identity Federation for GKE as the Authentication Type and then add your service account email.

    If you configure Config Sync using the Google Cloud CLI, add gcpserviceaccount as the secretType and then add your service account email to gcpServiceAccountEmail.

  4. After configuring Config Sync, create an IAM policy binding between the Kubernetes service account and the Google service account. The Kubernetes service account is not created until you configure Config Sync for the first time.

    If you are using clusters that are registered to a fleet, you only have to create the policy binding once per fleet. All clusters registered in a fleet share the same Workload Identity Federation for GKEpool. With fleet's concept of sameness, if you add the IAM policy to your Kubernetes service account in one cluster, then the Kubernetes service account from the same namespace on other clusters in the same fleet also get the same IAM policy.

    This binding lets the Config Sync Kubernetes service account act as the Google service account:

    gcloud iam service-accounts add-iam-policy-binding \
        GSA_NAME@PROJECT_ID.iam.gserviceaccount.com \
        --role=roles/iam.workloadIdentityUser \
        --member="serviceAccount:FLEET_HOST_PROJECT_ID.svc.id.goog[config-management-system/KSA_NAME]" \
        --project=PROJECT_ID
    

Replace the following:

  • PROJECT_ID: the organization's project ID.
  • FLEET_HOST_PROJECT_ID: if you're using GKE Workload Identity Federation for GKE, this is the same as PROJECT_ID. If you're using fleet Workload Identity Federation for GKE, this is the project ID of the fleet that your cluster is registered to.
  • GSA_NAME: the custom Google service account that you want to use to connect to Artifact Registry. The service account must have the Artifact Registry Reader (roles/artifactregistry.reader) IAM role.
  • KSA_NAME: the Kubernetes service account for the reconciler.
    • For root repositories, if the RootSync name is root-sync, use root-reconciler. Otherwise, use root-reconciler-ROOT_SYNC_NAME. If you install Config Sync using the Google Cloud console or the Google Cloud CLI, Config Sync automatically creates a RootSync object named root-sync.
  • REPOSITORY: the name of the repository.
  • POLICY_FILE: the JSON or YAML file with the Identity and Access Management policy.

Compute Engine default service account

If your repository is in Cloud Source Repositories, and your cluster is GKE with Workload Identity Federation for GKE disabled, you can use gcenode as your authentication type.

If you configure Config Sync using the Google Cloud console, select Google Cloud Repository as the Authentication Type.

If you configure Config Sync using the Google Cloud CLI, add gcenode as the secretType.

Selecting either Google Cloud Repository or gcenode lets you use the Compute Engine default service account. You must grant the Cloud Source Repositories Reader (roles/source.reader) IAM role to the Compute Engine default service account. For more information about Cloud Source Repositories roles and permissions, see Grant permissions to view repositories.

gcloud projects add-iam-policy-binding PROJECT_ID \
  --role=roles/source.reader \
  --member="serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com"

Replace PROJECT_ID with your organization's project ID, and replace PROJECT_NUMBER with your organization's project number.

GitHub App

If your repository is in GitHub, you can use githubapp as your authentication type.

To use a GitHub App, complete the following steps:

  1. Follow the instructions on GitHub to provision a GitHub App and give it permission to read from your repository.

  2. Add the GitHub App configuration to a new Secret in the cluster:

    Using Client ID

    kubectl create ns config-management-system && \
    kubectl create secret generic git-creds \
      --namespace=config-management-system \
      --from-literal=github-app-client-id=CLIENT_ID \
      --from-literal=github-app-installation-id=INSTALLATION_ID \
      --from-file=github-app-private-key=/path/to/GITHUB_PRIVATE_KEY \
      --from-literal=github-app-base-url=BASE_URL
    
    • Replace CLIENT_ID with the client ID for the GitHub App.
    • Replace INSTALLATION_ID with the installation ID for the GitHub App.
    • Replace /path/to/GITHUB_PRIVATE_KEY with the name of the file containing the private key.
    • Replace BASE_URL with the base URL for the GitHub API endpoint. This is only needed when the repository is not hosted at www.github.com. The argument can otherwise be omitted and will default to https://api.github.com/.

    Using Application ID

    kubectl create ns config-management-system && \
    kubectl create secret generic git-creds \
      --namespace=config-management-system \
      --from-literal=github-app-application-id=APPLICATION_ID \
      --from-literal=github-app-installation-id=INSTALLATION_ID \
      --from-file=github-app-private-key=/path/to/GITHUB_PRIVATE_KEY \
      --from-literal=github-app-base-url=BASE_URL
    
    • Replace APPLICATION_ID with the application ID for the GitHub App.
    • Replace INSTALLATION_ID with the installation ID for the GitHub App.
    • Replace /path/to/GITHUB_PRIVATE_KEY with the name of the file containing the private key.
    • Replace BASE_URL with the base URL for the GitHub API endpoint. This is only needed when the repository is not hosted at www.github.com. The argument can otherwise be omitted and will default to https://api.github.com/.
  3. Delete the private key from the local disk or otherwise protect it.

  4. When you configure Config Sync and add the URL for your Git repository, use the githubapp auth type.

Grant Config Sync read-only access to OCI

Config Sync needs read-only access to your OCI image stored in Artifact Registry so that it can read the configs included in the image and apply them to your clusters.

If your image does not require authentication for read-only access, you can continue to configure Config Sync and use none as your authentication type. For example, if your image is public and can be accessed by anyone on the internet, then you don't need to authenticate.

However, most users need to create credentials to access restricted images. Config Sync supports the following mechanisms for authentication:

  • Kubernetes service account (k8sserviceaccount)
  • Google service account (gcpserviceaccount)
  • Compute Engine default service account (gcenode)

Kubernetes service account

If you store your OCI image in Artifact Registry and your cluster uses GKE Workload Identity Federation for GKE or fleet Workload Identity Federation for GKE, you can use k8sserviceaccount as your authentication type in version 1.17.2 and later. This option is recommended over gcpserviceaccount due to its simplified configuration process.

  1. Grant the Artifact Registry Reader (roles/artifactregistry.reader) IAM role to the Kubernetes service account with the Workload Identity Federation for GKE pool. For more information about Artifact Registry roles and permissions, see Configure roles and permissions for Artifact Registry.

    • Grant project-wide permission if the same permissions apply to all repositories in the project.

      gcloud projects add-iam-policy-binding PROJECT_ID \
            --role=roles/artifactregistry.reader \
            --member="serviceAccount:FLEET_HOST_PROJECT_ID.svc.id.goog[config-management-system/KSA_NAME]"
      
    • Grant repository-specific permission when you want service accounts to have different levels of access for each repository in your project.

      gcloud artifacts repositories add-iam-policy-binding REPOSITORY \
         --location=LOCATION \
         --role=roles/artifactregistry.reader \
         --member="serviceAccount:FLEET_HOST_PROJECT_ID.svc.id.goog[config-management-system/KSA_NAME]" \
         --project=PROJECT_ID
      

    Replace the following:

    • PROJECT_ID: the organization's project ID.
    • FLEET_HOST_PROJECT_ID: if you're using GKE Workload Identity Federation for GKE, this is the same as PROJECT_ID. If you're using fleet Workload Identity Federation for GKE, this is the project ID of the fleet that your cluster is registered to.
    • KSA_NAME: the Kubernetes service account for the reconciler.
      • For root repositories, if the RootSync name is root-sync, use root-reconciler. Otherwise, use root-reconciler-ROOT_SYNC_NAME. If you install Config Sync using the Google Cloud console or the Google Cloud CLI, Config Sync automatically creates a RootSync object named root-sync.
    • REPOSITORY: the ID of the repository.
    • LOCATION: the regional or multi-regional location of the repository.

Google service account

If you store your OCI image in Artifact Registry and your cluster uses GKE Workload Identity Federation for GKE or fleet Workload Identity Federation for GKE, you can use gcpserviceaccount as your authentication type. Starting from version 1.17.2, it's recommended to use k8sserviceaccount instead. This option eliminates the extra steps of creating a Google service account and the associated IAM policy binding.

  1. If you don't already have a service account, create a service account.

  2. Grant the Artifact Registry Reader (roles/artifactregistry.reader) IAM role to the Google service account. For more information about Artifact Registry roles and permissions, see Configure roles and permissions for Artifact Registry.

    • Grant project-wide permission if the same permissions apply to all repositories in the project.

      gcloud projects add-iam-policy-binding PROJECT_ID  \
         --role=roles/artifactregistry.reader \
         --member="serviceAccount:GSA_NAME@PROJECT_ID.iam.gserviceaccount.com"
      
    • Grant repository-specific permission when you want service accounts to have different levels of access for each repository in your project.

      gcloud artifacts repositories add-iam-policy-binding REPOSITORY \
         --location=LOCATION \
         --role=roles/artifactregistry.reader \
         --member="serviceAccount:GSA_NAME@PROJECT_ID.iam.gserviceaccount.com" \
         --project=PROJECT_ID
      
  3. Create an IAM policy binding between the Kubernetes service account and the Google service account by running the following command:

    gcloud iam service-accounts add-iam-policy-binding
      GSA_NAME@PROJECT_ID.iam.gserviceaccount.com \
        --role=roles/iam.workloadIdentityUser \
        --member="serviceAccount:FLEET_HOST_PROJECT_ID.svc.id.goog[config-management-system/KSA_NAME]" \
        --project=PROJECT_ID
    

Replace the following:

  • PROJECT_ID: the organization's project ID.
  • FLEET_HOST_PROJECT_ID: if you're using GKE Workload Identity Federation for GKE, this is the same as PROJECT_ID. If you're using fleet Workload Identity Federation for GKE, this is the project ID of the fleet that your cluster is registered to.
  • GSA_NAME: the custom Google service account that you want to use to connect to Artifact Registry. The service account must have the Artifact Registry Reader (roles/artifactregistry.reader) IAM role.
  • KSA_NAME: the Kubernetes service account for the reconciler.
    • For root repositories, if the RootSync name is root-sync, use root-reconciler. Otherwise, use root-reconciler-ROOT_SYNC_NAME. If you install Config Sync using the Google Cloud console or the Google Cloud CLI, Config Sync automatically creates a RootSync object named root-sync.
  • REPOSITORY: the ID of the repository.
  • LOCATION: the regional or multi-regional location of the repository.

Compute Engine default service account

If you store your Helm chart in Artifact Registry and your cluster is GKE with Workload Identity Federation for GKE disabled, you can use gcenode as your authentication type. Config Sync uses the Compute Engine default service account. You must grant your Compute Engine default service account reader access to Artifact Registry.

  1. Grant the Compute Engine service account read permission to Artifact Registry by running the following command:

    gcloud projects add-iam-policy-binding PROJECT_ID \
       --member=serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com \
       --role=roles/artifactregistry.reader
    

    Replace PROJECT_ID with your organization's project ID, and replace PROJECT_NUMBER with your organization's project number.

Configure Config Sync for a Certificate Authority

For servers configured with certificates from a Certificate Authority (CA) that is not already trusted, Config Sync can be configured to use a CA certificate to verify HTTPS connections to the server. This is supported for Git, Helm, or OCI servers. The CA certificate must include full SSL certificates (Root/Intermediate/Leaf). If your server is already using a trusted CA or you are not connecting over HTTPS, you can skip this step and leave caCertSecretRef unset.

RootSync

  1. Fetch the CA certificate which was used to issue the certificate for your Git server and save it to a file.

  2. For RootSync objects, the Secret must be created in the config-management-system namespace. For example:

    kubectl create ns config-management-system && 
    kubectl create secret generic ROOT_CA_CERT_SECRET_NAME
    --namespace=config-management-system
    --from-file=cert=/path/to/CA_CERT_FILE

  3. When you configure Config Sync, set the value of the caCertSecretRef.name field in the RootSync object to ROOT_CA_CERT_SECRET_NAME.

RepoSync

  1. Fetch the CA certificate which was used to issue the certificate for your Git server and save it to a file.

  2. For RepoSync objects, the Secret must be created in the same namespace as the RepoSync. For example:

    kubectl create ns REPO_SYNC_NAMESPACE && 
    kubectl create secret generic NAMESPACE_CA_CERT_SECRET_NAME
    --namespace=REPO_SYNC_NAMESPACE
    --from-file=cert=/path/to/CA_CERT_FILE

  3. When you configure the RepoSync, set the value of the caCertSecretRef.name field in the RepoSync object to NAMESPACE_CA_CERT_SECRET_NAME.

Grant Config Sync read-only access to Helm

Config Sync needs read-only access to your Helm repository so that it can read the Helm charts in your repository and install them in your clusters.

If your repository does not require authentication for read-only access, you can continue to configure Config Sync and use none as your authentication type. For example, if your Helm repository is public and can be accessed by anyone on the internet, then you don't need to authenticate.

However, most users need to create credentials to access private Helm repositories. Config Sync supports the following mechanisms for authentication:

  • Token (token)
  • Kubernetes service account (k8sserviceaccount)
  • Google service account (gcpserviceaccount)
  • Compute Engine default service account (gcenode)

Token

Create a Secret with a Helm repository username and password:

kubectl create secret generic SECRET_NAME \
    --namespace=config-management-system \
    --from-literal=username=USERNAME \
    --from-literal=password=PASSWORD

Replace the following:

  • SECRET_NAME: the name that you want to give your Secret.
  • USERNAME: the Helm repository username.
  • PASSWORD: the Helm repository password.

When you Configure Config Sync, you will use the Secret name you chose for spec.helm.secretRef.name.

Kubernetes service account

If you store your Helm chart in Artifact Registry and your cluster uses GKE Workload Identity Federation for GKE or fleet Workload Identity Federation for GKE, you can use k8sserviceaccount as your authentication type in version 1.17.2 and later. This option is recommended over gcpserviceaccount due to its simplified configuration process.

  1. Grant the Artifact Registry Reader (roles/artifactregistry.reader) IAM role to the Kubernetes service account with the Workload Identity Federation for GKE pool. For more information about Artifact Registry roles and permissions, see Configure roles and permissions for Artifact Registry.

    • Grant project-wide permission if the same permissions apply to all repositories in the project.

      gcloud projects add-iam-policy-binding PROJECT_ID \
         --role=roles/artifactregistry.reader \
         --member="serviceAccount:FLEET_HOST_PROJECT_ID.svc.id.goog[config-management-system/KSA_NAME]"
      
    • Grant repository-specific permission when you want service accounts to have different levels of access for each repository in your project.

      gcloud artifacts repositories add-iam-policy-binding REPOSITORY \
         --location=LOCATION \
         --role=roles/artifactregistry.reader \
         --member="serviceAccount:FLEET_HOST_PROJECT_ID.svc.id.goog[config-management-system/KSA_NAME]" \
         --project=PROJECT_ID
      

    Replace the following:

    • PROJECT_ID: the organization's project ID.
    • FLEET_HOST_PROJECT_ID: if you're using GKE Workload Identity Federation for GKE, this is the same as PROJECT_ID. If you're using fleet Workload Identity Federation for GKE, this is the project ID of the fleet that your cluster is registered to.
    • KSA_NAME: the Kubernetes service account for the reconciler.
      • For root repositories, if the RootSync name is root-sync, use root-reconciler. Otherwise, use root-reconciler-ROOT_SYNC_NAME.
    • REPOSITORY: the ID of the repository.
    • LOCATION: the regional or multi-regional location of the repository.

Google service account

If you store your Helm chart in Artifact Registry and your cluster uses GKE Workload Identity Federation for GKE or fleet Workload Identity Federation for GKE, you can use gcpserviceaccount as your authentication type. Starting from version 1.17.2, it's recommended to use k8sserviceaccount instead. This option eliminates the extra steps of creating a Google service account and the associated IAM policy binding.

  1. If you don't already have a service account, create a service account.

  2. Grant the Artifact Registry Reader (roles/artifactregistry.reader) IAM role to the Google service account. For more information about Artifact Registry roles and permissions, see Configure roles and permissions for Artifact Registry.

    • Grant project-wide permission if the same permissions apply to all repositories in the project.

      gcloud projects add-iam-policy-binding PROJECT_ID  \
            --role=roles/artifactregistry.reader \
            --member="serviceAccount:GSA_NAME@PROJECT_ID.iam.gserviceaccount.com"
      
    • Grant repository-specific permission when you want service accounts to have different levels of access for each repository in your project.

      gcloud artifacts repositories add-iam-policy-binding REPOSITORY \
         --location=LOCATION \
         --role=roles/artifactregistry.reader \
         --member="serviceAccount:GSA_NAME@PROJECT_ID.iam.gserviceaccount.com" \
         --project=PROJECT_ID
      
  3. Create an IAM policy binding between the Kubernetes service account and the Google service account by running the following command:

    gcloud iam service-accounts add-iam-policy-binding
      GSA_NAME@PROJECT_ID.iam.gserviceaccount.com \
        --role=roles/iam.workloadIdentityUser \
        --member="serviceAccount:FLEET_HOST_PROJECT_ID.svc.id.goog[config-management-system/KSA_NAME]"
        --project=PROJECT_ID
    

Replace the following:

  • PROJECT_ID: the organization's project ID.
  • FLEET_HOST_PROJECT_ID: if you're using GKE Workload Identity Federation for GKE, this is the same as PROJECT_ID. If you're using fleet Workload Identity Federation for GKE, this is the project ID of the fleet that your cluster is registered to.
  • GSA_NAME: the custom Google service account that you want to use to connect to Artifact Registry. The service account must have the Artifact Registry Reader (roles/artifactregistry.reader) IAM role.
  • KSA_NAME: the Kubernetes service account for the reconciler.
    • For root repositories, if the RootSync name is root-sync, use root-reconciler. Otherwise, use root-reconciler-ROOT_SYNC_NAME.
  • REPOSITORY: the ID of the repository.
  • LOCATION: the regional or multi-regional location of the repository.

Compute Engine default service account

If you store your Helm chart in Artifact Registry and your cluster is GKE with Workload Identity Federation for GKE disabled, you can use gcenode as your authentication type. Config Sync uses the Compute Engine default service account. You must grant your Compute Engine default service account reader access to Artifact Registry. You might need to grant the storage-ro access scope to grant read-only permission to pull images.

  1. Grant the Compute Engine service account read permission to Artifact Registry:

    gcloud projects add-iam-policy-binding PROJECT_ID \
        --member=serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com \
        --role=roles/artifactregistry.reader
    

    Replace PROJECT_ID with your organization's project ID, and replace PROJECT_NUMBER with your organization's project number.

Configure Config Sync

To configure syncing from the root repository, you need to create a RootSync object that syncs your root repository to the cluster. You can only create one root repository per cluster and the root repository can be either an unstructured repository or an hierarchical repository.

  1. If you're using the Config Sync admission webhook (the admission webhook is disabled by default) and are installing Config Sync in a private cluster, add a firewall rule to allow port 10250. The Config Sync admission webhook uses port 10250 for drift prevention.

  2. Wait for the RootSync and RepoSync CRDs to be available:

    until kubectl get customresourcedefinitions rootsyncs.configsync.gke.io reposyncs.configsync.gke.io; do date; sleep 1; echo ""; done
    
  3. Save one of the following manifests as root-sync.yaml. Use the manifest version that corresponds to the source type for your configs.

    Git

    # root-sync.yaml
    apiVersion: configsync.gke.io/v1beta1
    kind: RootSync
    metadata:
      name: ROOT_SYNC_NAME
      namespace: config-management-system
    spec:
      sourceType: git
      sourceFormat: ROOT_FORMAT
      git:
        repo: ROOT_REPOSITORY
        revision: ROOT_REVISION
        branch: ROOT_BRANCH
        dir: ROOT_DIRECTORY
        auth: ROOT_AUTH_TYPE
        gcpServiceAccountEmail: ROOT_EMAIL
        secretRef:
          name: ROOT_SECRET_NAME
        noSSLVerify: ROOT_NO_SSL_VERIFY
        caCertSecretRef:
          name: ROOT_CA_CERT_SECRET_NAME
    

    Replace the following:

    • ROOT_SYNC_NAME: add the name of your RootSync object.
    • ROOT_FORMAT: add unstructured to use an unstructured repository or add hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default value is hierarchy. We recommend that you add unstructured as this format lets you organize your configs in the way that is most convenient to you.
    • ROOT_REPOSITORY: add the URL of the Git repository to use as the root repository. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/anthos-config-management-samples uses the HTTPS protocol. This field is required.
    • ROOT_REVISION: add the Git revision (tag or hash) or branch to sync from. This field is optional and the default value is HEAD. When using a hash, it must be a full hash, and not an abbreviated form.
    • ROOT_BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master. We recommend using the revision field to specify a branch name for simplicity. If both the revision field and the branch field are specified, revision takes precedence over branch.
    • ROOT_DIRECTORY: add the path in the Git repository to the root directory that contains the configuration that you want to sync to. This field is optional and the default is the root directory (/) of the repository.
    • ROOT_AUTH_TYPE: add one of the following authentication types:

      • none: Use no authentication
      • ssh: Use a SSH key pair
      • cookiefile: Use a cookiefile
      • token: Use a token
      • gcpserviceaccount: Use a Google service account to access a Cloud Source Repositories.
      • gcenode: Use a Google service account to access a Cloud Source Repositories. Only select this option if Workload Identity Federation for GKE is not enabled in your cluster.

      For more information on these authentication types, see Granting Config Sync read-only access to Git.

      This field is required.

    • ROOT_EMAIL: If you added gcpserviceaccount as your ROOT_AUTH_TYPE, add your Google service account email address. For example, acm@PROJECT_ID.iam.gserviceaccount.com.

    • ROOT_SECRET_NAME: add the name of your Secret. If this field is set, you must add the Secret's public key to the Git provider. This field is optional.

    • ROOT_NO_SSL_VERIFY: To disable the SSL certificate verification, set this field to true. The default value is false.

    • ROOT_CA_CERT_SECRET_NAME: add the name of your Secret. If this field is set, your Git provider must be using a certificate issued by this certificate authority (CA). The Secret must contain the CA certificate under a key named cert. This field is optional.

      To learn more about how to configure the Secret object for the CA certificate, see Configure Certificate Authority

    For an explanation of the fields and a complete list of fields that you can add to the spec field, see RootSync fields.

    This manifest creates a RootSync object that uses Git as the source.

    OCI

    # root-sync.yaml
    apiVersion: configsync.gke.io/v1beta1
    kind: RootSync
    metadata:
      name: ROOT_SYNC_NAME
      namespace: config-management-system
    spec:
      sourceType: oci
      sourceFormat: ROOT_FORMAT
      oci:
        image: ROOT_IMAGE
        dir: ROOT_DIRECTORY
        auth: ROOT_AUTH_TYPE
        gcpServiceAccountEmail: ROOT_EMAIL
        caCertSecretRef:
          name: ROOT_CA_CERT_SECRET_NAME
    

    Replace the following:

    • ROOT_SYNC_NAME: add the name of your RootSync object.
    • ROOT_FORMAT: add unstructured to use an unstructured repository or add hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default value is hierarchy. We recommend that you add unstructured as this format lets you organize your configs in the way that is most convenient to you.
    • ROOT_IMAGE: the URL of the OCI image to use as the root repository, for example LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY_NAME/PACKAGE_NAME. By default, the image is pulled from the latest tag, but you can pull in images by TAG or DIGEST instead. Specify TAG or DIGEST in the PACKAGE_NAME:
      • To pull by TAG: LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY_NAME/PACKAGE_NAME:TAG
      • To pull by DIGEST: LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY_NAME/PACKAGE_NAME@sha256:DIGEST
    • ROOT_DIRECTORY: add the path in the repository to the root directory that contains the configuration that you want to sync to. This field is optional and the default is the root directory (/) of the repository.
    • ROOT_AUTH_TYPE: add one of the following authentication types:

      • none: Use no authentication
      • gcenode: Use the Compute Engine default service account to access an image in Artifact Registry. Only select this option if Workload Identity Federation for GKE is not enabled in your cluster.
      • gcpserviceaccount: Use a Google service account to access an image.

      This field is required.

    • ROOT_EMAIL: If you added gcpserviceaccount as your ROOT_AUTH_TYPE, add your Google service account email address. For example, acm@PROJECT_ID.iam.gserviceaccount.com.

    • ROOT_CA_CERT_SECRET_NAME: add the name of your Secret. If this field is set, your OCI provider must be using a certificate issued by this certificate authority (CA). The Secret must contain the CA certificate under a key named cert. This field is optional.

    To learn more about how to configure the Secret object for the CA certificate, see Configure Certificate Authority

    For an explanation of the fields and a complete list of fields that you can add to the spec field, see RootSync fields.

    This manifest creates a RootSync object that uses an OCI image as the source.

    Helm

    # root-sync.yaml
    apiVersion: configsync.gke.io/v1beta1
    kind: RootSync
    metadata:
      name: ROOT_SYNC_NAME
      namespace: config-management-system
    spec:
      sourceType: helm
      sourceFormat: ROOT_FORMAT
      helm:
        repo: ROOT_HELM_REPOSITORY
        chart: HELM_CHART_NAME
        version: HELM_CHART_VERSION
        releaseName: HELM_RELEASE_NAME
        namespace: HELM_RELEASE_NAMESPACE
        values:
          foo:
            bar: VALUE_1
          baz:
          - qux: VALUE_2
            xyz: VALUE_3
        includeCRDs: HELM_INCLUDE_CRDS
        auth: ROOT_AUTH_TYPE
          gcpServiceAccountEmail: ROOT_EMAIL
          secretRef:
            name: ROOT_SECRET_NAME
        caCertSecretRef:
          name: ROOT_CA_CERT_SECRET_NAME
    

    Replace the following:

    • ROOT_SYNC_NAME: add the name of your RootSync object.
    • ROOT_FORMAT: add unstructured to use an unstructured repository or add hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default value is hierarchy. We recommend that you add unstructured as this format lets you organize your configs in the way that is most convenient to you.
    • ROOT_HELM_REPOSITORY: the URL of the Helm repository to use as the root repository. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/anthos-config-management-samples uses the HTTPS protocol. This field is required.
    • HELM_CHART_NAME: add the name of your Helm chart. This field is required.
    • HELM_CHART_VERSION: the version of your chart. This field is optional. If no value is specified, the latest version is used.
    • HELM_RELEASE_NAME: the name of the Helm release. This field is optional.
    • HELM_RELEASE_NAMESPACE: the target namespace for a release. It only sets a namespace for resources that contain namespace: {{ .Release.Namespace }} in their templates. This field is optional. If no value is specified, the default namespace config-management-system is used.
    • HELM_INCLUDE_CRDS: set to true if you want the Helm template to also generate a CustomResourceDefinition. This field is optional. If no value is specified, the default is false and a CRD won't be generated.
    • VALUE: values to use instead of default values that accompany the Helm chart. Format this field the same way as helm chart's values.yaml file. This field is optional.
    • ROOT_AUTH_TYPE: add one of the following authentication types:

      • none: Use no authentication
      • token: Use a username and password to access a private Helm repository.
      • gcenode: Use the Compute Engine default service account to access an image in Artifact Registry. Only select this option if Workload Identity Federation for GKE is not enabled in your cluster.
      • gcpserviceaccount: Use a Google service account to access an image.

      This field is required.

    • ROOT_EMAIL: If you added gcpserviceaccount as your ROOT_AUTH_TYPE, add your Google service account email address. For example, acm@PROJECT_ID.iam.gserviceaccount.com.

    • ROOT_SECRET_NAME: add the name of your Secret if token is the ROOT_AUTH_TYPE. This field is optional.

    • ROOT_CA_CERT_SECRET_NAME: add the name of your Secret. If this field is set, your Helm provider must be using a certificate issued by this certificate authority (CA). The Secret must contain the CA certificate under a key named cert. This field is optional.

    To learn more about how to configure the Secret object for the CA certificate, see Configure Certificate Authority

    For an explanation of the fields and a complete list of fields that you can add to the spec field, see RootSync fields.

    This manifest creates a RootSync object that uses Helm as the source.

  4. Apply the changes:

    kubectl apply -f root-sync.yaml
    

Verify the sync status of the root repository

You can use the nomos status command to inspect the sync status of the root repository:

nomos status

You should see output similar to the following example:

my_managed_cluster-1
  --------------------
  <root>   git@github.com:foo-corp/acme/admin@main
  SYNCED   f52a11e4

Verify the RootSync installation

When you create a RootSync object, Config Sync creates a reconciler with the root-reconciler prefix. A reconciler is a Pod that is deployed as a Deployment. It syncs manifests from a Git repository to a cluster.

You can verify that the RootSync object is working correctly by checking the status of the root-reconciler Deployment:

kubectl get -n config-management-system deployment \
    -l configsync.gke.io/sync-name=ROOT_SYNC_NAME

Replace ROOT_SYNC_NAME with the name of RootSync.

You should see output similar to the following example:

NAME              READY   UP-TO-DATE   AVAILABLE   AGE
root-reconciler   1/1     1            1           3h42m

For further ways to explore the status of your RootSync object, see Monitoring RootSync and RepoSync objects.

After you have finished configuring your root repository, you can optionally choose to configure syncing from multiple repositories. These repositories are helpful if you want a repository that contains namespace-scoped configs synced to a particular namespace across clusters.

Upgrade Config Sync

The steps to upgrade Config Sync depend on the version you're upgrading to and from. Starting in version 1.20.0, Config Sync is provided as a standalone install without the ConfigManagement Operator. If you're upgrading from a version before 1.20.0 to version 1.20.0 or later, you must first uninstall the ConfigManagement Operator before upgrading.

If upgrading from an unsupported version, you should perform a step-by-step upgrade with increments to no more than three minor versions at a time. For example, if the current Config Sync version is 1.14.0, first upgrade to version 1.17.0, and then to version 1.20.0.

Uninstall the ConfigManagement Operator

If you're upgrading from a version before 1.20.0 to version 1.20.0 or later, you must first uninstall the ConfigManagement Operator before upgrading.

You can check if Config Management is installed on your cluster by running the following command:

kubectl get configmanagement

If the output is not empty, Config Management is installed on the cluster. To uninstall Config Management, complete the following steps to uninstall Config Management but leave Config Sync installed on the cluster. We recommend using the nomos CLI to uninstall the ConfigManagement Operator because it has a richer interface and more robust error handling. You should only use the shell script if you don't have access to the nomos CLI.

  1. Make sure the nomos CLI is on the latest version.

  2. Run the following command to update the cluster in your current kubectl context:

    nomos migrate --remove-configmanagement
    

shell script

Copy the following shell script to a file and then run it to update the cluster in your current kubectl context.

 #!/bin/bash

 set -euox pipefail

 hnc_enabled="$(kubectl get configmanagements.configmanagement.gke.io config-management -o=jsonpath="{.spec.hierarchyController.enabled}" --ignore-not-found)"

 if [[ "${hnc_enabled}" == "true" ]]; then
   echo "Hierarchy Controller is enabled on the ConfigManagement object. It must be disabled before migrating."
   echo "This can be done by unsetting the spec.hierarchyController field on ConfigManagement."
   exit 1
 fi

 kubectl delete deployment -n config-management-system config-management-operator --ignore-not-found --cascade=foreground

 if kubectl get configmanagement config-management &> /dev/null ; then
   kubectl patch configmanagement config-management --type="merge" -p '{"metadata":{"finalizers":[]}}'
   kubectl delete configmanagement config-management --cascade=orphan --ignore-not-found
 fi

 kubectl delete clusterrolebinding config-management-operator --ignore-not-found
 kubectl delete clusterrole config-management-operator --ignore-not-found
 kubectl delete serviceaccount -n config-management-system config-management-operator --ignore-not-found
 kubectl delete customresourcedefinition configmanagements.configmanagement.gke.io --ignore-not-found

Install the new Config Sync version

To upgrade Config Sync, complete the following steps for each enrolled cluster:

1.20.0 or later

  1. Download the Config Sync manifest and nomos commands for the new version.

  2. Extract the archive:

    tar -xzvf config-sync.tar.gz
  3. In the archive you extracted in the previous step, follow the instructions in the provided README to edit the kustomization.

  4. To update the Config Sync installation, apply the rendered manifest that you built following the README instructions:

    kubectl apply -f CONFIG_SYNC_MANIFEST
    

    Replace CONFIG_SYNC_MANIFEST with the name of the rendered manifest.

  5. Replace the nomos command on all clients with the new version. This change ensures that the nomos command can always get the status of all enrolled clusters and can validate configs for them.

1.19.2 or earlier

  1. Download the Config Sync manifest and nomos commands for the new version.

  2. Apply the Config Sync manifest:

    kubectl apply -f config-management-operator.yaml
    

    This command updates the ConfigManagement Operator image. Kubernetes retrieves the new version and restarts the Config Sync Pod using the new version. When Config Sync starts, it runs a reconcile loop that applies the set of manifests bundled in the new image. This updates and restarts each component Pod.

  3. Create a file named config-management.yaml and copy the following YAML file into it:

    # config-management.yaml
    apiVersion: configmanagement.gke.io/v1
    kind: ConfigManagement
    metadata:
      name: config-management
    spec:
      # The `enableMultiRepo` field is set to true to enable RootSync and RepoSync APIs.
      enableMultiRepo: true
      preventDrift: PREVENT_DRIFT
    

    Replace the following:

    • PREVENT_DRIFT: If set to true, enables the Config Sync admission webhook to prevent drifts by rejecting conflicting changes from being pushed to live clusters. The default setting is false. Config Sync always remediates drifts no matter the value of this field.
  4. Apply the changes:

    kubectl apply -f config-management.yaml
    
  5. Replace the nomos command on all clients with the new version. This change ensures that the nomos command can always get the status of all enrolled clusters and can validate configs for them.

Uninstall Config Sync

To uninstall Config Sync, complete the following steps:

  1. A central administrator should remove the root repo:

    1. If you've enabled the webhook and want to keep your resources, disable drift prevention for abandoned resources. If you haven't enabled the webhook, you don't have to take any additional steps to keep your resources.

    2. Delete the RootSync object by running the following command:

      kubectl delete -f root-sync.yaml
      
  2. Remove any repositories.

  3. With the manifests you used to install Config Sync, you can also uninstall Config Sync.

       kubectl delete -f CONFIG_SYNC_MANIFEST --ignore-not-found
    

What's next