Installing Config Sync

The Config Sync Operator is a controller that manages Config Sync in a Kubernetes cluster. Follow these steps to install and configure the Operator in each cluster you want to manage using Config Sync.

Resource requests

The following table lists Kubernetes resource requirements for Config Sync components. For more information, see Managing Resources for Containers in the Kubernetes documentation.

Deployment name CPU request (m) per replica Memory request (Mi) per replica Single or multiple repositories
git-importer 450 300 Single
monitor Default1 Default Single
resource-group-controller-manager 100+default 50+default Multi
admission-webhook2 100 20 Multi
otel-collector 200 400 Multi
reconciler-manager 200 120 Multi
reconciler (one per RootSync and RepoSync) 250+default 200+default Multi
hnc-controller-manager 100+default 150+default Hierarchy Controller
gke-hc-controller-manager 100+default 50+default Hierarchy Controller

1The default resource request uses a CPU request of 10 and a memory request of 10 Mi.

2The admission webhook has two replicas, so when calculating the total resource requests, you need to double the value.

Summary of total resource requests

The following table lists the combined amount of resource requests for Config Sync, depending on what features you are using.

Feature CPU Memory
Standard mode 460 m 310 Mi
Multi-repo mode 710 m + 260 m * (number of RootSync and RepoSync objects) 620 Mi + 210 Mi * (number of RootSync and RepoSync objects)
Hierarchy Controller 220 m 220 Mi

Before you begin

This section describes prerequisites you must meet before installing Config Sync on GKE.

Preparing your local environment

Before you install Operator, make sure you have prepared your local environment by completing the following tasks:

  • Install and initialize the Cloud SDK, which provides the gcloud, gsutil, kubectl, and nomos commands used in these instructions. If you use Cloud Shell, Cloud SDK comes pre-installed.

  • kubectl is not installed by default by Cloud SDK. To install kubectl, use the following command:

    gcloud components install kubectl
    
  • Authenticate to Google Cloud using the gcloud auth login command so that you can download components of Config Sync.

Preparing your clusters

Your clusters must run GKE version 1.14.x or later.

Preparing permissions

The Google Cloud user installing Config Sync needs Identity and Access Management (IAM) permissions to create new Roles in your cluster.

Enrolling a cluster

To enroll a cluster in Config Sync, complete the following steps:

  1. Deploy the Operator
  2. Grant the Operator read-only access to Git
  3. Configure the Operator

Deploying the Operator

After ensuring that you meet all the prerequisites, you can deploy the Operator by downloading and applying a YAML manifest.

  1. Download the latest version of the Operator CRD using the following command. To download a specific version instead, see Downloads.

    gsutil cp gs://config-management-release/released/latest/config-sync-operator.yaml config-sync-operator.yaml
    
  2. Apply the CRD:

    kubectl apply -f config-sync-operator.yaml

If this fails, see Troubleshooting.

Granting Operator read-only access to Git

Config Sync needs read-only access to your Git repository so it can read the configs committed to the repository and apply them to your clusters. If credentials are required, they are stored in the git-creds Secret on each enrolled cluster.

If your repository does not require authentication for read-only access, you can continue to configure Config Sync and set secretType to none. For example, if you can browse the repository using a web interface without logging in, or if you can use git clone to create a clone of the repository locally without providing credentials or using saved credentials, then you do not need to authenticate. In this case, you do not need to create a git-creds Secret.

However, most users need to create credentials because read access to their repository is restricted. Config Sync supports the following mechanisms for authentication:

  • SSH key pair
  • cookiefile
  • Token
  • Google Service Account (Google Source Repositories only)

The mechanism that you choose depends on what your repository supports. If all mechanisms are available, we recommend using an SSH key pair. GitHub, Cloud Source Repositories, and Bitbucket all support using an SSH key pair. If your organization hosts your repository and you don't know which authentication methods are supported, contact your administrator.

SSH key pair

An SSH key pair consists of two files, a public key and a private key. The public key typically has a .pub extension.

To use an SSH key pair, complete the following steps:

  1. Create an SSH key pair to allow Config Sync to authenticate to your Git repository. This step is necessary if you need to authenticate to the repository to clone it or read from it. Skip this step if a security administrator will provide you with a key pair. You can use a single key pair for all clusters, or a key pair per cluster, depending on your security and compliance requirements.

    The following command creates a 4096-bit RSA key. Lower values are not recommended:

    ssh-keygen -t rsa -b 4096 \
    -C "GIT_REPOSITORY_USERNAME" \
    -N '' \
    -f /path/to/KEYPAIR_FILENAME
    

    Replace the following:

    • GIT_REPOSITORY_USERNAME: add the username that you want Config Sync to use to authenticate to the repository.
    • /path/to/KEYPAIR_FILENAME: add a path to the key pair.

    If you are using a third-party Git repository host such as GitHub, or you want to use a service account with Cloud Source Repositories, we recommend that you use a separate account.

  2. Configure your repository to recognize the newly created public key. Refer to the documentation for your Git hosting provider. Instructions for some popular Git hosting providers are included for convenience:

  3. Add the private key to a new Secret in the cluster:

    kubectl create ns config-management-system && \
    kubectl create secret generic git-creds \
     --namespace=config-management-system \
     --from-file=ssh=/path/to/KEYPAIR_PRIVATE_KEY_FILENAME
    

    Replace /path/to/KEYPAIR_PRIVATE_KEY_FILENAME with the name of the private key (the one without the .pub suffix).

  4. Delete the private key from the local disk or otherwise protect it.

cookiefile

The process for acquiring a cookiefile depends on the configuration of your repository. For an example, see Generate static credentials in the Cloud Source Repositories documentation. The credentials are usually stored in the .gitcookies file in your home directory, or they might be provided to you by a security administrator.

To use a cookiefile, complete the following steps:

  1. After you create and obtain the cookiefile, add it to a new Secret in the cluster.

    kubectl create ns config-management-system && \
    kubectl create secret generic git-creds \
     --namespace=config-management-system \
     --from-file=cookie_file=/path/to/COOKIEFILE
    

    Replace /path/to/COOKIEFILE with the appropriate path and filename.

  2. Protect the contents of the cookiefile if you still need it locally. Otherwise, delete it.

Token

If your organization does not permit the use of SSH keys, you might prefer to use a token. With Config Sync, you can use GitHub's Personal Access Tokens (PAT) or Bitbucket's App Password as your token.

To create a Secret using your token, complete the following steps:

  1. Create a token using GitHub or Bitbucket.

    • GitHub: Create a PAT. Grant the token the repo scope, so it can read from private repositories. Because you bind a PAT to a GitHub account, we also recommend that you create a machine user and bind your PAT to the machine user.

    • Bitbucket: Create an App Password.

  2. After you create and obtain the token, add it to a new Secret in the cluster:

    kubectl create ns config-management-system && \
    kubectl create secret generic git-creds \
      --namespace="config-management-system" \
      --from-literal=username=USERNAME \
      --from-literal=token=TOKEN
    

    Replace the following:

    • USERNAME: add the username that you want to use.
    • TOKEN: add the token that you created in the previous step.
  3. Protect the token if you still need it locally. Otherwise, delete it.

Google Service Account

If your repository is in a Cloud Source Repositories, you can use secretType: gcenode to give Config Sync access to a repository in the same project as your managed cluster.

Prerequisites

Before you begin, ensure the following prerequisites are met:

  • Access scopes for the nodes in the cluster must include cloud-source-repos-ro. By default, the Compute Engine default service account PROJECT_ID-compute@developer.gserviceaccount.com has source.reader access to the repository for the same project, but if needed you can add the role with the following command:

    gcloud projects add-iam-policy-binding PROJECT_ID \
      --member serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com \
      --role roles/source.reader
    

    Replace the following:

    • PROJECT_ID: add your organization's project ID.
    • PROJECT_NUMBER: add your organization's project number.
  • Access scopes for the nodes in the cluster must include cloud-source-repos-ro. You can add this scope by including cloud-source-repos-ro in the --scopes list specified at cluster creation time, or by using the cloud-platform scope at cluster creation time:

    gcloud container clusters create example-cluster --scopes=cloud-platform
    

Using a Cloud Source Repositories as your SyncRepo

After these prerequisites are met, set spec.git.syncRepo for the URL of the wanted Cloud Source Repositories when you configure the operator.

To use a Cloud Source Repositories as your SyncRepo, complete the following steps:

  1. List all repositories:

    gcloud source repos list
    
  2. From the output, copy the URL from the repository you want to use:

    REPO_NAME  PROJECT_ID  URL
    my-repo    my-project  https://source.developers.google.com/p/my-project/r/my-repo-csr
    
  3. Add the URL as your syncRepo. For example:

    spec.git.syncRepo: https://source.developers.google.com/p/my-project/r/my-repo-csr
    

Using a Cloud Source Repositories with Workload Identity

If Workload Identity is enabled on your cluster, additional steps are required to use secretType: gcenode. After completing the preceding steps, and configuring Config Sync (which you do in the following section) create an IAM policy binding between the Kubernetes service account and the Google service account. The Kubernetes service account is not created until you configure Config Sync for the first time.

This binding allows the Config Sync Kubernetes service account to act as the Compute Engine default service account:

gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT_ID.svc.id.goog[config-management-system/importer]" \
  PROJECT_NUMBER-compute@developer.gserviceaccount.com

Replace the following: * PROJECT_ID: your organization's project ID * PROJECT_NUMBER: your organization's project number

After you've created the binding, add an annotation to the Config Sync Kubernetes service account using the email address of the Compute Engine default service account:

kubectl annotate serviceaccount -n config-management-system importer \
  iam.gke.io/gcp-service-account=PROJECT_NUMBER-compute@developer.gserviceaccount.com

Replace PROJECT_NUMBER with your organization's project number.

Configuring the Operator

To configure the behavior of Config Sync, create a configuration file for the ConfigManagement CustomResource, then apply it using the kubectl apply command.

  1. Create a file config-management.yaml and copy the following YAML file into it.

    # config-management.yaml
    
    apiVersion: configmanagement.gke.io/v1
    kind: ConfigManagement
    metadata:
      name: config-management
    spec:
      # clusterName is required and must be unique among all managed clusters
      clusterName: CLUSTER_NAME
      sourceFormat: FORMAT
      git:
        syncRepo: REPO
        syncBranch: BRANCH
        secretType: TYPE
        policyDir: "DIRECTORY"
    

    Replace the following:

    • CLUSTER_NAME: add the name of the registered cluster that you want to apply this configuration to.
    • FORMAT: add unstructured to use an unstructured repository or add hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default value is hierarchy. We recommend that you add unstructured as this format lets you organize your configs in the way that is most convenient to you.
    • REPO: add the URL of the Git repository to use as the source of truth. This field is required.
    • BRANCH: add the branch of the repository to sync from. The default is the main (master) branch.
    • TYPE: add one of the following SecretTypes:

      • none
      • ssh
      • cookiefile
      • token
      • gcenode
    • DIRECTORY: the path in the Git repository to the root directory that contains the configuration that you want to sync to. The default is the root directory of the repository.

    For a complete list of fields that you can add to the spec field, see ConfigManagement fields.

  2. Apply the configuration with the kubectl apply command:

    kubectl apply -f config-management.yaml
    

    You might need to create separate configuration files for each cluster, or each type of cluster. You can specify the cluster using the --context option:

    kubectl apply -f config-management1.yaml --context=CLUSTER_NAME
    

    Replace CLUSTER_NAME with the name of the cluster you want to use.

Verifying the installation

You can use the nomos status command to check if the Operator is installed successfully. A valid installation with no problems has a status of PENDING or SYNCED. An invalid or incomplete installation has a status of NOT INSTALLED OR NOT CONFIGURED. The output also includes any reported errors.

When the Operator is deployed successfully, it runs in a Pod whose name begins with config-management-operator, in the kube-system namespace. The Pod may take a few moments to initialize. Verify that the Pod is running:

kubectl -n kube-system get pods | grep config-management

If the Pod is running, the command's response is similar (but not identical) to the following:

config-management-operator-6f988f5fdd-4r7tr 1/1 Running 0 26s

You can also verify that the config-management-system namespace exists:

kubectl get ns | grep 'config-management-system'

The command's output is similar to the following:

config-management-system Active 1m

If the commands don't return output similar to that shown here, view the logs to see what went wrong:

kubectl -n kube-system logs -l k8s-app=config-management-operator

You can also use kubectl get events to check if Config Sync has created any events.

kubectl get events -n kube-system

It is possible to have an invalid configuration that isn't detected right away, such as a missing or invalid git-creds Secret. For troubleshooting steps, see Valid but incorrect ConfigManagement object in the Troubleshooting section of this topic.

Upgrading Config Sync versions

This section provides general instructions for upgrading to the current version. Before upgrading, check the release notes for any specific instructions.

Run these commands for each enrolled cluster.

  1. Download the Operator manifest and nomos commands for the new version.

  2. Apply the Operator manifest:

    kubectl apply -f config-sync-operator.yaml
    

    This command updates the Operator image. Kubernetes retrieves the new version and restarts the Operator Pod using the new version. When the Operator starts, it runs a reconcile loop that applies the set of manifests bundled in the new image. This updates and restarts each component Pod.

  3. Replace the nomos or nomos.exe command on all clients with the new version. This ensures that the nomos command can always get the status of all enrolled clusters and can validate configs for them.

Uninstalling the Operator from a cluster

Follow these instructions to uninstall the Operator from a cluster. You must follow these steps for each cluster that you no longer want to manage by using Config Sync.

  1. Delete the ConfigManagement object from the cluster:

    kubectl delete configmanagement --all

    The following things happen:

    • Any ClusterRoles and ClusterRoleBindings created in the cluster by Config Sync are deleted from the cluster.
    • Any admission controller configurations installed by Config Sync are deleted.
    • The contents of the config-management-system namespace are deleted, with the exception of the git-creds Secret. Config Sync cannot function without the config-management-system namespace. Any CustomResourceDefinitions created or modified by Config Sync are removed from the clusters where they were created or modified. The CustomResourceDefinitions (CRDs) required to run the Operator still exist, because from the point of view of Kubernetes, they were added by the user who installed the Operator. Information about removing these is covered in the next step.
  2. At this point, the Operator still exists in your cluster, but does nothing.

    If you are using GKE and decide you no longer want to use Config Sync at all, you can uninstall the Operator by following these steps:

    1. Verify that the config-management-system namespace is empty after deleting the ConfigManagement object in the previous step. Wait until the kubectl -n config-management-system get all command returns No resources found.

    2. Delete the config-management-system namespace:

      kubectl delete ns config-management-system
      
    3. Delete the ConfigManagement CustomResourceDefinition:

      kubectl delete crd configmanagements.configmanagement.gke.io
      
    4. Delete all Config Sync objects from the kube-system namespace:

      kubectl -n kube-system delete all -l k8s-app=config-management-operator
      

Troubleshooting

The following sections help you troubleshoot your Config Sync installation.

Insufficient CPU

The output of kubectl get events might include an event with the type FailedScheduling. The event looks like the following example:

LAST SEEN   TYPE      REASON              OBJECT                                             MESSAGE
9s          Warning   FailedScheduling    pod/config-management-operator-74594dc8f6    0/1 nodes are available: 1 Insufficient cpu.

To fix this error, choose one of the following options:

  • Add a node to an existing GKE node pool.
  • Create a node pool with larger nodes.

Error: attempt to grant extra privileges

If you receive the following error:

Error from server (Forbidden): error when creating "config-sync-operator.yaml": clusterroles.rbac.authorization.k8s.io "config-management-operator" is forbidden: attempt to grant extra privileges: [...] ruleResolutionErrors=[]

when you try to apply the config-management-operator.yaml file, then you have fewer permissions than are required for the installation. Review Preparing permissions to ensure you have the required permissions.

Valid but incorrect ConfigManagement object

If installation fails due to a problem with the ConfigManagement object that is not due to a YAML or JSON syntax error, the ConfigManagement object might be instantiated in the cluster, but might not work correctly. In this situation, you can use the nomos status command to check for errors in the ConfigManagement object.

A valid installation with no problems has a status of PENDING or SYNCED.

An invalid installation has a status of NOT CONFIGURED and lists one of the following errors:

  • missing git-creds Secret
  • missing required syncRepo field
  • git-creds Secret is missing the key specified by secretType

To fix the problem, correct the configuration error. Depending on the type of error, you might need to re-apply the ConfigManagement manifest to the cluster.

If the problem is that you forgot to create the git-creds Secret, Config Sync detects the Secret as soon as you create it, and you do not need to re-apply the configuration.

What's next