Installing Config Sync - Anthos

Config Sync lets cluster operators manage Kubernetes resources by using files, called configs, stored in Git repositories.

This page shows you how to enable and configure Config Sync as part of Anthos Config Management. Follow these steps to install and configure Anthos Config Management in each cluster that you want to manage.

Before you begin

Before you install Config Sync, make sure that you have prepared your environment, clusters, and roles and enabled Anthos Config Management.

Preparing your local environment

Prepare your local environment by completing the following tasks:

  1. Create, or have access to, a Git repository that can contain the configs that Config Sync syncs to.
  2. Enable the Anthos API for your project:

    Console

    In the Google Cloud Console, go to the Anthos API page.

    Go to the Anthos API page

    • Click Enable.

    gcloud

    Run the following command:

    gcloud services enable anthos.googleapis.com
    
  3. Install and initialize the Cloud SDK, which provides the gcloud, gsutil, kubectl, and nomos commands used in these instructions. If you use Cloud Shell, the Cloud SDK comes pre-installed.

  4. kubectl is not installed by default by the Cloud SDK. To install kubectl, use the following command:

    gcloud components install kubectl
    
  5. Use the gcloud auth login command to authenticate to Google Cloud so that you can download components of Anthos Config Management.

Preparing your cluster

Prepare your cluster by completing the following tasks:

  1. Ensure that your cluster is on an Anthos supported platform and version.

  2. We recommend that you enable Workload Identity on your Anthos clusters. Workload Identity is the recommended way to access Google Cloud services from applications running within Google Kubernetes Engine (GKE) due to its improved security properties and manageability.

  3. If you are using a GKE cluster, keep the following points in mind:

  • If you want to use Google Service Account when granting Config Sync access to your Git repository, then access scopes for the nodes in the cluster must include read only scope for Cloud Source Repositories. You can add this scope by including cloud-source-repos-ro in the -- scopes list specified at cluster creation time, or by using the cloud- platform scope at cluster creation time. For example:
    gcloud container clusters create CLUSTER_NAME --scopes=cloud-platform
    
    If you don't want to use Workload Identity, you can use the Compute Engine default service account. By default, the Compute Engine default service account PROJECT_ID-compute@developer.gserviceaccount.com has source.reader access to the repository for the same project, but if needed, you can add the role with the following command:
    gcloud projects add-iam-policy-binding PROJECT_ID 
    --member serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com
    --role roles/source.reader
    Replace the following:
    • PROJECT_ID: your organization's project ID
    • PROJECT_NUMBER: your organization's project number

    You cannot modify access scopes after you create a node pool. However, you can create a new node pool with the proper access scope while using the same cluster. The default gke-default scope does not include cloud-source-repos-ro.

  1. Register your cluster to an Anthos fleet by using Connect. Your project's fleet provides a unified way to view and manage your clusters and their workloads as part of Anthos, including clusters outside Google Cloud. Anthos charges apply only to your registered clusters.

Preparing permissions

Depending on the installation method that you use, you need to grant the user that is installing Config Sync different permissions.

Console

The Google Cloud user installing Config Sync needs the GKE Hub Admin role to administer your cluster through the Hub API. The Hub Admin role provides full access to a fleet and its related resources.

To grant the Hub Admin Role, complete the following steps:

  1. In the Cloud Console, go to the IAM page.

    Go to the IAM page

  2. Click Add.

  3. In the New members field, enter an email address. You can add individuals, service accounts, or Google Groups as members.

  4. In the Select a role drop-down list, search for and select GKE Hub Admin.

  5. Click Save.

gcloud

The Google Cloud user installing Config Sync needs the GKE Hub Admin (roles/gkehub.admin) role to administer your cluster through the Hub API.The Hub Admin role provides full access to a fleet and its related resources.

To grant the Hub Admin Role, complete the following steps:

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member=MEMBER \
  --role=roles/gkehub.admin

Replace the following:

  • MEMBER: An identifier for the member, which usually has the following form: member-type:id For example, user:my-user@example.com. For a full list of the values that member can have, see the Policy Binding reference.

  • PROJECT_ID: your project ID

kubectl

If you are using GKE, the Google Cloud user installing Config Sync needs IAM permissions to create new roles in your cluster:

gcloud container clusters get-credentials CLUSTER_NAME

kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole cluster-admin --user USER_ACCOUNT

Replace the following:

  • CLUSTER_NAME: your cluster name
  • USER_ACCOUNT: your Google Cloud account's email address

Depending on how you configured the gcloud command-line tool on your local system, you might need to add the --project and --zone fields.

Enabling Anthos Config Management

If you are using Anthos Config Management for the first time, enable the feature by completing the following steps.

Console

  1. In the Google Cloud Console, go to the Anthos Features page.

    Go to the Anthos Features page

  2. In the Config Management row, click Enable.

  3. In the confirmation window, click Enable Config Management.

gcloud

Run the following command:

 gcloud alpha container hub config-management enable

kubectl

The Operator is a controller that manages Config Sync in a Kubernetes cluster.

After ensuring that you meet all the prerequisites, you can deploy the Operator by downloading and applying a YAML manifest.

  1. Download the latest version of the Operator CustomResourceDefinition (CRD) by using the following command. To download a specific version instead, see Downloads.

    gsutil cp gs://config-management-release/released/latest/config-management-operator.yaml config-management-operator.yaml
    
  2. Apply the CRD:

    kubectl apply -f config-management-operator.yaml

If this command fails, see Troubleshooting.

Installing Config Sync

In the following sections, you grant Config Sync access to your repository. After you have granted access, you configure the installation.

Granting Config Sync read-only access to Git

Config Sync needs read-only access to your Git repository so that it can read the configs committed to the repository and apply them to your clusters.

If your repository does not require authentication for read-only access, you can continue to configure Config Sync and use none as your authentication type. For example, if you can browse the repository using a web interface without logging in, or if you can use git clone to create a clone of the repository locally without providing credentials or using saved credentials, then you do not need to authenticate. In this case, you do not need to create a Secret.

However, most users need to create credentials because read access to their repository is restricted. If credentials are required, they are stored in the git-creds Secret on each enrolled cluster (unless you are using a Google service account).

Config Sync supports the following mechanisms for authentication:

  • SSH key pair
  • cookiefile
  • Token
  • Google Service Account (Google Source Repositories only)

The mechanism that you choose depends on what your repository supports. If all mechanisms are available, we recommend using an SSH key pair. GitHub, Cloud Source Repositories, and Bitbucket all support using an SSH key pair. If your organization hosts your repository and you don't know which authentication methods are supported, contact your administrator.

SSH key pair

An SSH key pair consists of two files, a public key and a private key. The public key typically has a .pub extension.

To use an SSH key pair, complete the following steps:

  1. Create an SSH key pair to allow Config Sync to authenticate to your Git repository. This step is necessary if you need to authenticate to the repository to clone it or read from it. Skip this step if a security administrator provides you with a key pair. You can use a single key pair for all clusters, or a key pair per cluster, depending on your security and compliance requirements.

    The following command creates a 4096-bit RSA key. Lower values are not recommended:

    ssh-keygen -t rsa -b 4096 \
    -C "GIT_REPOSITORY_USERNAME" \
    -N '' \
    -f /path/to/KEYPAIR_FILENAME
    

    Replace the following:

    • GIT_REPOSITORY_USERNAME: the username that you want Config Sync to use to authenticate to the repository
    • /path/to/KEYPAIR_FILENAME: a path to the key pair

    If you are using a third-party Git repository host such as GitHub, or you want to use a service account with Cloud Source Repositories, we recommend that you use a separate account.

  2. Configure your repository to recognize the newly created public key. Refer to the documentation for your Git hosting provider. Instructions for some popular Git hosting providers are included for convenience:

  3. Add the private key to a new Secret in the cluster:

    kubectl create ns config-management-system && \
    kubectl create secret generic SECRET_NAME \
     --namespace=config-management-system \
     --from-file=ssh=/path/to/KEYPAIR_PRIVATE_KEY_FILENAME
    

    Replace the following:

    • SECRET_NAME: add the name of the Secret. Use git-creds as the name if you plan to configure Config Sync through the Cloud Console or using the gcloud tool in the next step.
    • /path/to/KEYPAIR_PRIVATE_KEY_FILENAME: add the name of the private key (the one without the .pub suffix).
  4. Delete the private key from the local disk or otherwise protect it.

cookiefile

The process for acquiring a cookiefile depends on the configuration of your repository. For an example, see Generate static credentials in the Cloud Source Repositories documentation. The credentials are usually stored in the .gitcookies file in your home directory, or they might be provided to you by a security administrator.

To use a cookiefile, complete the following steps:

  1. After you create and obtain the cookiefile, add it to a new Secret in the cluster.

    kubectl create ns config-management-system && \
    kubectl create secret generic SECRET_NAME \
     --namespace=config-management-system \
     --from-file=cookie_file=/path/to/COOKIEFILE
    

    Replace the following:

    • SECRET_NAME: add the name of the Secret. Use git-creds as the name if you plan to configure Config Sync through the Cloud Console or using the gcloud tool in the next step.
    • /path/to/COOKIEFILE: add the appropriate path and filename.
  2. Protect the contents of the cookiefile if you still need it locally. Otherwise, delete it.

Token

If your organization does not permit the use of SSH keys, you might prefer to use a token. With Config Sync, you can use GitHub's personal access tokens (PATs) or Bitbucket's app password as your token.

To create a Secret using your token, complete the following steps:

  1. Create a token using GitHub or Bitbucket:

    • GitHub: Create a PAT. Grant the token the repo scope so that it can read from private repositories. Because you bind a PAT to a GitHub account, we also recommend that you create a machine user and bind your PAT to the machine user.

    • Bitbucket: Create an app password.

  2. After you create and obtain the token, add it to a new Secret in the cluster:

    kubectl create ns config-management-system && \
    kubectl create secret generic SECRET_NAME \
      --namespace="config-management-system" \
      --from-literal=username=USERNAME \
      --from-literal=token=TOKEN
    

    Replace the following:

    • USERNAME: the username that you want to use.
    • SECRET_NAME: add the name of the Secret. Use git-creds as the name if you plan to configure Config Sync through the Cloud Console or using the gcloud tool in the next step.
    • TOKEN: the token that you created in the previous step.
  3. Protect the token if you still need it locally. Otherwise, delete it.

Google Service Account

If your repository is in Cloud Source Repositories, you can give Config Sync access to a repository in the same project as your managed cluster.

To use a repository in Cloud Source Repositories as your SyncRepo, complete the following steps:

  1. List all repositories:

    gcloud source repos list
    
  2. From the output, copy the URL from the repository that you want to use. For example:

    REPO_NAME  PROJECT_ID  URL
    my-repo    my-project  https://source.developers.google.com/p/my-project/r/my-repo-csr
    
  3. When you configure Config Sync in the next section, add the value that you copied in the URL field (if you are using the Google Cloud Console) or add the value that you copied in the repo field of your ConfigManagement object (if you are using the gcloud command-line tool or kubectl commands).

Configuring Config Sync

The Google Cloud Console guides you through the installation process and automates many of the steps. You can also use the gcloud command-line tool or kubectl commands to complete the installation.

If you are using a GKE cluster, we recommend that you use the Cloud Console to configure Config Sync. If you are using Anthos clusters on VMware, you should use kubectl commands to install Config Sync because private registries aren't supported in the Cloud Console or the gcloud tool.

To configure Config Sync, complete the following steps.

Console

  1. In the Cloud Console, go to the Anthos Config Management page.

    Go to Anthos Config Management

  2. Select your registered clusters and click Configure.

  3. In the Git Repository Authentication for ACM section, select one of the following options:

    • None
    • SSH
    • Cookiefile
    • Token
    • Google Cloud Repository
  4. If you have not done so already, follow the instructions in the Google Cloud Console to grant Config Sync read-only access to Git.

  5. Click Continue.

  6. In the ACM settings for your clusters section, complete the following:

    1. In the Version field, select version 1.7.0 or later for Anthos Config Management, which enables syncing from multiple repositories by default.
    2. Select the Enable Config Sync checkbox.
      1. In the URL field, add the URL of the Git repository to use as the source of truth. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/csp-config-management/ uses the HTTPS protocol. If you don't enter a protocol, the URL is treated as an HTTPS URL. This field is required.
      2. In the Branch field, add the branch of the repository to sync from. The default is the main (master) branch.
      3. In the Tag/Commit field, add the Git revision (tag or hash) to check out. The default is HEAD.
      4. In the Policy directory field, add the path within the repository to the top of the policy hierarchy to sync. The default is the root directory of the repository.
      5. In the Sync wait field, add the period in seconds between consecutive syncs. The default is 15 seconds.
      6. In the Git proxy field, enter the URL for the HTTPS proxy to be used when communicating with the Git repository. This is an optional field, and if it's left blank, no proxy is used.
      7. In the Source format field, choose either hierarchy (default) or unstructured (recommended). We recommend that you select unstructured as this format lets you organize your configs in the way that is most convenient to you.
  7. Click Done. You are taken back to the Anthos Config Management page. After a few minutes, you should see Synced in the status column next to the clusters that you configured. If you see Error, click on the word Error for more information.

gcloud

  1. Create a file named config-management.yaml and copy the following YAML file into it:

    # config-management.yaml
    
    apiVersion: configmanagement.gke.io/v1
    kind: ConfigManagement
    metadata:
      name: config-management
    spec:
      # Enable multi-repo mode to use additional features
      enableMultiRepo: true
      # Use spec.git for generating a RootSync resource in multi-repo mode
      enableLegacyFields: true
      sourceFormat: FORMAT
      git:
        syncRepo: REPO
        syncBranch: BRANCH
        secretType: TYPE
        policyDir: "DIRECTORY"
    

    Replace the following:

    • FORMAT: add unstructured to use an unstructured repository or add hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default value is hierarchy. We recommend that you add unstructured as this format lets you organize your configs in the way that is most convenient to you.
    • REPO: add the URL of the Git repository to use as the source of truth. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/csp-config-management/ uses the HTTPS protocol. If you don't enter a protocol, the URL is treated as an HTTPS URL. This field is required.
    • BRANCH: the branch of the repository to sync from. The default is the main (master) branch.
    • TYPE: one of the following SecretTypes:

      • none
      • ssh
      • cookiefile
      • token
      • gcenode
    • DIRECTORY: the path in the Git repository to the root directory that contains the configuration that you want to sync to. The default is the root directory of the repository.

      For a complete list of fields that you can add to the spec field, see ConfigManagement fields.

  2. Apply the config-management.yaml file:

     gcloud alpha container hub config-management apply \
         --membership=MEMBERSHIP_NAME \
         --config=CONFIG_YAML_PATH \
         --project=PROJECT_ID
    

    Replace the following:

    • MEMBERSHIP_NAME: the membership name that you chose when you registered your cluster.
    • CONFIG_YAML_PATH: the path to your config-management.yaml file
    • PROJECT_ID: your project ID

kubectl

To configure the behavior of Anthos Config Management, create a configuration file for the ConfigManagement CustomResource, and then apply it by using the kubectl apply command:

  1. Create a file named config-management.yaml and copy the following YAML file into it.

    # config-management.yaml
    
    apiVersion: configmanagement.gke.io/v1
    kind: ConfigManagement
    metadata:
      name: config-management
    spec:
      # clusterName is required and must be unique among all managed clusters
      clusterName: CLUSTER_NAME
      # Enable multi-repo mode to use additional features
      enableMultiRepo: true
    

    Replace CLUSTER_NAME with the name of the registered cluster that you want to apply this configuration to.

  2. Apply the configuration with the kubectl apply command:

    kubectl apply -f config-management.yaml
    

    You might need to create separate configuration files for each cluster or each type of cluster. You can specify the cluster using the --context option:

    kubectl apply -f config-management1.yaml --context=CLUSTER_NAME
    

    Replace CLUSTER_NAME with the name of the cluster that you want to use.

  3. Create a file root-sync.yaml and copy the following YAML file into it.

    To configure syncing from the root repository, you need to enable multi-repo mode in your ConfigManagement object and create a RootSync object that syncs your root repository to the cluster. For more information, see syncing from multiple repositories.

    # root-sync.yaml
    # If you are using a Config Sync version earlier than 1.7,
    # use: apiVersion: configsync.gke.io/v1alpha1
    apiVersion: configsync.gke.io/v1beta1
    kind: RootSync
    metadata:
      name: root-sync
      namespace: config-management-system
    spec:
      sourceFormat: FORMAT
      git:
        repo: REPO
        branch: BRANCH
        dir: "DIRECTORY"
        auth: TYPE
        secretRef:
          name: SECRET_NAME
    

    Replace the following:

    • FORMAT: add unstructured to use an unstructured repository or add hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default value is hierarchy. We recommend that you add unstructured as this format lets you organize your configs in the way that is most convenient to you.
    • REPO: add the URL of the Git repository to use as the source of truth. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/csp-config-management/ uses the HTTPS protocol. If you don't enter a protocol, the URL is treated as an HTTPS URL. This field is required.
    • BRANCH: add the branch of the repository to sync from. The default is the main (master) branch.
    • DIRECTORY: the path in the Git repository to the root directory that contains the configuration that you want to sync to. The default is the root directory of the repository.
    • TYPE: add one of the followingSecretTypes`:

      • none
      • ssh
      • cookiefile
      • token
      • gcenode
    • SECRET_NAME: add the name of your Secret. If you are using a Secret, you must add the Secret's public key to the Git provider. This field is optional.

  4. Apply the RootSync to your cluster:

    kubectl apply -f root-sync.yaml

    For a complete list of fields that you can add to the spec field, see ConfigManagement fields.

Using Cloud Source Repositories with Workload Identity

If Workload Identity is enabled on your cluster and you used a Google service account as your authentication type, you need to complete additional steps.

After completing the preceding steps, create an IAM policy binding between the Kubernetes service account and the Google service account. The Kubernetes service account is not created until you configure Config Sync for the first time.

This binding lets the Config Sync Kubernetes service account act as the Compute Engine default service account:

gcloud iam service-accounts add-iam-policy-binding \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:PROJECT_ID.svc.id.goog[config-management-system/importer]" \
    PROJECT_NUMBER-compute@developer.gserviceaccount.com

Replace the following:

  • PROJECT_ID: your organization's project ID
  • PROJECT_NUMBER: your organization's project number

After you've created the binding, add an annotation to the Config Sync Kubernetes service account using the email address of the Compute Engine default service account:

kubectl annotate serviceaccount -n config-management-system importer \
    iam.gke.io/gcp-service-account=PROJECT_NUMBER-compute@developer.gserviceaccount.com

Replace PROJECT_NUMBER with your organization's project number.

Verifying the installation

After you have installed and configured Config Sync, you can verify that the installation completed successfully.

Console

  1. In the Cloud Console, go to the Anthos Config Management page.

    Go to Anthos Config Management

  2. View the Status column. A successful installation has a status of Synced.

For a detailed view of your cluster status:

  • Select the cluster that you want to explore. The Cluster details page appears. On this page, you can see details for your cluster and details for your Config Sync installation.

gcloud

Run the following command:

gcloud alpha container hub config-management status \
    --project=PROJECT_ID

Replace PROJECT_ID with your project's ID.

A successful installation has a status of SYNCED. If you see an error after running the preceding command, make sure that you created the git-creds Secret. If you have created the Secret, try rerunning the following command:

gcloud alpha container hub config-management apply

kubectl

To check whether Config Sync is installed successfully, you can use the nomos status command. A valid installation with no problems has a status of PENDING or SYNCED. An invalid or incomplete installation has a status of NOT INSTALLED or NOT CONFIGURED. The output also includes any reported errors.

When Config Sync is deployed successfully, it runs in a Pod whose name begins with config-management-operator in the kube-system namespace. The Pod might take a few moments to initialize.

Verify that the Pod is running:

kubectl -n kube-system get pods | grep config-management

If the Pod is running, the command's response is similar to the following example:

config-management-operator-6f988f5fdd-4r7tr 1/1 Running 0 26s

You can also verify that the config-management-system namespace exists:

kubectl get ns | grep 'config-management-system'

The command's output is similar to the following example:

config-management-system Active 1m

If the commands don't return output similar to the preceding example, view the logs to see what went wrong:

kubectl -n kube-system logs -l k8s-app=config-management-operator

You can also use kubectl get events to check whether Config Sync has created any events.

kubectl get events -n kube-system

It's possible to have an invalid configuration that isn't detected right away, such as a missing or invalid git-creds Secret. For troubleshooting steps, see Valid but incorrect ConfigManagement object in the Troubleshooting section of this page.

What's next