Syncing from multiple repositories

Enabling multi-repo mode in Config Sync lets you sync configurations from multiple repositories to the same set of clusters. To use multi-repo mode, you need to create a single root repository and can also optionally create namespace repositories. A root repository can sync cluster-scoped and namespace-scoped configs and is typically managed by an administrator. Namespace repositories contain namespace-scoped configs synced to a particular namespace across clusters. To learn more about these types of repositories, see the Repositories section in the Config Sync overview.

Each repository has a single Git reference (a repository branch, commit or tag, and directory tuple) that you can manage separately. This configuration decouples the config deployment lifecycle for different teams. It also provides you with more autonomy in choosing where you want to place the repository and how to structure it.

The following diagram shows you an overview of how teams might sync their clusters to a single root repository (managed by an admin) and multiple namespace repositories (managed by application operators):

A central administrator controlling multiple configs and app operators
controlling their own namespace configs.

In the preceding diagram, a central administrator manages the centralized infrastructure for the organization and enforces policies on the cluster and on all namespaces in the organization. The application operators, who are responsible for managing live deployments, apply configurations to the applications in the namespaces that they work on.

Before you begin

Complete the following tasks before enabling multi-repo mode:

  • Create, or have access to, Git repositories.
  • Create, or have access to, a Google Cloud project.
  • Create, or have access to, a Google Kubernetes Engine (GKE) cluster that meets the following requirements:
    • Is a standard cluster. Config Sync is not supported on Autopilot clusters.
    • Uses a GKE version between 1.18 and 1.20.
    • Has a node pool that uses a machine type with at least 4 vCPU.
  • Use a Config Sync version of 1.5.3 or later.
  • If you're using version 1.7 and are installing Config Sync in a private cluster, add a firewall rule to allow port 8676. The Config Sync admission webhook uses port 8676 for drift prevention. Starting from version 1.8.0, the port is switched to 10250 and is open by default in private clusters.

Limitations

Consider the following limitations when you use multi-repo mode:

Configuring syncing from the root repository

To configure syncing from the root repository, you need to enable multi-repo mode in your ConfigManagement object and create a RootSync object that syncs your root repository to the cluster. You can only create one root repository per cluster and the root repository can be either an unstructured repository or an hierarchical repository.

The following sections describe three different methods for enabling multi-repo mode:

  1. If you haven't previously installed Config Sync, create a new root Git repository configuration.
  2. If you have previously installed Config Sync, move the root Git repository configuration to RootSync.
  3. If you have previously installed Config Sync and you need to use the spec.git fields, keep the root Git repository configuration in the ConfigManagement YAML file.

1. New configuration

If you have not previously installed Config Sync, complete the following tasks to enable multi-repo mode:

  1. Complete the following sections in Manually installing Config Sync:

    1. Before you begin
    2. Deploying the operator
    3. Granting Operator read-only access to Git
  2. If you're using version 1.7 and are installing Config Sync in a private cluster, add a firewall rule to allow port 8676. The Config Sync admission webhook uses port 8676 for drift prevention. Starting from version 1.8.0, the port is switched to 10250 and is open by default in private clusters.

  3. Create a file named config-management.yaml and copy the following YAML file into it:

    # config-management.yaml
    apiVersion: configmanagement.gke.io/v1
    kind: ConfigManagement
    metadata:
      name: config-management
    spec:
      enableMultiRepo: true
    
  4. Apply the changes:

    kubectl apply -f config-management.yaml
    
  5. Create a RootSync object:

    # root-sync.yaml
    # If you are using a Config Sync version earlier than 1.7.0,
    # use: apiVersion: configsync.gke.io/v1alpha1
    apiVersion: configsync.gke.io/v1beta1
    kind: RootSync
    metadata:
      name: root-sync
      namespace: config-management-system
    spec:
      sourceFormat: ROOT_FORMAT
      git:
        repo: ROOT_REPOSITORY
        revision: ROOT_REVISION
        branch: ROOT_BRANCH
        dir: "ROOT_DIRECTORY"
        auth: ROOT_AUTH_TYPE
        gcpServiceAccountEmail: ROOT_EMAIL
        secretRef:
          name: ROOT_SECRET_NAME
    

    Replace the following:

    • ROOT_FORMAT: add unstructured to use an unstructured repository or add hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default value is hierarchy. We recommend that you add unstructured as this format lets you organize your configs in the way that is most convenient to you.
    • ROOT_REPOSITORY: add the URL of the Git repository to use as the root repository. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/anthos-config-management-samples uses the HTTPS protocol. If you don't enter a protocol, the URL is treated as an HTTPS URL. This field is required.
    • ROOT_REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
    • ROOT_BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
    • ROOT_DIRECTORY: add the path in the Git repository to the root directory that contains the configuration that you want to sync to. This field is optional and the default is the root directory (/) of the repository.
    • ROOT_AUTH_TYPE: add one of the following authentication types:

      • none: Use no authentication
      • ssh: Use a SSH key pair
      • cookiefile: Use a cookiefile
      • token: Use a token
      • gcpserviceaccount: Use a Google service account to access a Cloud Source Repositories.
      • gcenode: Use a Google service account to access a Cloud Source Repositories. Only select this option if Workload Identity is not enabled in your cluster.

      For more information on these authentication types, see Granting Config Sync read-only access to Git.

      This field is required.

    • ROOT_EMAIL: If you added gcpserviceaccount as your AUTH_TYPE, add your Google service account email address. For example, acm@PROJECT_ID.iam.gserviceaccount.com.

    • ROOT_SECRET_NAME: add the name of your Secret. If you are using a Secret, you must add the Secret's public key to the Git provider. This field is optional.

  6. Apply the changes:

    kubectl apply -f root-sync.yaml
    

2. Moving configuration

If you have previously installed Config Sync, you can move the Git repository configurations from your existing ConfigManagement object to a RootSync object.

If configure the root repository by moving the configuration, complete the following tasks:

  1. If you're using version 1.7 and installed Config Sync in a private cluster, add a firewall rule to allow port 8676. The Config Sync admission webhook uses port 8676 for drift prevention. Starting from version 1.8.0, the port is switched to 10250 and is open by default in private clusters.
  2. Open your ConfigManagement object.
  3. Make a copy of the values in the spec.git fields. You use these values when you create the RootSync object.
  4. Remove all of the spec.git fields (including git:) from the ConfigManagement object.
  5. In the ConfigManagement object, set the spec.enableMultiRepo field to true:

    # config-management.yaml
    apiVersion: configmanagement.gke.io/v1
    kind: ConfigManagement
    metadata:
      name: config-management
    spec:
      enableMultiRepo: true
    
  6. Apply the changes:

    kubectl apply -f config-management.yaml
    
  7. Using the values you copied from the ConfigManagement object, create the RootSync object. For example:

    # root-sync.yaml
    # If you are using a Config Sync version earlier than 1.7.0,
    # use: apiVersion: configsync.gke.io/v1alpha1
    apiVersion: configsync.gke.io/v1beta1
    kind: RootSync
    metadata:
      name: root-sync
      namespace: config-management-system
    spec:
      sourceFormat: ROOT_FORMAT
      git:
        repo: ROOT_REPOSITORY
        revision: ROOT_REVISION
        branch: ROOT_BRANCH
        dir: "ROOT_DIRECTORY"
        auth: ROOT_AUTH_TYPE
        gcpServiceAccountEmail: ROOT_EMAIL
        secretRef:
          name: ROOT_SECRET_NAME
    

    Replace the following:

    • ROOT_FORMAT: add unstructured to use an unstructured repository or add hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default value is hierarchy. We recommend that you add unstructured as this format lets you organize your configs in the way that is most convenient to you.
    • ROOT_REPOSITORY: add the URL of the Git repository to use as the root repository. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/anthos-config-management-samples uses the HTTPS protocol. If you don't enter a protocol, the URL is treated as an HTTPS URL. This field is required.
    • ROOT_REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
    • ROOT_BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
    • ROOT_DIRECTORY: add the path in the Git repository to the root directory that contains the configuration that you want to sync to. This field is optional and the default is the root directory (/) of the repository.
    • ROOT_AUTH_TYPE: add one of the following authentication types:

      • none: Use no authentication
      • ssh: Use a SSH key pair
      • cookiefile: Use a cookiefile
      • token: Use a token
      • gcpserviceaccount: Use a Google service account to access a Cloud Source Repositories.
      • gcenode: Use a Google service account to access a Cloud Source Repositories. Only select this option if Workload Identity is not enabled in your cluster.

      For more information on these authentication types, see Granting Config Sync read-only access to Git.

      This field is required.

    • ROOT_EMAIL: If you added gcpserviceaccount as your AUTH_TYPE, add your Google service account email address. For example, acm@PROJECT_ID.iam.gserviceaccount.com.

    • ROOT_SECRET_NAME: add the name of your Secret. If you are using a Secret, you must add the Secret's public key to the Git provider. This field is optional.

  8. Apply the changes:

    kubectl apply -f root-sync.yaml
    

3. Keeping configuration

If you have previously installed Config Sync, you can keep an existing root Git repository configured in your ConfigManagement object. However, this method is not recommended because it will be deprecated.

To configure the root repository by keeping the root repository in the ConfigManagement object, complete the following tasks:

  1. If you're using version 1.7 and installed Config Sync in a private cluster, add a firewall rule to allow port 8676. The Config Sync admission webhook uses port 8676 for drift prevention. Starting from version 1.8.0, the port is switched to 10250 and is open by default in private clusters.

  2. In the ConfigManagement object, set the spec.enableMultiRepo and the spec.enableLegacyFields fields to true, set the spec.sourceFormat to unstructured to use an unstructured repository or hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default value is hierarchy. We recommend that you add unstructured as this format lets you organize your configs in the way that is most convenient to you.

    # config-management.yaml
    apiVersion: configmanagement.gke.io/v1
    kind: ConfigManagement
    metadata:
      name: config-management
    spec:
      enableMultiRepo: true
      enableLegacyFields: true
      sourceFormat: FORMAT
      git:
        syncRepo: REPO
        syncBranch: BRANCH
        secretType: TYPE
        policyDir: "DIRECTORY"
    # ...other fields...
    
  3. Apply the changes:

    kubectl apply -f config-management.yaml
    

Applying config-management.yaml with enableMultiRepo: true and enableLegacyFields: true set automatically generates the RootSync object on the cluster. Do not change the RootSync configuration that is generated.

To add optional namespace repositories, proceed to the Configuring syncing from namespace repositories section. If you only want to sync from a single repository and continue to use your current workflow, you can skip that section.

Verifying the sync status of the root repository

You can use the nomos status command to inspect the sync status of the root repository:

nomos status

You should see output similar to the following example:

my_managed_cluster-1
  --------------------
  <root>   git@github.com:foo-corp/acme/admin@main
  SYNCED   f52a11e4

Verifying the RootSync installation

When you create a RootSync object, Config Sync creates a reconciler called root-reconciler. A reconciler is a Pod that is deployed as a Deployment. It syncs manifests from a Git repository to a cluster.

You can verify that the RootSync object is working correctly by checking the status of the root-reconciler Deployment:

kubectl get -n config-management-system deployment/root-reconciler

You should see output similar to the following example:

NAME              READY   UP-TO-DATE   AVAILABLE   AGE
root-reconciler   1/1     1            1           3h42m

For further ways to explore the status of your RootSync object, see the Exploring the RootSync and RepoSync objects section.

Configuring syncing from namespace repositories

After you have set up your root repository, you can choose to set up namespace repositories that non-administrative users can control. Namespace repositories must be unstructured.

To configure namespace repositories, you need to assign permissions and create a RepoSync object that syncs your namespace repository to the cluster. You can choose between two methods for configuring namespace repositories:

  • Control namespace repositories in the root repository. This method centralizes all configuration of namespace repositories in the root repository, allowing a central administrator complete control of the setup.

  • Control namespace repositories with the Kubernetes API. This method delegates control of the namespace repository to the namespace owners.

  • Optional: Enable the admission webhook in the namespace repository. If you want to enable the Config Sync admission webhook as an extra step to perform drift prevention then follow the steps outlined in enabling admission webhook for namespace repository.

Controlling namespace repositories in the root repository

In this method, the central administrator manages the setup of namespace repositories directly from the root repository. Because Config Sync manages the namespace repository objects, this method prevents any local changes to the namespace repository definitions.

To use this method, complete the following tasks:

  1. In the root repository, declare a namespace configuration:

    # ROOT_REPO/namespaces/NAMESPACE/namespace.yaml
    apiVersion: v1
    kind: Namespace
    metadata:
      name: NAMESPACE
    

    Replace NAMESPACE with a name for your namespace.

  2. In the root repository, create a RepoSync object in the same namespace:

    # ROOT_REPO/namespaces/NAMESPACE/repo-sync.yaml
     # If you are using a Config Sync version earlier than 1.8.0,
     # use: apiVersion: configsync.gke.io/v1alpha1
    apiVersion: configsync.gke.io/v1beta1
    kind: RepoSync
    metadata:
      name: repo-sync
      namespace: NAMESPACE
    spec:
      # Since this is for a namespace repository, the format should be unstructured
      sourceFormat: unstructured
      git:
       repo: NAMESPACE_REPOSITORY
       revision: NAMESPACE_REVISION
       branch: NAMESPACE_BRANCH
       dir: "NAMESPACE_DIRECTORY"
       auth: NAMESPACE_AUTH_TYPE
       gcpServiceAccountEmail: NAMESPACE_EMAIL
       secretRef:
         name: NAMESPACE_SECRET_NAME
    

    Replace the following:

    • NAMESPACE: add the name of your namespace.
    • NAMESPACE_REPOSITORY: add the URL of the Git repository to use as the namespace repository. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/anthos-config-management-samples uses the HTTPS protocol. If you don't enter a protocol, the URL is treated as an HTTPS URL. This field is required.
    • NAMESPACE_REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
    • NAMESPACE_BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
    • NAMESPACE_DIRECTORY: add the path in the Git repository to the root directory that contains the configuration that you want to sync to. This field is optional and the default is the root directory (/) of the repository.
    • NAMESPACE_AUTH_TYPE: add one of the following authentication types:

      • none: Use no authentication
      • ssh: Use a SSH key pair
      • cookiefile: Use a cookiefile
      • token: Use a token
      • gcpserviceaccount: Use a Google service account to access a Cloud Source Repositories.
      • gcenode: Use a Google service account to access a Cloud Source Repositories. Only select this option if Workload Identity is not enabled in your cluster.

        For more information on these authentication types, see Granting Config Sync read-only access to Git.

      This field is required.

    • NAMESPACE_EMAIL: If you added gcpserviceaccount as your AUTH_TYPE, add your Google service account email address. For example, acm@PROJECT_ID.iam.gserviceaccount.com.

    • NAMESPACE_SECRET_NAME: add the name you intend to give your Secret. This field is optional.

    There can be at most one RepoSync object per namespace. To enforce this restriction, the object name must be repo-sync. All configs contained in the directory referenced by the RepoSync must also be in the same namespace as the RepoSync object.

  3. In the root repository, declare a RoleBinding configuration that grants the ns-reconciler-NAMESPACE service account permission to manage objects in the namespace. Config Sync automatically creates the ns-reconciler-NAMESPACE service account when the RepoSync config is synced to the cluster.

    To declare the RoleBinding, create the following object:

    # ROOT_REPO/namespaces/NAMESPACE/sync-rolebinding.yaml
     kind: RoleBinding
     apiVersion: rbac.authorization.k8s.io/v1
     metadata:
       name: syncs-repo
       namespace: NAMESPACE
     subjects:
     - kind: ServiceAccount
       name: ns-reconciler-NAMESPACE
       namespace: config-management-system
     roleRef:
       kind: ClusterRole
       name: RECONCILER_ROLE
       apiGroup: rbac.authorization.k8s.io
    

    Replace the following:

    • NAMESPACE: add the name of your namespace.
    • RECONCILER_ROLE: as the central administrator, you can set the RECONCILER_ROLE to enforce what kinds of configuration can be synced from the namespace repository. You can choose one of the following roles:

      • A default ClusterRole:

        • admin
        • edit

        To learn more, see User-facing roles.

      • A user-defined ClusterRole or Role declared in the root repository. This role allows for fine grained permissions.

  4. Commit the changes to the root repository:

     git add .
     git commit -m 'Setting up new namespace repository.'
     git push
    
  5. If required, create a Secret based on your preferred authentication method. If you used none as your authentication type, you can skip this step.

    The Secret must meet following requirements:

    • Create the Secret in the same namespace as the RepoSync.
    • The Secret's name must match the spec.git.secretRef name you defined in repo-sync.yaml.
    • You must add the Secret's public key to the Git provider.
  6. To verify the configuration, use kubectl get on one of the objects in the namespace repository. For example:

    kubectl get rolebindings -n NAMESPACE
    

Controlling namespace repositories with the Kubernetes API

In this method, the central administrator only declares the namespace in the root repository and delegates declaration of the RepoSync object to the application operator.

Central administrator tasks

The central administrator completes the following tasks:

  1. In the root repository, declare a namespace configuration:

    # ROOT_REPO/namespaces/NAMESPACE/namespace.yaml
     apiVersion: v1
     kind: Namespace
     metadata:
       name: NAMESPACE
    

    Replace NAMESPACE with a name for your namespace.

  2. In the root repository, declare a RoleBinding to give the application operators permissions. Use RBAC escalation prevention to ensure that the application operator cannot later apply a role binding with permissions not granted by this role binding.

    To declare the RoleBinding, create the following manifest:

    # ROOT_REPO/namespaces/NAMESPACE/operator-rolebinding.yaml
     kind: RoleBinding
     # Add RBAC escalation prevention
     apiVersion: rbac.authorization.k8s.io/v1
     metadata:
       name: operator
       namespace: NAMESPACE
     subjects:
     - kind: User
       name: USERNAME
       apiGroup: rbac.authorization.k8s.io
     roleRef:
       kind: ClusterRole
       name: OPERATOR_ROLE
       apiGroup: rbac.authorization.k8s.io
    

    Replace the following:

    • NAMESPACE: add the namespace you created in the root repository.
    • USERNAME: add the username of the application operator.
    • OPERATOR_ROLE: as the central administrator, you can set OPERATOR_ROLE to enforce what kinds of configurations can be synced from the namespace repository. You can choose one of the following roles:

      • A default ClusterRole:

        • admin
        • edit

        To learn more, see User-facing roles.

      • A user-defined ClusterRole or Role declared in the root repository. This role allows for fine grained permissions.

  3. Commit the changes to the root repository:

     git add .
     git commit -m 'Setting up new namespace repository.'
     git push
    

Application operator tasks

The application operator completes the following tasks:

  1. Declare a RoleBinding configuration that grants the auto-provisioned ns-reconciler-NAMESPACE service account permission to manage objects in the namespace. Config Sync automatically creates the ns-reconciler-NAMESPACE service account when the RepoSync config is synced to the cluster.

    To declare the RoleBinding, create the following manifest:

    # sync-rolebinding.yaml
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: syncs-repo
      namespace: NAMESPACE
    subjects:
    - kind: ServiceAccount
      name: ns-reconciler-NAMESPACE
      namespace: config-management-system
    roleRef:
      kind: ClusterRole
      name: RECONCILER_ROLE
      apiGroup: rbac.authorization.k8s.io
    

    Replace the following:

    • NAMESPACE: add the namespace you created in the root repository.
    • RECONCILER_ROLE: as the application operator you can set RECONCILER_ROLE to enforce what kinds of configuration can be synced from the namespace repository. You can only further restrict the set of permissions the central administrator has granted you. As a result, this role cannot be more permissive than the OPERATOR_ROLE that the central administrator declared in the previous section.
  2. Apply the RoleBinding configuration:

    kubectl apply -f sync-rolebinding.yaml
    
  3. If required, create a Secret based on your preferred authentication method. If you used none as your authentication type, you can skip this step.

    The Secret must meet the following requirements:

    • Create the Secret in the same namespace as the RepoSync.
    • The Secret's name must match the spec.git.secretRef name you defined in root-sync.yaml.
    • You must add the Secret's public key to the Git provider.
  4. Declare a RepoSync configuration:

    # repo-sync.yaml
    # If you are using a Config Sync version earlier than 1.8.0,
    # use: apiVersion: configsync.gke.io/v1alpha1
    apiVersion: configsync.gke.io/v1beta1
    kind: RepoSync
    metadata:
      name: repo-sync
      namespace: NAMESPACE
    spec:
      # Since this is for a namespace repository, the format should be unstructured
      sourceFormat: unstructured
      git:
       repo: NAMESPACE_REPOSITORY
       revision: NAMESPACE_REVISION
       branch: NAMESPACE_BRANCH
       dir: "NAMESPACE_DIRECTORY"
       auth: NAMESPACE_AUTH_TYPE
       gcpServiceAccountEmail: NAMESPACE_EMAIL
       secretRef:
         name: NAMESPACE_SECRET_NAME
    

    Replace the following:

    • NAMESPACE_REPOSITORY: add the URL of the Git repository to use as the namespace repository. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/anthos-config-management-samples uses the HTTPS protocol. If you don't enter a protocol, the URL is treated as an HTTPS URL. This field is required.
    • NAMESPACE_REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
    • NAMESPACE_BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
    • NAMESPACE_DIRECTORY: add the path in the Git repository to the root directory that contains the configuration that you want to sync to. This field is optional and the default is the root directory (/) of the repository.
    • NAMESPACE_AUTH_TYPE: add one of the following authentication types:

      • none: Use no authentication
      • ssh: Use a SSH key pair
      • cookiefile: Use a cookiefile
      • token: Use a token
      • gcpserviceaccount: Use a Google service account to access a Cloud Source Repositories.
      • gcenode: Use a Google service account to access a Cloud Source Repositories. Only select this option if Workload Identity is not enabled in your cluster.

        For more information on these authentication types, see Granting Config Sync read-only access to Git.

      This field is required.

    • NAMESPACE_EMAIL: If you added gcpserviceaccount as your AUTH_TYPE, add your Google service account email address. For example, acm@PROJECT_ID.iam.gserviceaccount.com.

    • NAMESPACE_SECRET_NAME: add the name you intend to give your Secret. This field is optional.

    There can be at most one RepoSync object per namespace. To enforce this, the object's name must be repo-sync. All configs contained in the directory referenced by the RepoSync must also be in the same namespace as the RepoSync object.

  5. Apply the RepoSync configuration:

    kubectl apply -f repo-sync.yaml
    
  6. To verify the configuration, use kubectl get on one of the objects in the namespace repository. For example:

    kubectl get rolebindings -n NAMESPACE
    

Verifying the sync status of the namespace repository

You can use the nomos status command to inspect the sync status of the namespace repository:

nomos status

You should see output similar to the following example:

my_managed_cluster-1
  --------------------
  <root>   git@github.com:foo-corp/acme/admin@main
  SYNCED   f52a11e4
  --------------------
  bookstore  git@github.com:foo-corp/acme/bookstore@v1
  SYNCED     34d1a8c8

In this example output, the namespace repository is configured for namespace named bookstore.

Verifying the RepoSync installation

When you create a RepoSync object, Config Sync creates a reconciler called ns-reconciler-NAMESPACE, where NAMESPACE is the namespace you created your RepoSync object in.

You can verify that the RepoSync object is working correctly by checking the status of the ns-reconciler-NAMESPACE Deployment:

kubectl get -n config-management-system deployment/ns-reconciler-NAMESPACE

Replace NAMESPACE with the namespace that you created your namespace repository in.

For further ways to explore the status of your RepoSync object, see the Exploring the RootSync and RepoSync objects section.

Prevent config drift

Config Sync reduces the risk of "shadow ops" through drift detection. When unvetted changes are pushed to live clusters, Config Sync detects and remediates any drift from the source of truth in Git. Furthermore, starting from version 1.7.0, Config Sync multi-repo mode provides an admission webhook that rejects conflicting changes from being pushed to live clusters. This provides better user feedback when manual changes are attempted, rather than allowing it and silently remediating the change.

In addition, this admission webhook prevents Config Sync metadata and annotations from being manually modified, which would otherwise cause errors in the configuration for admission webhook or reconcilers.

Enabling admission webhook in namespace repositories

The admission webhook that rejects conflicting changes from being pushed to live clusters is installed only when Config Sync is configured in the multi-repo mode. By default, the webhook only protects the root repository. The namespace repositories are not protected by the webhook as the Config Sync reconciler for each namespace repo does not have permission to read or update the ValidatingWebhookConfiguration objects at the cluster level.

This lack of permission results in an error seen in the logs for the namespace reconcilers similar to the following:

  Failed to update admission webhook: KNV2013: applying changes to admission webhook: Insufficient permission. To fix, make sure the reconciler has sufficient permissions.: validatingwebhookconfigurations.admissionregistration.k8s.io "admission-webhook.configsync.gke.io" is forbidden: User "system:serviceaccount:config-management-system:ns-reconciler-NAMESPACE" cannot update resource "validatingwebhookconfigurations" in API group "admissionregistration.k8s.io" at the cluster scope

This error can be ignored if the webhook protection is not needed for the namespace repo. However, if you want to add this protection, then complete the following steps to grant this permission to the reconciler for each namespace repository.

To use this method, complete the following tasks:

  1. In the root repository, declare a new ClusterRole configuration that is used to grant permission to the Config Sync admission webhook. This ClusterRole only needs to be defined once per cluster:

    # ROOT_REPO/cluster-roles/webhook-role.yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: admission-webhook-role
    rules:
    - apiGroups: ["admissionregistration.k8s.io"]
      resources: ["validatingwebhookconfigurations"]
      resourceNames: ["admission-webhook.configsync.gke.io"]
      verbs: ["get", "update"]
    
  2. For each namespace repository where the admission webhook permission needs to be granted, declare a ClusterRoleBinding configuration to grant access to the admission webhook:

    # ROOT_REPO/NAMESPACE/sync-webhook-rolebinding.yaml
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: syncs-webhook
    subjects:
    - kind: ServiceAccount
      name: ns-reconciler-NAMESPACE
      namespace: config-management-system
    roleRef:
      kind: ClusterRole
      name: admission-webhook-role
      apiGroup: rbac.authorization.k8s.io
    

    Replace NAMESPACE with a name for your namespace.

  3. Commit the changes to the root repository:

    git add .
    git commit -m 'Providing namespace repository the permission to update the admission webhook.'
    git push
    
    
  4. To verify, use kubectl to make sure the ClusterRole and ClusterRoleBinding have been created:

    kubectl get clusterrole admission-webhook-role
    kubectl get clusterrolebindings syncs-webhook
    

Stopping and resuming syncing

To learn how to stop syncing from multiple repositories, see stopping and resuming syncing from multiple repositories.

Removing the root and namespace repositories

Before removing the root repository, you should remove all the namespace repositories first. Removing the root repository before the namespace repositories might cause the namespace admission controller to fail to clean up namespaces. If this happens, the hanging namespaces would have the following condition:

    message: 'Failed to delete all resource types, 1 remaining: admission webhook
      "v1.admission-webhook.configsync.gke.io" denied the request: requester is not
      authorized to delete managed resources'
    reason: ContentDeletionFailed
    status: "True"
    type: NamespaceDeletionContentFailure

Removing a namespace repository

Select the Root repository method or Kubernetes API method tab to view the relevant instructions.

Root repository method

If you used the Control namespace repositories in the root repository method, a central administrator can remove a namespace repository by configuring the RepoSync object to sync from an empty Git repository.

Kubernetes API method

If you used the Control namespace repositories with the Kubernetes API method, application operators can follow the following two steps to remove a namespace repository:

  1. Remove a namespace repository by configuring the RepoSync object to sync from an empty Git repository.

  2. Delete the RepoSync object by running the following command:

    kubectl delete -f repo-sync.yaml
    

Removing the root repository

A central administrator can remove the root repository by following the following two steps:

  1. Remove the root repository by configuring the RootSync object to sync from an empty Git repository.

  2. Delete the RootSync object by running the following command:

     kubectl delete -f root-sync.yaml
    

Ignoring object mutations

In Config Sync versions 1.7.0 and later, if you don't want Config Sync to maintain the state of the object an the cluster after it exists, you can add the client.lifecycle.config.k8s.io/mutation: ignore annotation to the object that you want Config Sync to ignore mutations in. This action is useful when you want Config Sync to create an object and then ignore any conflicting changes to this object in the cluster.

The following example shows you how to add the annotation to an object:

metadata:
  annotations:
    client.lifecycle.config.k8s.io/mutation: ignore 

You cannot manually modify this annotation on managed objects in the cluster.

Conflict resolution

When you are working with two repositories, conflicts might arise when the same object (matching groups, kinds, names, and namespaces) is declared both in the root and a namespace repository. When this conflict happens, only the declaration in the root repository is applied to the cluster.

If this conflict happens, the RootSync won't report any problems since the root repository takes precedence. The RepoSync reports error KNV 1060 in its status.

Since the root repository always takes precedence, you can resolve the conflict by updating the root repository to match the namespace repository, or by deleting the conflicting object in the namespace repository.

Exploring the RootSync and RepoSync objects

The following sections show you different ways that you can explore your RootSync and RepoSync objects.

Viewing logs

For further information on potential errors, you can view the logs for the RootSync and RepoSync objects.

To view the logs for the RootSync reconciler, run the following command:

kubectl logs -n config-management-system deployment/root-reconciler CONTAINER_NAME

Replace CONTAINER_NAME with either git-sync, reconciler, or otel-agent. The git-sync container pulls configs from a Git repository to a local directory that the reconciler container can read from. The reconciler container applies those configs to the cluster. The otel-agent container is an OpenTelemetry agent for retrieving and exporting Config Sync metrics.

To view the logs of the RepoSync reconciler, run the following command:

kubectl logs -n config-management-system deployment/ns-reconciler-NAMESPACE CONTAINER_NAME

Replace the following:

  • NAMESPACE: the namespace that you created your namespace repository in.
  • CONTAINER_NAME: either git-sync, reconciler, or otel-agent.

Viewing synced commits

You can check which commit is synced to the cluster.

To view details for the RootSync, run the following command:

kubectl get rootsync root-sync -n config-management-system

To view details for the RepoSync, run the following command:

kubectl get reposync repo-sync -n config-management-system

You should see output similar to the following example:

NAME        SOURCECOMMIT                               SYNCCOMMIT
root-sync   66882815f0ef5517df27e864fb1315e97756ab72   66882815f0ef5517df27e864fb1315e97756ab72

The value in the SOURCECOMMIT column is the commit from the Git repository that should be synced to the cluster. The value in the SYNCCOMMIT column is the commit that is currently deployed to the cluster. If the two values in the SOURCECOMMIT and SYNCCOMMIT columns are the same, the expected commit has been deployed to the cluster.

Viewing object details

To view details of the RootSync and RepoSync objects, use kubectl describe. This command can provide you with further information about potential errors.

To view details for the RootSync object, run the following command:

kubectl describe rootsync root-sync -n config-management-system

To view details for the RepoSync object, run the following command:

kubectl describe reposync repo-sync -n config-management-system

Viewing if a resource is ready

To learn if the resources synced to the cluster are ready, view the reconciliation status. For example, this command can show you if a synced Deployment is ready to serve traffic.

For a Git repository synced to the cluster, the reconciliation status of all resources are aggregated in a resource called ResourceGroup. For each RootSync or RepoSync object, a ResourceGroup is generated to capture the set of resources applied to the cluster and aggregate their statuses.

To view the reconciliation status for the RootSync object, run the following command:

kubectl get resourcegroup.kpt.dev root-sync -n config-management-system -o yaml

To view the reconciliation status for the RepoSync object, run the following command:

kubectl get resourcegroup.kpt.dev repo-sync -n NAMESPACE -o yaml

In the output, you see all of the ResourceGroup resource statuses. For example, the following output shows that a Deployment named nginx-deployment is ready:

resourceStatuses:
- group: apps
  kind: Deployment
  name: nginx-deployment
  namespace: default
  status: Current

What's next