Syncing from multiple repositories

Enabling multi-repo mode in Config Sync lets you sync configurations from multiple repositories to the same set of clusters. To use multi-repo mode, you need to create a single root repository and can also optionally create namespace repositories. A root repository can sync cluster-scoped and namespace-scoped configs and is typically managed by an administrator. Namespace repositories contain namespace-scoped configs synced to a particular namespace across clusters. To learn more about these types of repositories, see the Repositories section in the Config Sync overview.

Each repository has a single Git reference (a repository branch, commit or tag, and directory tuple) that you can manage separately. This configuration decouples the config deployment lifecycle for different teams. It also provides you with more autonomy in choosing where you want to place the repository and how to structure it.

The following diagram shows you an overview of how teams might use a root repository and namespace repositories:

A central administrator controlling multiple configs and app operators
controlling their own namespace configs.

In this diagram, a central administrator manages the centralized infrastructure for the organization and enforces policies on the cluster and on all namespaces in the organization.

The application operators, who are responsible for managing live deployments, apply configurations to the applications in the namespaces that they work on.

Before you begin

Complete the following tasks before enabling multi-repo mode:

  • Create, or have access to, Git repositories.
  • Create, or have access to, a Google Cloud project.
  • Create, or have access to, a Google Kubernetes Engine (GKE) cluster that meets the following requirements:
    • Is a standard cluster. Config Sync is not supported on Autopilot clusters.
    • Uses a GKE version between 1.18 and 1.20.
    • Has a node pool that uses a machine type with at least 4 vCPU.
  • Use a Config Sync version of 1.5.3 or later.
  • If you're using version 1.7.0 or later and are installing Config Sync in a private cluster, add a firewall rule to allow port 8676. The Config Sync admission webhook uses port 8676 for drift prevention.

Limitations

Consider the following limitations when you use multi-repo mode:

Configuring syncing from the root repository

To configure syncing from the root repository, you need to enable multi-repo mode in your ConfigManagement object and create a RootSync object that syncs your root repository to the cluster. You can only create one root repository per cluster and the root repository can be either an unstructured repository or an hierarchical repository.

The following sections describe three different methods for enabling multi-repo mode:

  1. If you haven't previously installed Config Sync, create a new root Git repository configuration.
  2. If you have previously installed Config Sync, move the root Git repository configuration to RootSync.
  3. If you have previously installed Config Sync and you need to use the spec.git fields, keep the root Git repository configuration in the ConfigManagement YAML file.

1. New configuration

If you have not previously installed Config Sync, complete the following tasks to enable multi-repo mode:

  1. Complete the following sections in Installing Config Sync:

    1. Before you begin
    2. Deploying the operator
    3. Granting Operator read-only access to Git
  2. If you're using version 1.7.0 or later and are installing Config Sync in a private cluster, add a firewall rule to allow port 8676. The Config Sync admission webhook uses port 8676 for drift prevention.

  3. Create a file named config-management.yaml and copy the following YAML file into it:

    # config-management.yaml
    apiVersion: configmanagement.gke.io/v1
    kind: ConfigManagement
    metadata:
      name: config-management
    spec:
      enableMultiRepo: true
    
  4. Apply the changes:

    kubectl apply -f config-management.yaml
    
  5. Create a RootSync object:

    # root-sync.yaml
    # If you are using a Config Sync version earlier than 1.7.0,
    # use: apiVersion: configsync.gke.io/v1alpha1
    apiVersion: configsync.gke.io/v1beta1
    kind: RootSync
    metadata:
      name: root-sync
      namespace: config-management-system
    spec:
      sourceFormat: FORMAT
      git:
        repo: REPOSITORY
        revision: REVISION
        branch: BRANCH
        dir: "DIRECTORY"
        auth: AUTH_TYPE
        secretRef:
          name: SECRET_NAME
    

    Replace the following:

    • FORMAT: add unstructured to use an unstructured repository or add hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default value is hierarchy. We recommend that you add unstructured as this format lets you organize your configs in the way that is most convenient to you.
    • REPOSITORY: add the URL of the Git repository to use as the root repository. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/csp-config-management/ uses the HTTPS protocol. If you don't enter a protocol, the URL is treated as an HTTPS URL. This field is required.
    • REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
    • BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
    • DIRECTORY: add the path in the Git repository to the root directory that contains the configuration that you want to sync to. This field is optional and the default is the root directory (/) of the repository.
    • AUTH_TYPE: add one of the following authentication types:

      • none
      • ssh
      • cookiefile
      • token
      • gcenode: If Workload Identity is enabled in your cluster, you cannot use gcenode. This field is required.
    • SECRET_NAME: add the name of your Secret. If you are using a Secret, you must add the Secret's public key to the Git provider. This field is optional.

  6. Apply the changes:

    kubectl apply -f root-sync.yaml
    

2. Moving configuration

If you have previously installed Config Sync, you can move the Git repository configurations from your existing ConfigManagement object to a RootSync object.

If configure the root repository by moving the configuration, complete the following tasks:

  1. If you're using version 1.7.0 or later and installed Config Sync in a private cluster, add a firewall rule to allow port 8676. The Config Sync admission webhook uses port 8676 for drift prevention.
  2. Open your ConfigManagement object.
  3. Make a copy of the values in the spec.git fields. You use these values when you create the RootSync object.
  4. Remove all of the spec.git fields (including git:) from the ConfigManagement object.
  5. In the ConfigManagement object, set the spec.enableMultiRepo field to true:

    # config-management.yaml
    apiVersion: configmanagement.gke.io/v1
    kind: ConfigManagement
    metadata:
      name: config-management
    spec:
      enableMultiRepo: true
    
  6. Apply the changes:

    kubectl apply -f config-management.yaml
    
  7. Using the values you copied from the ConfigManagement object, create the RootSync object. For example:

    # root-sync.yaml
    # If you are using a Config Sync version earlier than 1.7.0,
    # use: apiVersion: configsync.gke.io/v1alpha1
    apiVersion: configsync.gke.io/v1beta1
    kind: RootSync
    metadata:
      name: root-sync
      namespace: config-management-system
    spec:
      sourceFormat: FORMAT
      git:
        repo: REPOSITORY
        revision: REVISION
        branch: BRANCH
        dir: "DIRECTORY"
        auth: AUTH_TYPE
        secretRef:
          name: SECRET_NAME
    

    Replace the following:

    • FORMAT: add unstructured to use an unstructured repository or add hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default value is hierarchy. We recommend that you add unstructured as this format lets you organize your configs in the way that is most convenient to you.
    • REPOSITORY: add the URL of the Git repository to use as the root repository. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/csp-config-management/ uses the HTTPS protocol. If you don't enter a protocol, the URL is treated as an HTTPS URL. This field is required.
    • REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
    • BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
    • DIRECTORY: add the path in the Git repository to the root directory that contains the configuration that you want to sync to. This field is optional and the default is the root directory (/) of the repository.
    • AUTH_TYPE: add one of the following authentication types:

      • none
      • ssh
      • cookiefile
      • token
      • gcenode: If Workload Identity is enabled in your cluster, you cannot use gcenode. This field is required.
    • SECRET_NAME: add the name of your Secret. If you are using a Secret, you must add the Secret's public key to the Git provider. This field is optional.

  8. Apply the changes:

    kubectl apply -f root-sync.yaml
    

3. Keeping configuration

If you have previously installed Config Sync, you can keep an existing root Git repository configured in your ConfigManagement object. However, this method is not recommended because it will be deprecated.

To configure the root repository by keeping the root repository in the ConfigManagement object, complete the following tasks:

  1. If you're using version 1.7.0 or later and installed Config Sync in a private cluster, add a firewall rule to allow port 8676. The Config Sync admission webhook uses port 8676 for drift prevention.

  2. In the ConfigManagement object, set the spec.enableMultiRepo and the spec.enableLegacyFields fields to true, set the spec.sourceFormat to unstructured to use an unstructured repository or hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default value is hierarchy. We recommend that you add unstructured as this format lets you organize your configs in the way that is most convenient to you.

    # config-management.yaml
    apiVersion: configmanagement.gke.io/v1
    kind: ConfigManagement
    metadata:
      name: config-management
    spec:
      enableMultiRepo: true
      enableLegacyFields: true
      sourceFormat: FORMAT
      git:
        syncRepo: REPO
        syncBranch: BRANCH
        secretType: TYPE
        policyDir: "DIRECTORY"
    # ...other fields...
    
  3. Apply the changes:

    kubectl apply -f config-management.yaml
    

Applying config-management.yaml with enableMultiRepo: true and enableLegacyFields: true set automatically generates the RootSync object on the cluster. Do not change the RootSync configuration that is generated.

To add optional namespace repositories, proceed to the Configuring syncing from namespace repositories section. If you only want to sync from a single repository and continue to use your current workflow, you can skip that section.

Verifying the sync status of the root repository

You can use the nomos status command to inspect the sync status of the root repository:

nomos status

You should see output similar to the following example:

my_managed_cluster-1
  --------------------
  <root>   git@github.com:foo-corp/acme/admin@main
  SYNCED   f52a11e4

Verifying the RootSync installation

When you create a RootSync object, Config Sync creates a reconciler called root-reconciler. A reconciler is a Pod that is deployed as a Deployment. It syncs manifests from a Git repository to a cluster.

You can verify that the RootSync object is working correctly by checking the status of the root-reconciler Deployment:

kubectl get -n config-management-system deployment/root-reconciler

You should see output similar to the following example:

NAME              READY   UP-TO-DATE   AVAILABLE   AGE
root-reconciler   1/1     1            1           3h42m

For further ways to explore the status of your RootSync object, see the Exploring the RootSync and RepoSync objects section.

Configuring syncing from namespace repositories

After you have set up your root repository, you can choose to set up namespace repositories that non-administrative users can control. Namespace repositories must be unstructured.

To configure namespace repositories, you need to assign permissions and create a RepoSync object that syncs your namespace repository to the cluster. You can choose between two methods for configuring namespace repositories:

  • Control namespace repositories in the root repository. This method centralizes all configuration of namespace repositories in the root repository, allowing a central administrator complete control of the setup.

  • Control namespace repositories with the Kubernetes API. This method delegates control of the namespace repository to the namespace owners.

Controlling namespace repositories in the root repository

In this method, the central administrator manages the setup of namespace repositories directly from the root repository. Because Config Sync manages the namespace repository objects, this method prevents any local changes to the namespace repository definitions.

To use this method, complete the following tasks:

  1. In the root repository, declare a namespace configuration:

    # ROOT_REPO/namespaces/NAMESPACE/namespace.yaml
    apiVersion: v1
    kind: Namespace
    metadata:
      name: NAMESPACE
    

    Replace NAMESPACE with a name for your namespace.

  2. In the root repository, create a RepoSync object in the same namespace:

    # ROOT_REPO/namespaces/NAMESPACE/repo-sync.yaml
     # If you are using a Config Sync version earlier than 1.8.0,
     # use: apiVersion: configsync.gke.io/v1alpha1
    apiVersion: configsync.gke.io/v1beta1
    kind: RepoSync
    metadata:
      name: repo-sync
      namespace: NAMESPACE
    spec:
      # Since this is for a namespace repository, the format should be unstructured
      sourceFormat: unstructured
      git:
       repo: REPOSITORY
       revision: REVISION
       branch: BRANCH
       dir: "DIRECTORY"
       auth: AUTH_TYPE
       secretRef:
         name: SECRET_NAME
    

    Replace the following:

    • NAMESPACE: add the name of your namespace.
    • REPOSITORY: add the URL of the Git repository to use as the namespace repository. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/csp-config-management/ uses the HTTPS protocol. If you don't enter a protocol, the URL is treated as an HTTPS URL. This field is required.
    • REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
    • BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
    • DIRECTORY: add the path in the Git repository to the root directory that contains the configuration that you want to sync to. This field is optional and the default is the root directory (/) of the repository.
    • AUTH_TYPE: add one of the following authentication types:

      • none
      • ssh
      • cookiefile
      • token
      • gcenode: If Workload Identity is enabled in your cluster, you cannot use gcenode.

      This field is required.

    • SECRET_NAME: add the name you intend to give your Secret. This field is optional.

    There can be at most one RepoSync object per namespace. To enforce this restriction, the object name must be repo-sync. All configs contained in the directory referenced by the RepoSync must also be in the same namespace as the RepoSync object.

  3. In the root repository, declare a RoleBinding configuration that grants the ns-reconciler-NAMESPACE service account permission to manage objects in the namespace. Config Sync automatically creates the ns-reconciler-NAMESPACE service account when the RepoSync config is synced to the cluster.

    To declare the RoleBinding, create the following object:

    # ROOT_REPO/namespaces/NAMESPACE/sync-rolebinding.yaml
     kind: RoleBinding
     apiVersion: rbac.authorization.k8s.io/v1
     metadata:
       name: syncs-repo
       namespace: NAMESPACE
     subjects:
     - kind: ServiceAccount
       name: ns-reconciler-NAMESPACE
       namespace: config-management-system
     roleRef:
       kind: ClusterRole
       name: RECONCILER_ROLE
       apiGroup: rbac.authorization.k8s.io
    

    Replace the following:

    • NAMESPACE: add the name of your namespace.
    • RECONCILER_ROLE: as the central administrator, you can set the RECONCILER_ROLE to enforce what kinds of configuration can be synced from the namespace repository. You can choose one of the following roles:

      • A default ClusterRole:

        • admin
        • edit

        To learn more, see User-facing roles.

      • A user-defined ClusterRole or Role declared in the root repository. This role allows for fine grained permissions.

  4. Commit the changes to the root repository:

     git add .
     git commit -m 'Setting up new namespace repository.'
     git push
    
  5. If required, create a Secret based on your preferred authentication method. If you used none as your authentication type, you can skip this step.

    The Secret must meet following requirements:

    • Create the Secret in the same namespace as the RepoSync.
    • The Secret's name must match the spec.git.secretRef name you defined in repo-sync.yaml.
    • You must add the Secret's public key to the Git provider.
  6. To verify the configuration, use kubectl get on one of the objects in the namespace repository. For example:

    kubectl get rolebindings -n NAMESPACE
    

Controlling namespace repositories with the Kubernetes API

In this method, the central administrator only declares the namespace in the root repository and delegates declaration of the RepoSync object to the application operator.

Central administrator tasks

The central administrator completes the following tasks:

  1. In the root repository, declare a namespace configuration:

    # ROOT_REPO/namespaces/NAMESPACE/namespace.yaml
     apiVersion: v1
     kind: Namespace
     metadata:
       name: NAMESPACE
    

    Replace NAMESPACE with a name for your namespace.

  2. In the root repository, declare a RoleBinding to give the application operators permissions. Use RBAC escalation prevention to ensure that the application operator cannot later apply a role binding with permissions not granted by this role binding.

    To declare the RoleBinding, create the following manifest:

    # ROOT_REPO/namespaces/NAMESPACE/operator-rolebinding.yaml
     kind: RoleBinding
     # Add RBAC escalation prevention
     apiVersion: rbac.authorization.k8s.io/v1
     metadata:
       name: operator
       namespace: NAMESPACE
     subjects:
     - kind: User
       name: USERNAME
       apiGroup: rbac.authorization.k8s.io
     roleRef:
       kind: ClusterRole
       name: OPERATOR_ROLE
       apiGroup: rbac.authorization.k8s.io
    

    Replace the following:

    • NAMESPACE: add the namespace you created in the root repository.
    • USERNAME: add the username of the application operator.
    • OPERATOR_ROLE: as the central administrator, you can set OPERATOR_ROLE to enforce what kinds of configurations can be synced from the namespace repository. You can choose one of the following roles:

      • A default ClusterRole:

        • admin
        • edit

        To learn more, see User-facing roles.

      • A user-defined ClusterRole or Role declared in the root repository. This role allows for fine grained permissions.

  3. Commit the changes to the root repository:

     git add .
     git commit -m 'Setting up new namespace repository.'
     git push
    

Application operator tasks

The application operator completes the following tasks:

  1. Declare a RoleBinding configuration that grants the auto-provisioned ns-reconciler-NAMESPACE service account permission to manage objects in the namespace. Config Sync automatically creates the ns-reconciler-NAMESPACE service account when the RepoSync config is synced to the cluster.

    To declare the RoleBinding, create the following manifest:

    # sync-rolebinding.yaml
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: syncs-repo
      namespace: NAMESPACE
    subjects:
    - kind: ServiceAccount
      name: ns-reconciler-NAMESPACE
      namespace: config-management-system
    roleRef:
      kind: ClusterRole
      name: RECONCILER_ROLE
      apiGroup: rbac.authorization.k8s.io
    

    Replace the following:

    • NAMESPACE: add the namespace you created in the root repository.
    • RECONCILER_ROLE: as the application operator you can set RECONCILER_ROLE to enforce what kinds of configuration can be synced from the namespace repository. You can only further restrict the set of permissions the central administrator has granted you. As a result, this role cannot be more permissive than the OPERATOR_ROLE that the central administrator declared in the previous section.
  2. Apply the RoleBinding configuration:

    kubectl apply -f sync-rolebinding.yaml
    
  3. If required, create a Secret based on your preferred authentication method. If you used none as your authentication type, you can skip this step.

    The Secret must meet the following requirements:

    • Create the Secret in the same namespace as the RepoSync.
    • The Secret's name must match the spec.git.secretRef name you defined in root-sync.yaml.
    • You must add the Secret's public key to the Git provider.
  4. Declare a RepoSync configuration:

    # repo-sync.yaml
    # If you are using a Config Sync version earlier than 1.8.0,
    # use: apiVersion: configsync.gke.io/v1alpha1
    apiVersion: configsync.gke.io/v1beta1
    kind: RepoSync
    metadata:
      name: repo-sync
      namespace: NAMESPACE
    spec:
      # Since this is for a namespace repository, the format should be unstructured
      sourceFormat: unstructured
      git:
       repo: REPOSITORY
       revision: REVISION
       branch: BRANCH
       dir: "DIRECTORY"
       auth: AUTH_TYPE
       secretRef:
         name: SECRET_NAME
    

    Replace the following:

    • REPOSITORY: add the URL of the Git repository to use as the namespace repository. You can enter URLs using either the HTTPS or SSH protocol. For example, https://github.com/GoogleCloudPlatform/csp-config-management/ uses the HTTPS protocol. If you don't enter a protocol, the URL is treated as an HTTPS URL. This field is required.
    • REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
    • BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
    • DIRECTORY: add the path in the Git repository to the root directory that contains the configuration that you want to sync to. This field is optional and the default is the root directory (/) of the repository.
    • AUTH_TYPE: add one of the following authentication types:

      • none
      • ssh
      • cookiefile
      • token
      • gcenode: If Workload Identity is enabled in your cluster, you cannot use gcenode.

      This field is required.

    • SECRET_NAME: add the name you intend to give your Secret. This field is optional.

    There can be at most one RepoSync object per namespace. To enforce this, the object's name must be repo-sync. All configs contained in the directory referenced by the RepoSync must also be in the same namespace as the RepoSync object.

  5. Apply the RepoSync configuration:

    kubectl apply -f repo-sync.yaml
    
  6. To verify the configuration, use kubectl get on one of the objects in the namespace repository. For example:

    kubectl get rolebindings -n NAMESPACE
    

Verifying the sync status of the namespace repository

You can use the nomos status command to inspect the sync status of the namespace repository:

nomos status

You should see output similar to the following example:

my_managed_cluster-1
  --------------------
  <root>   git@github.com:foo-corp/acme/admin@main
  SYNCED   f52a11e4
  --------------------
  bookstore  git@github.com:foo-corp/acme/bookstore@v1
  SYNCED     34d1a8c8

In this example output, the namespace repository is configured for namespace named bookstore.

Verifying the RepoSync installation

When you create a RepoSync object, Config Sync creates a reconciler called ns-reconciler-NAMESPACE, where NAMESPACE is the namespace you created your RepoSync object in.

You can verify that the RepoSync object is working correctly by checking the status of the ns-reconciler-NAMESPACE Deployment:

kubectl get -n config-management-system deployment/ns-reconciler-NAMESPACE

Replace NAMESPACE with the namespace that you created your namespace repository in.

For further ways to explore the status of your RepoSync object, see the Exploring the RootSync and RepoSync objects section.

Prevent config drift

Config Sync reduces the risk of "shadow ops" through drift detection. When unvetted changes are pushed to live clusters, Config Sync detects and remediates any drift from the source of truth in Git. Furthermore, starting from version 1.7.0, Config Sync multi-repo mode provides an admission webhook that rejects conflicting changes from being pushed to live clusters. This provides better user feedback when manual changes are attempted, rather than allowing it and silently remediating the change.

In addition, this admission webhook prevents Config Sync metadata and annotations from being manually modified, which would otherwise cause errors in the configuration for admission webhook or reconcilers.

Stopping and resuming syncing

The following sections show you how to temporarily stop and resume syncing. You might need to do this if an incorrect config is accidentally committed to your repository.

Only a central administrator can stop syncing in the root repository.

The ability to stop syncing in namespace repositories depends on which configuration method was used for your namespace repositories.

  • If the Control namespace repositories in the root repository method was used, a central administrator is the only one who can stop and resume syncing.

  • If the Control namespace repositories with the Kubernetes API method was used, application operators can stop and resume syncing from the namespace repositories that they work on.

Stopping syncing

The following sections show you how to stop syncing for the root repository and namespace repositories.

Stopping syncing from the root repository

To stop syncing from the root repository, a central administrator can run the following command:

kubectl -n config-management-system scale deployment root-reconciler --replicas=0

This command reduces the replicas count in the root-reconciler Deployment to 0.

Stopping syncing from namespace repositories

Select the Root repository method or Kubernetes API method tab to view the relevant instructions.

Root repository method

If the Control namespace repositories in the root repository method was used, central administrators can run the following commands to stop syncing from a namespace repository:

kubectl -n config-management-system scale deployment ns-reconciler-NAMESPACE --replicas=0

The command reduces the replica count in the ns-reconciler-NAMESPACE Deployment to 0.

Kubernetes API method

If the Control namespace repositories with the Kubernetes API method was used, application operators can stop syncing by running the following commands:

  1. Retrieve the RepoSync configuration and save it to use it later when you want to resume syncing:

    kubectl -n NAMESPACE get reposyncs repo-sync -oyaml > repo-sync.yaml
    

    Replace NAMESPACE with the namespace of your RepoSync object.

  2. Delete the RepoSync configuration:

    kubectl -n NAMESPACE delete reposyncs repo-sync
    

    This command triggers the Reconciler Manager to remove the namespace reconciler (ns-reconciler-NAMESPACE) from NAMESPACE and stops syncing.

Stopping admission webhook from blocking drifts (optional)

To stop Config Sync from performing drift prevention, after stopping the reconcilers, you can stop the admission webhook by running the following command:

kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io admission-webhook.configsync.gke.io

This step is optional, and is only required when you need to make conflicting changes to managed resources in the cluster. This requires root reconciler to be stopped first; otherwise, the root reconciler tries to generate the webhook configs after the deletion.

Resuming syncing

This section shows you how to resume syncing for the root repository and for namespace repositories.

Resuming syncing from the root repository

To resume syncing from a root repository, a central administrator can run the following command:

kubectl -n config-management-system scale deployment root-reconciler --replicas=1

This command scales the root-reconciler Deployment to 1 replica.

Resuming syncing from a namespace repository

Select the Root repository method or Kubernetes API method tab to view the relevant instructions.

Root repository method

If you used the Control namespace repositories in the root repository method, a central administrator can run the following command:

kubectl -n config-management-system scale deployment ns-reconciler-NAMESPACE --replicas=1

This command scales the ns-reconciler-NAMESPACE Deployment to 1 replica.

Kubernetes API method

If you used the Control namespace repositories with the Kubernetes API method, application operators can resume syncing by reapplying repo-sync.yaml which contains the RepoSync configuration:

kubectl apply -f repo-sync.yaml

This command triggers tne Reconciler Manager to create a namespace reconciler process and create a ns-reconciler-NAMESPACE Deployment.

Resuming admission webhook

After the root reconciler is resumed, webhook configurations is generated automatically. To validate that it's generated successfully, run the following command:

kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io admission-webhook.configsync.gke.io

Removing the root and namespace repositories

To remove the root repository, delete the RootSync object. For example,

kubectl delete -f root-sync.yaml

To uninstall a namespace repository, delete a RepoSync object. This action uninstalls the matching namespace repository.

Ignoring object mutations

In Config Sync versions 1.7.0 and later, if you don't want Config Sync to maintain the state of the object an the cluster after it exists, you can add the client.lifecycle.config.k8s.io/mutation: ignore annotation to the object that you want Config Sync to ignore mutations in. This action is useful when you want Config Sync to create an object and then ignore any conflicting changes to this object in the cluster.

The following example shows you how to add the annotation to an object:

metadata:
  annotations:
    client.lifecycle.config.k8s.io/mutation: ignore 

You cannot manually modify this annotation on managed objects in the cluster.

Conflict resolution

When you are working with two repositories, conflicts might arise when the same object (matching groups, kinds, names, and namespaces) is declared both in the root and a namespace repository. When this conflict happens, only the declaration in the root repository is applied to the cluster.

If this conflict happens, the RootSync won't report any problems since the root repository takes precedence. The RepoSync reports error KNV 1060 in its status.

Since the root repository always takes precedence, you can resolve the conflict by updating the root repository to match the namespace repository, or by deleting the conflicting object in the namespace repository.

Exploring the RootSync and RepoSync objects

The following sections show you different ways that you can explore your RootSync and RepoSync objects.

Viewing logs

For further information on potential errors, you can view the logs for the RootSync and RepoSync objects.

To view the logs for the RootSync reconciler, run the following command:

kubectl logs -n config-management-system deployment/root-reconciler CONTAINER_NAME

Replace CONTAINER_NAME with either git-sync, reconciler, or otel-agent. The git-sync container pulls configs from a Git repository to a local directory that the reconciler container can read from. The reconciler container applies those configs to the cluster. The otel-agent container is an OpenTelemetry agent for retrieving and exporting Config Sync metrics.

To view the logs of the RepoSync reconciler, run the following command:

kubectl logs -n config-management-system deployment/ns-reconciler-NAMESPACE CONTAINER_NAME

Replace the following:

  • NAMESPACE: the namespace that you created your namespace repository in.
  • CONTAINER_NAME: either git-sync, reconciler, or otel-agent.

Viewing synced commits

You can check which commit is synced to the cluster.

To view details for the RootSync, run the following command:

kubectl get rootsync root-sync -n config-management-system

To view details for the RepoSync, run the following command:

kubectl get reposync repo-sync -n config-management-system

You should see output similar to the following example:

NAME        SOURCECOMMIT                               SYNCCOMMIT
root-sync   66882815f0ef5517df27e864fb1315e97756ab72   66882815f0ef5517df27e864fb1315e97756ab72

The value in the SOURCECOMMIT column is the commit from the Git repository that should be synced to the cluster. The value in the SYNCCOMMIT column is the commit that is currently deployed to the cluster. If the two values in the SOURCECOMMIT and SYNCCOMMIT columns are the same, the expected commit has been deployed to the cluster.

Viewing object details

To view details of the RootSync and RepoSync objects, use kubectl describe. This command can provide you with further information about potential errors.

To view details for the RootSync object, run the following command:

kubectl describe rootsync root-sync -n config-management-system

To view details for the RepoSync object, run the following command:

kubectl describe reposync repo-sync -n config-management-system

Viewing if a resource is ready

To learn if the resources synced to the cluster are ready, view the reconciliation status. For example, this command can show you if a synced Deployment is ready to serve traffic.

For a Git repository synced to the cluster, the reconciliation status of all resources are aggregated in a resource called ResourceGroup. For each RootSync or RepoSync object, a ResourceGroup is generated to capture the set of resources applied to the cluster and aggregate their statuses.

To view the reconciliation status for the RootSync object, run the following command:

kubectl get resourcegroup.kpt.dev root-sync -n config-management-system -o yaml

To view the reconciliation status for the RepoSync object, run the following command:

kubectl get resourcegroup.kpt.dev repo-sync -n NAMESPACE -o yaml

In the output, you see all of the ResourceGroup resource statuses. For example, the following output shows that a Deployment named nginx-deployment is ready:

resourceStatuses:
- group: apps
  kind: Deployment
  name: nginx-deployment
  namespace: default
  status: Current

Troubleshooting

The following sections help you troubleshoot problems when you are syncing from multiple repositories.

Error: admission webhook denied a request

If you receive the following error when you try to apply a change to a field that Config Sync manages, then you might have have made a conflicting change:

error: OBJECT could not be patched: admission webhook "v1.admission-webhook.configsync.gke.io"
denied the request: fields managed by Config Sync can not be modified

When you declare a field in a config and your repository is synced to a cluster, Config Sync manages that field. Any change you attempt to make to that field is a conflicting change.

For example, if you have a Deployment config in your repository with a label of environment:prod and you try to change that label to environment:dev in your cluster, there would be a conflicting change and you would receive the preceding error message. However, if you add a new label (for example, tier:frontend) to the Deployment, there would not be a conflict.

If you want Config Sync to ignore any changes to an object, you can add the annotation described in Ignoring object mutations.

Error: admission webhook request i/o timeout

If you receive the following error when reconciler tries to apply a config to the cluster:

KNV2009: Internal error occurred: failed calling webhook "v1.admission-webhook.configsync.gke.io": Post https://admission-webhook.config-management-system.svc:8676/admission-webhook?timeout=3s: dial tcp 10.1.1.186:8676: i/o timeout

This can be due to admission webhook port 8676 being blocked by the firewall to the control plane network. To resolve this issue, add a firewall rule to allow port 8676, which the Config Sync admission webhook uses for drift prevention.

Error: admission webhook connection refused

If you receive the following error when reconciler tries to apply a config to the cluster:

KNV2009: Internal error occurred: failed calling webhook "v1.admission-webhook.configsync.gke.io": Post "https://admission-webhook.config-management-system.svc:8676/admission-webhook?timeout=3s": dial tcp 10.92.2.14:8676: connect: connection refused

This means that the admission webhook isn't ready yet. It's a transient error you might see when bootstrapping Config Sync.

If the issue persists, take a look at the admission webhook Deployment to see if its Pods can be scheduled and are healthy.

kubectl describe deploy admission-webhook -n config-management-system

kubectl get pods -n config-management-system -l app=admission-webhook

ResourceGroup fields keep changing

Occasionally, your ResourceGroup can enter a loop that keeps updating the ResourceGroup's spec. If this happens, you might notice the following issues:

  • The metadata.generation of a ResourceGroup keeps increasing in a short period of time.
  • The ResourceGroup spec keeps changing.
  • The ResourceGroup spec doesn't include the status.resourceStatuses of the resources being synced to the cluster.

If you observe these issues, it means that some of the resources in your Git repositories failed to be applied to the cluster. The cause of these issues is that you are missing the permissions that you need to apply those resources.

You can verify that the permissions are missing by getting the RepoSync resource status:

kubectl get reposync repo-sync -n NAMESPACE -o yaml

Replace NAMESPACE with the namespace that you created your namespace repository in.

If you see the following messages in the status, it means that the reconciler in NAMESPACE lacks the permission needed to apply the resource:

errors:
  - code: "2009"
    errorMessage: |-
      KNV2009: deployments.apps "nginx-deployment" is forbidden: User "system:serviceaccount:config-management-system:ns-reconciler-     default" cannot get resource "deployments" in API group "apps" in the namespace "default"

      For more information, see https://g.co/cloud/acm-errors#knv2009

To fix this issue, you need to declare a RoleBinding configuration that grants the ns-reconciler-NAMESPACE service account permission to manage the failed resource in that namespace. Details on how to add a RoleBinding are included in the Configuring syncing from namespace repositories section.

Large number of resources in Git repository

When the Git repository synced by either the RepoSync or RootSync objects contains configuration for more than a few thousand resources, it can cause the ResourceGroup to exceed the etcd object size limit. When this happens, you cannot view the aggregated status for resources in your Git repository. While you won't be able to view the aggregated status, your repository is still synced. If you don't see any errors in the RepoSync or RootSync objects, that means your Git repository is synced to the cluster successfully.

To check if the ResourceGroup resource exceeds the etcd object size limit, you need to check both the ResourceGroup resource status and the log of the ResourceGroup controller:

  1. Check the ResourceGroup status with the following command:

    • To check the RootSync, run the following command:
     kubectl get resourcegroup.kpt.dev root-sync -n config-management-system
    
    • To check the RepoSync, run the following command:
    # For the RepoSync:
    kubectl get resourcegroup.kpt.dev repo-sync -n NAMESPACE
    

    You should see output similar to the following example:

    NAME        RECONCILING   STALLED   AGE
    root-sync   True          False     35m
    

    If the value in the RECONCILING column is True, it means that the ResourceGroup resource is still reconciling.

  2. Check the logs for the ResourceGroup controller with the following command:

    kubectl logs deployment/resource-group-controller-manager -c manager -n resource-group-system
    

    If you see the following error in the output, the ResourceGroup resource is too large and exceeds the etcd object size limit:

    "error":"etcdserver: request is too large"
    

To prevent the ResourceGroup from getting too large, reduce the number of resources in your Git repository. For example, split the repository and use one RootSync object with multiple RepoSync objects instead of using only one RootSync object to sync all the resources.

What's next