Syncing from multiple repositories

Enabling multi-repo mode in Config Sync lets you sync configurations from multiple repositories to the same set of clusters. To use multi-repo mode, you need to create a single root repository and can also optionally create namespace repositories. A root repository can sync cluster-scoped and namespace-scoped configs and is typically managed by an administrator. Namespace repositories contain namespace-scoped configs synced to a particular namespace across clusters. To learn more about these types of repositories, see the Repositories section in the Config Sync overview.

Each repository has a single Git reference (a repository branch, commit or tag, and directory tuple) that you can manage separately. This configuration decouples the config deployment lifecycle for different teams. It also provides you with more autonomy in choosing where you want to place the repository and how to structure it.

The following diagram shows you an overview of how teams might use a root repository and namespace repositories:

A central administrator controlling multiple configs and app operators controlling their own namespace configs.

In this diagram, a central administrator manages the centralized infrastructure for the organization and enforces policies on the cluster and on all namespaces in the organization.

The application operators, who are responsible for managing live deployments, apply configurations to the applications in the namespaces that they work on.

Before you begin

Complete the following tasks before enabling multi-repo mode:

  • Create, or have access to, Git repositories.
  • Create, or have access to, a Google Cloud project.
  • Create, or have access to, a Google Kubernetes Engine (GKE) cluster that meets the following requirements:
    • Is a standard cluster. Config Sync is not supported on Autopilot clusters.
    • Uses a GKE version between 1.18 and 1.20.
    • Has a node pool that uses a machine type with at least 4 vCPU.
  • Use a Config Sync version of 1.5.3 or later.

Limitations

Consider the following limitations when you use multi-repo mode:

  • If you have previously installed Config Sync without enabling multi-repo mode, the spec.git fields in the ConfigManagement object, are not compatible with thespec.enableMultiRepo field. If you still need to use spec.git, see the third option in the Configuring syncing from the root repository section.
  • ClusterSelectors and NamespaceSelectors (including annotations pointing to Selectors) only work in the root repository.
  • If Workload Identity is enabled in your cluster, you cannot use gcenode in the auth field of your RootSync or RepoSync objects.
  • If you are using a Config Sync version earlier than 1.7, you need to use an apiVersion of configsync.gke.io/v1alpha1 in your RootSync and RepoSync objects. In addition, you cannot use the annotation described in Ignoring object mutations. Upgrade Config Sync to Config Sync version 1.7 or later to use these features.
  • If the Git repository being synced by either a RootSync or RepoSync object contains thousands of resources, you might encounter errors. Follow the instructions in the Large number of resources in a Git repository section to troubleshoot this issue.

Configuring syncing from the root repository

To configure syncing from the root repository, you need to enable multi-repo mode in your ConfigManagement object and create a RootSync object that syncs your root repository to the cluster. You can only create one root repository per cluster and the root repository can be either an unstructured repository or an hierarchical repository.

The following sections describe three different methods for enabling multi-repo mode:

  1. If you haven't previously installed Config Sync, create a new root Git repository configuration.
  2. If you have previously installed Config Sync, move the root Git repository configuration to RootSync.
  3. If you have previously installed Config Sync and you need to use the spec.git fields, keep the root Git repository configuration in the ConfigManagement YAML file.

1. New configuration

If you have not previously installed Config Sync, complete the following tasks to enable multi-repo mode:

  1. Complete the following sections in Installing Config Sync:

    1. Before you begin
    2. Deploying the operator
    3. Granting Operator read-only access to Git

  2. Create a file named config-management.yaml and copy the following YAML file into it:

    # config-management.yaml
apiVersion: configmanagement.gke.io/v1
kind: ConfigManagement
metadata:
  name: config-management
spec:
  enableMultiRepo: true

  3. Apply the changes:

    kubectl apply -f config-management.yaml

  4. Create a RootSync object:

    # root-sync.yaml
# If you are using a Config Sync version earlier than 1.7,
# use: apiVersion: configsync.gke.io/v1alpha1
apiVersion: configsync.gke.io/v1beta1
kind: RootSync
metadata:
  name: root-sync
  namespace: config-management-system
spec:
  sourceFormat: FORMAT
  git:
    repo: REPOSITORY
    revision: REVISION
    branch: BRANCH
    dir: "DIRECTORY"
    auth: AUTH_TYPE
    secretRef:
      name: SECRET_NAME

    Replace the following:

    • FORMAT: add unstructured to use an unstructured repository or add hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default is value is hierarchy. We recommend that you add unstructured as this format lets you organize your configs in the way that is most convinenent to you.
    • REPOSITORY: add the URL of the Git repository to use as the root repository. This field is required.
    • REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
    • BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
    • DIRECTORY: add the path in the Git repository to the root directory that contains the configuration that you want to sync to. This field is optional and the default is the root directory (/) of the repository.

    • AUTH_TYPE: add one of the following authentication types:

      • none
      • ssh
      • cookiefile
      • token
      • gcenode: If Workload Identity is enabled in your cluster, you cannot use gcenode. This field is required.

    • SECRET_NAME: add the name of your Secret. If you are using a Secret, you must add the Secret's public key to the Git provider. This field is optional.

  5. Apply the changes:

    kubectl apply -f root-sync.yaml

2. Moving configuration

If you have previously installed Config Sync, you can move the Git repository configurations from your existing ConfigManagement object to a RootSync object.

If configure the root repository by moving the configuration, complete the following tasks:

  1. Open your ConfigManagement object.
  2. Make a copy of the values in the spec.git fields. You use these values when you create the RootSync object.
  3. Remove all of the spec.git fields (including git:) from the ConfigManagement object.

  4. In the ConfigManagement object, set the spec.enableMultiRepo field to true:

    # config-management.yaml
apiVersion: configmanagement.gke.io/v1
kind: ConfigManagement
metadata:
  name: config-management
spec:
  enableMultiRepo: true

  5. Apply the changes:

    kubectl apply -f config-management.yaml

  6. Using the values you copied from the ConfigManagement object, create the RootSync object. For example:

    # root-sync.yaml
# If you are using a Config Sync version earlier than 1.7,
# use: apiVersion: configsync.gke.io/v1alpha1
apiVersion: configsync.gke.io/v1beta1
kind: RootSync
metadata:
  name: root-sync
  namespace: config-management-system
spec:
  sourceFormat: FORMAT
  git:
    repo: REPOSITORY
    revision: REVISION
    branch: BRANCH
    dir: "DIRECTORY"
    auth: AUTH_TYPE
    secretRef:
      name: SECRET_NAME

    Replace the following:

    • FORMAT: add unstructured to use an unstructured repository or add hierarchy to use a hierarchical repository. These values are case-sensitive. This field is optional and the default is value is hierarchy.
    • REPOSITORY: add the URL of the Git repository to use as the root repository. This field is required.
    • REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
    • BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
    • DIRECTORY: add the path in the Git repository to the root directory that contains the configuration that you want to sync to. This field is optional and the default is the root directory (/) of the repository.

    • AUTH_TYPE: add one of the following authentication types:

      • none
      • ssh
      • cookiefile
      • token
      • gcenode: If Workload Identity is enabled in your cluster, you cannot use gcenode. This field is required.

    • SECRET_NAME: add the name of your Secret. If you are using a Secret, you must add the Secret's public key to the Git provider. This field is optional.

  7. Apply the changes:

    kubectl apply -f root-sync.yaml

3. Keeping configuration

If you have previously installed Config Sync, you can keep an existing root Git repository configured in your ConfigManagement object. However, this method is not recommended because it will be deprecated.

To configure the root repository by keeping the root repository in the ConfigManagement object, complete the following tasks:

  1. In the ConfigManagement object, set the spec.enableMultiRepo and the spec.enableLegacyFields fields to true:

    # config-management.yaml
apiVersion: configmanagement.gke.io/v1
kind: ConfigManagement
metadata:
  name: config-management
spec:
  enableMultiRepo: true
  enableLegacyFields: true
  git:
    syncRepo: REPO
    syncBranch: BRANCH
    secretType: TYPE
    policyDir: "DIRECTORY"
# ...other fields...

  2. Apply the changes:

    kubectl apply -f config-management.yaml

Applying config-management.yaml with enableMultiRepo: true and enableLegacyFields: true set automatically generates the RootSync object on the cluster. Do not change the RootSync configuration that is generated.

To add optional namespace repositories, proceed to the Configuring syncing from namespace repositories section. If you only want to sync to a single repository and continue to use your current workflow, you can skip that section.

Verifying the RootSync installation

When you create a RootSync object, Config Sync creates a reconciler called root-reconciler. A reconciler is a Pod that is deployed as a Deployment. It syncs manifests from a Git repository to a cluster.

You can verify that the RootSync object is working correctly by checking the status of the root-reconciler Deployment:

kubectl get -n config-management-system deployment/root-reconciler

You should see output similar to the following example:

NAME              READY   UP-TO-DATE   AVAILABLE   AGE
root-reconciler   1/1     1            1           3h42m

For further ways to explore the status of your RootSync object, see the Exploring the RootSync and RepoSync objects section.

Configuring syncing from namespace repositories

After you have set up your root repository, you can choose to set up namespace repositories that non-administrative users can control. Namespace repositories must be unstructured.

To configure namespace repositories, you need to assign permissions and create a RepoSync object that syncs your namespace repository to the cluster. You can choose between two methods for configuring namespace repositories:

  • Control namespace repositories in the root repository. This method centralizes all configuration of namespace repositories in the root repository, allowing a central administrator complete control of the setup.

  • Control namespace repositories with the Kubernetes API. This method delegates control of the namespace repository to the namespace owners.

Controlling namespace repositories in the root repository

In this method, the central administrator manages the setup of namespace repositories directly from the root repository. Because Config Sync manages the namespace repository objects, this method prevents any local changes to the namespace repository definitions.

To use this method, complete the following tasks:

  1. In the root repository, declare a namespace configuration:

    # ROOT_REPO/namespaces/NAMESPACE/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: NAMESPACE

    Replace NAMESPACE with a name for your namespace.

  2. In the root repository, create a RepoSync object in the same namespace:

    # ROOT_REPO/namespaces/NAMESPACE/repo-sync.yaml
 # If you are using a Config Sync version earlier than 1.7,
 # use: apiVersion: configsync.gke.io/v1alpha1
apiVersion: configsync.gke.io/v1beta1
kind: RepoSync
metadata:
  name: repo-sync
  namespace: NAMESPACE
spec:
  # Since this is for a namespace repository, the format should be unstructured
  sourceFormat: unstructured
  git:
   repo: REPOSITORY
   revision: REVISION
   branch: BRANCH
   dir: "DIRECTORY"
   auth: AUTH_TYPE
   secretRef:
     name: SECRET_NAME

    Replace the following:

    • NAMESPACE: add the name of your namespace.
    • REPOSITORY: add the URL of the Git repository to use as the root repository. This field is required.
    • REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
    • BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
    • DIRECTORY: add the path in the Git repository to the root directory that contains the configuration that you want to sync to. This field is optional and the default is the root directory (/) of the repository.

    • AUTH_TYPE: add one of the following authentication types:

      • none
      • ssh
      • cookiefile
      • token
      • gcenode: If Workload Identity is enabled in your cluster, you cannot use gcenode.

      This field is required.

    • SECRET_NAME: add the name you intend to give your Secret. This field is optional.

    There can be at most one RepoSync object per namespace. To enforce this restriction, the object name must be repo-sync. All configs contained in the directory referenced by the RepoSync must also be in the same namespace as the RepoSync object.

  3. In the root repository, declare a RoleBinding configuration that grants the ns-reconciler-NAMESPACE service account permission to manage objects in the namespace. Config Sync automatically creates the ns-reconciler-NAMESPACE service account when the RepoSync config is synced to the cluster.

    To declare the RoleBinding, create the following object:

    # ROOT_REPO/namespaces/NAMESPACE/sync-rolebinding.yaml
 kind: RoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: syncs-repo
   namespace: NAMESPACE
 subjects:
 - kind: ServiceAccount
   name: ns-reconciler-NAMESPACE
   namespace: config-management-system
 roleRef:
   kind: ClusterRole
   name: RECONCILER_ROLE
   apiGroup: rbac.authorization.k8s.io

    Replace the following:

    • NAMESPACE: add the name of your namespace.

    • RECONCILER_ROLE: as the central administrator, you can set the RECONCILER_ROLE to enforce what kinds of configuration can be synced from the namespace repository. You can choose one of the following roles:

      • A default ClusterRole:

        • admin
        • edit

        To learn more, see User-facing roles.

      • A user-defined ClusterRole or Role declared in the root repository. This role allows for fine grained permissions.

  4. Commit the changes to the root repository:

     git add .
 git commit -m 'Setting up new namespace repository.'
 git push

  5. If required, create a Secret based on your preferred authentication method. If you used none as your authentication type, you can skip this step.

    The Secret must meet following requirements:

    • Create the Secret in the same namespace as the RepoSync.
    • The Secret's name must match the spec.git.secretRef name you defined in repo-sync.yaml.
    • You must add the Secret's public key to the Git provider.

  6. To verify the configuration, use kubectl get on one of the objects in the namespace repository. For example:

    kubectl get rolebindings -n NAMESPACE

Controlling namespace repositories with the Kubernetes API

In this method, the central administrator only declares the namespace in the root repository and delegates declaration of the RepoSync object to the application operator.

Central administrator tasks

The central administrator completes the following tasks:

  1. In the root repository, declare a namespace configuration:

    # ROOT_REPO/namespaces/NAMESPACE/namespace.yaml
 apiVersion: v1
 kind: Namespace
 metadata:
   name: NAMESPACE

    Replace NAMESPACE with a name for your namespace.

  2. In the root repository, declare a RoleBinding to give the application operators permissions. Use RBAC escalation prevention to ensure that the application operator cannot later apply a role binding with permissions not granted by this role binding.

    To declare the RoleBinding, create the following manifest:

    # ROOT_REPO/namespaces/NAMESPACE/operator-rolebinding.yaml
 kind: RoleBinding
 # Add RBAC escalation prevention
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: operator
   namespace: NAMESPACE
 subjects:
 - kind: User
   name: USERNAME
   apiGroup: rbac.authorization.k8s.io
 roleRef:
   kind: ClusterRole
   name: OPERATOR_ROLE
   apiGroup: rbac.authorization.k8s.io

    Replace the following:

    • NAMESPACE: add the namespace you created in the root repository.
    • USERNAME: add the username of the application operator.

    • OPERATOR_ROLE: as the central administrator, you can set OPERATOR_ROLE to enforce what kinds of configurations can be synced from the namespace repository. You can choose one of the following roles:

      • A default ClusterRole:

        • admin
        • edit

        To learn more, see User-facing roles.

      • A user-defined ClusterRole or Role declared in the root repository. This role allows for fine grained permissions.

  3. Commit the changes to the root repository:

     git add .
 git commit -m 'Setting up new namespace repository.'
 git push

Application operator tasks

The application operator completes the following tasks:

  1. Declare a RoleBinding configuration that grants the auto-provisioned ns-reconciler-NAMESPACE service account permission to manage objects in the namespace. Config Sync automatically creates the ns-reconciler-NAMESPACE service account when the RepoSync config is synced to the cluster.

    To declare the RoleBinding, create the following manifest:

    # sync-rolebinding.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: syncs-repo
  namespace: NAMESPACE
subjects:
- kind: ServiceAccount
  name: ns-reconciler-NAMESPACE
  namespace: config-management-system
roleRef:
  kind: ClusterRole
  name: RECONCILER_ROLE
  apiGroup: rbac.authorization.k8s.io

    Replace the following:

    • NAMESPACE: add the namespace you created in the root repository.
    • RECONCILER_ROLE: as the application operator you can set RECONCILER_ROLE to enforce what kinds of configuration can be synced from the namespace repository. You can only further restrict the set of permissions the central administrator has granted you. As a result, this role cannot be more permissive than the OPERATOR_ROLE that the central administrator declared in the previous section.

  2. Apply the RoleBinding configuration:

    kubectl apply -f sync-rolebinding.yaml

  3. If required, create a Secret based on your preferred authentication method. If you used none as your authentication type, you can skip this step.

    The Secret must meet the following requirements:

    • Create the Secret in the same namespace as the RepoSync.
    • The Secret's name must match the spec.git.secretRef name you defined in root-sync.yaml.
    • You must add the Secret's public key to the Git provider.

  4. Declare a RepoSync configuration:

    # repo-sync.yaml
# If you are using a Config Sync version earlier than 1.7,
# use: apiVersion: configsync.gke.io/v1alpha1
apiVersion: configsync.gke.io/v1beta1
kind: RepoSync
metadata:
  name: repo-sync
  namespace: NAMESPACE
spec:
  # Since this is for a namespace repository, the format should be unstructured
  sourceFormat: unstructured
  git:
   repo: REPOSITORY
   revision: REVISION
   branch: BRANCH
   dir: "DIRECTORY"
   auth: AUTH_TYPE
   secretRef:
     name: SECRET_NAME

    Replace the following:

    • REPOSITORY: add the URL of the Git repository to use as the root repository. This field is required.
    • REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
    • BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
    • DIRECTORY: add the path in the Git repository to the root directory that contains the configuration that you want to sync to. This field is optional and the default is the root directory (/) of the repository.

    • AUTH_TYPE: add one of the following authentication types:

      • none
      • ssh
      • cookiefile
      • token
      • gcenode: If Workload Identity is enabled in your cluster, you cannot use gcenode.

      This field is required.

    • SECRET_NAME: add the name you intend to give your Secret. This field is optional.

    There can be at most one RepoSync object per namespace. To enforce this, the object's name must be repo-sync. All configs contained in the directory referenced by the RepoSync must also be in the same namespace as the RepoSync object.

  5. Apply the RepoSync configuration:

    kubectl apply -f repo-sync.yaml

  6. To verify the configuration, use kubectl get on one of the objects in the namespace repository. For example:

    kubectl get rolebindings -n NAMESPACE

Verifying the RepoSync installation

When you create a RepoSync object, Config Sync creates a reconciler called ns-reconciler-NAMESPACE, where NAMESPACE is the namespace you created your RepoSync object in.

You can verify that the RepoSync object is working correctly by checking the status of the ns-reconciler-NAMESPACE Deployment:

kubectl get -n config-management-system deployment/ns-reconciler-NAMESPACE

Replace NAMESPACE with the namespace that you created your namespace repository in.

For further ways to explore the status of your RepoSync object, see the Exploring the RootSync and RepoSync objects section.

Stopping and resuming syncing

The following sections show you how to temporarily stop and resume syncing. You might need to do this if an incorrect config is accidentally committed to your repository.

Only a central administrator can stop syncing in the root repository.

The ability to stop syncing in namespace repositories depends on which configuration method was used for your namespace repositories.

  • If the Control namespace repositories in the root repository method was used, a central administrator is the only one who can stop and resume syncing.

  • If the Control namespace repositories with the Kubernetes API method was used, application operators can stop and resume syncing from the namespace repositories that they work on.

Stopping syncing

The following sections show you how to stop syncing for the root repository and namespace repositories.

Stopping syncing from the root repository

To stop syncing from the root repository, a central administrator can run the following command:

kubectl -n config-management-system scale deployment root-reconciler --replicas=0

This command reduces the replicas count in the root-reconciler Deployment to 0.

Stopping syncing from namespace repositories

Select the Root repository method or Kubernetes API method tab to view the relevant instructions.

Root repository method

If the Control namespace repositories in the root repository method was used, central administrators can run the following commands to stop syncing from a namespace repository:

kubectl -n config-management-system scale deployment ns-reconciler-NAMESPACE --replicas=0

The command reduces the replica count in the ns-reconciler-NAMESPACE Deployment to 0.

Kubernetes API method

If the Control namespace repositories with the Kubernetes API method was used, application operators can stop syncing by running the following commands:

  1. Retrieve the RepoSync configuration and save it to use it later when you want to resume syncing:

    kubectl -n NAMESPACE get reposyncs repo-sync -oyaml > repo-sync.yaml

    Replace NAMESPACE with the namespace of your RepoSync object.

  2. Delete the RepoSync configuration:

    kubectl -n NAMESPACE delete reposyncs repo-sync

    This command triggers the Reconciler Manager to remove the namespace reconciler (ns-reconciler-NAMESPACE) from NAMESPACE and stops syncing.

Resuming syncing

This section shows you how to resume syncing for the root repository and for namespace repositories.

Resuming syncing from the root repository

To resume syncing from a root repository, a central administrator can run the following command:

kubectl -n config-management-system scale deployment root-reconciler --replicas=1

This command scales the root-reconciler Deployment to 1 replica.

Resuming syncing from a namespace repository

Select the Root repository method or Kubernetes API method tab to view the relevant instructions.

Root repository method

If you used the Control namespace repositories in the root repository method, a central administrator can run the following command:

kubectl -n config-management-system scale deployment ns-reconciler-NAMESPACE --replicas=1

This command scales the ns-reconciler-NAMESPACE Deployment to 1 replica.

Kubernetes API method

If you used the Control namespace repositories with the Kubernetes API method, application operators can resume syncing by reapplying repo-sync.yaml which contains the RepoSync configuration:

kubectl apply -f repo-sync.yaml

This command triggers tne Reconciler Manager to create a namespace reconciler process and create a ns-reconciler-NAMESPACE Deployment.

Removing the root and namespace repositories

To remove the root repository, delete the RootSync object. For example,

kubectl delete -f root-sync.yaml

To uninstall a namespace repository, delete a RepoSync object. This action uninstalls the matching namespace repository.

Ignoring object mutations

In Config Sync versions 1.7 and later, if you don't want Config Sync to maintain the state of the object an the cluster after it exists, you can add the client.lifecycle.config.k8s.io/mutation: ignore annotation to the object that you want Config Sync to ignore mutations in. This action is useful when you want Config Sync to create an object and then ignore any conflicting changes to this object in the cluster.

The following example shows you how to add the annotation to an object:

metadata:
  annotations:
    client.lifecycle.config.k8s.io/mutation: ignore

You cannot manually modify this annotation on managed objects in the cluster.

Conflict resolution

When you are working with two repositories, conflicts might arise when the same object (matching groups, kinds, names, and namespaces) is declared both in the root and a namespace repository. When this conflict happens, only the declaration in the root repository is applied to the cluster.

If this conflict happens, the RootSync won't report any problems since the root repository takes precedence. The RepoSync reports error KNV 1060 in its status.

Since the root repository always takes precedence, you can resolve the conflict by updating the root repository to match the namespace repository, or by deleting the conflicting object in the namespace repository.

Exploring the RootSync and RepoSync objects

The following sections show you different ways that you can explore your RootSync and RepoSync objects.

Viewing logs

For further information on potential errors, you can view the logs for the RootSync and RepoSync objects.

To view the logs for the RootSync reconciler, run the following command:

kubectl logs -n config-management-system deployment/root-reconciler CONTAINER_NAME

Replace CONTAINER_NAME with either git-sync or reconciler. The git-sync container pulls configs from a Git repository to a local directory that the reconciler container can read from. The reconciler container applies those configs to the cluster.

To view the logs of the RepoSync reconciler, run the following command:

kubectl logs -n config-management-system deployment/ns-reconciler-NAMESPACE CONTAINER_NAME

Replace the following:

  • NAMESPACE: the namespace that you created your namespace repository in.
  • CONTAINER_NAME: either git-sync or reconciler.

Viewing synced commits

You can check which commit is synced to the cluster.

To view details for the RootSync, run the following command:

kubectl get rootsync root-sync -n config-management-system

To view details for the RepoSync, run the following command:

kubectl get reposync repo-sync -n config-management-system

You should see output similar to the following example:

NAME        SOURCECOMMIT                               SYNCCOMMIT
root-sync   66882815f0ef5517df27e864fb1315e97756ab72   66882815f0ef5517df27e864fb1315e97756ab72

The value in the SOURCECOMMIT column is the commit from the Git repository that should be synced to the cluster. The value in the SYNCCOMMIT column is the commit that is currently deployed to the cluster. If the two values in the SOURCECOMMIT and SYNCCOMMIT columns are the same, the expected commit has been deployed to the cluster.

Viewing object details

To view details of the RootSync and RepoSync objects, use kubectl describe. This command can provide you with further information about potential errors.

To view details for the RootSync object, run the following command:

kubectl describe rootsync root-sync -n config-management-system

To view details for the RepoSync object, run the following command:

kubectl describe reposync repo-sync -n config-management-system

Viewing if a resource is ready

To learn if the resources synced to the cluster are ready, view the reconciliation status. For example, this command can show you if a synced Deployment is ready to serve traffic.

For a Git repository synced to the cluster, the reconciliation status of all resources are aggregated in a resource called ResourceGroup. For each RootSync or RepoSync object, a ResourceGroup is generated to capture the set of resources applied to the cluster and aggregate their statuses.

To view the reconciliation status for the RootSync object, run the following command:

kubectl get resourcegroup.kpt.dev root-sync -n config-management-system -o yaml

To view the reconciliation status for the RepoSync object, run the following command:

kubectl get resourcegroup.kpt.dev repo-sync -n NAMESPACE -o yaml

In the output, you see all of the ResourceGroup resource statuses. For example, the following output shows that a Deployment named nginx-deployment is ready:

resourceStatuses:
- group: apps
  kind: Deployment
  name: nginx-deployment
  namespace: default
  status: Current

Troubleshooting

The following sections help you troubleshoot problems when you are syncing to multiple repositories.

Error: admission webhook denied a request

If you receive the following error when you try to apply a change to a field that Config Sync manages, then you might have have made a conflicting change:

error: OBJECT could not be patched: admission webhook "v1.admission-webhook.configsync.gke.io"
denied the request: fields managed by Config Sync can not be modified

When you declare a field in a config and your repository is synced to a cluster, Config Sync manages that field. Any change you attempt to make to that field is a conflicting change.

For example, if you have a Deployment config in your repository with a label of environment:prod and you try to change that label to environment:dev in your cluster, there would be a conflicting change and you would receive the preceding error message. However, if you add a new label (for example, tier:frontend) to the Deployment, there would not be a conflict.

If you want Config Sync to ignore any changes to an object, you can add the annotation described in Ignoring object mutations.

Using nomos status to display errors

You can use the nomos status command to print errors from all of your clusters on the command line.

ResourceGroup fields keep changing

Occasionally, your ResourceGroup can enter a loop that keeps updating the ResourceGroup's spec. If this happens, you might notice the following issues:

  • The metadata.generation of a ResourceGroup keeps increasing in a short period of time.
  • The ResourceGroup spec keeps changing.
  • The ResourceGroup spec doesn't include the status.resourceStatuses of the resources being synced to the cluster.

If you observe these issues, it means that some of the resources in your Git repositories failed to be applied to the cluster. The cause of these issues is that you are missing the permissions that you need to apply those resources.

You can verify that the permissions are missing by getting the RepoSync resource status:

kubectl get reposync repo-sync -n NAMESPACE -o yaml

Replace NAMESPACE with the namespace that you created your namespace repository in.

If you see the following messages in the status, it means that the reconciler in NAMESPACE lacks the permission needed to apply the resource:

errors:
  - code: "2009"
    errorMessage: |-
      KNV2009: deployments.apps "nginx-deployment" is forbidden: User "system:serviceaccount:config-management-system:ns-reconciler-     default" cannot get resource "deployments" in API group "apps" in the namespace "default"

      For more information, see https://g.co/cloud/acm-errors#knv2009

To fix this issue, you need to declare a RoleBinding configuration that grants the ns-reconciler-NAMESPACE service account permission to manage the failed resource in that namespace. Details on how to add a RoleBinding are included in the Configuring syncing from namespace repositories section.

Large number of resources in Git repository

When the Git repository synced by either the RepoSync or RootSync objects contains configuration for more than a few thousand resources, it can cause the ResourceGroup to exceed the etcd object size limit. When this happens, you cannot view the aggregated status for resources in your Git repository. While you won't be able to view the aggregated status, your repository is still synced. If you don't see any errors in the RepoSync or RootSync objects, that means your Git repository is synced to the cluster successfully.

To check if the ResourceGroup resource exceeds the etcd object size limit, you need to check both the ResourceGroup resource status and the log of the ResourceGroup controller:

  1. Check the ResourceGroup status with the following command:

    • To check the RootSync, run the following command:
     kubectl get resourcegroup.kpt.dev root-sync -n config-management-system
    • To check the RepoSync, run the following command:
    # For the RepoSync:
kubectl get resourcegroup.kpt.dev repo-sync -n NAMESPACE

    You should see output similar to the following example:

    NAME        RECONCILING   STALLED   AGE
root-sync   True          False     35m

    If the value in the RECONCILING column is True, it means that the ResourceGroup resource is still reconciling.

  2. Check the logs for the ResourceGroup controller with the following command:

    kubectl logs deployment/resource-group-controller-manager -c manager -n resource-group-system

    If you see the following error in the output, the ResourceGroup resource is too large and exceeds the etcd object size limit:

    "error":"etcdserver: request is too large"

To prevent the ResourceGroup from getting too large, reduce the number of resources in your Git repository. For example, split the repository and use one RootSync object with multiple RepoSync objects instead of using only one RootSync object to sync all the resources.

Known issue in 1.6.2: unable to sync Git repository using ssh, cookiefile, or token

There is a known issue for multi-repo mode in 1.6.2. The security context for configuring the ownership of volumes is missing from one object in the operator manifest. Since the context is missing, the reconcilers are unable to sync Git repositories when ssh, cookiefile, or token is the authentication method.

To verify your failure is caused by this issue, check the corresponding Deployment object, either root-reconciler or ns-reconciler-NAMESPACE.

  kubectl get deployment root-reconciler -n config-management-system -o yaml
  # or
  kubectl get deployment ns-reconciler-NAMESPACE -n config-management-system -o yaml

Replace NAMESPACE with the name of your namespace.

In the output, if you don't see the following fields, the failure is caused by the missing configuration:

    securityContext:
      fsGroup: 65533

To fix the issue, you need to be granted the permission for updating Deployment objects in the config-management-system namespace.

To fix this issue for one reconciler, you need to add the missing fields in the corresponding Deployment object. For example, here are the steps to add the missing fields to the root-reconciler object:

  1. Get the root-reconciler object.

    kubectl get deployment root-reconciler -n config-management-system -o yaml > deployment.yaml

  2. Add the missing configuration under .spec.template.spec in deployment.yaml.

     securityContext:
   fsGroup: 65533

  3. Apply the Deployment object.

    kubectl apply -f deployment.yaml

To fix this issue for the entire cluster, you need to stop the Config Sync Operator, re-run it with a patch for the missing configuration. Here are the steps:

  1. Create the configuration for ConfigManagement object with patches for the missing configuration. The patch should be for the ConfigMap reconciler- manager-cm in the namespace config-management-system, which is the object in the Config Sync Operator that misses the configuration. The patch content should be put into the ConfigManagement object under the field .spec.patches. Here is an example for a ConfigManagement object with the correct patch for the ConfigMap. You can copy the .spec.patches from this example and paste it to your ConfigManagement configuration file.

    # file config-management.yaml
apiVersion: configmanagement.gke.io/v1
kind: ConfigManagement
metadata:
  name: config-management
spec:
  enableMultiRepo: true 
  patches:
  - apiVersion: v1
    kind: ConfigMap
    metadata:
      name: reconciler-manager-cm
      namespace: config-management-system
      labels:
        configmanagement.gke.io/system: "true"
        configmanagement.gke.io/arch: "csmr"
    data:
      deployment.yaml: |
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: reconciler
          namespace: config-management-system
          labels:
            app: reconciler
            configmanagement.gke.io/system: "true"
            configmanagement.gke.io/arch: "csmr"
        spec:
          minReadySeconds: 10
          replicas: 1
          strategy:
            type: Recreate
          selector:
            matchLabels:
              app: reconciler
          template:
            metadata:
              labels:
                app: reconciler
            spec:
              serviceAccountName: reconciler
              containers:
              - name: reconciler
                image: gcr.io/config-management-release/reconciler:v1.6.2-rc.2
                command:
                - /reconciler
                args:
                - "--v=0"
                - "--git-dir=/repo/root/rev"
                volumeMounts:
                  - name: repo
                    mountPath: /repo
                    readOnly: true
                envFrom:
                  - configMapRef:
                      name: reconciler
                  - configMapRef:
                      name: source-format
                      optional: true
                resources:
                  requests:
                    cpu: "150m"
                    memory: "100Mi"
              - name: git-sync
                image: gcr.io/config-management-release/git-sync:v3.1.6-gke.1__linux_amd64
                args: ["--root=/repo/root", "--dest=rev", "--max-sync-failures=30", "--v=5"]
                volumeMounts:
                - name: repo
                  mountPath: /repo
                - name: git-creds
                  mountPath: /etc/git-secret
                  readOnly: true
                securityContext:
                  runAsUser: 65533
                envFrom:
                - configMapRef:
                    name: git-sync
              volumes:
              - name: repo
              - name: git-creds
                secret:
                  secretName: git-creds
                  defaultMode: 288
              securityContext:
                fsGroup: 65533

  2. Reduce the replicas count in the reconciler-manager Deployment in Config Sync Operator to 0.

    kubectl -n config-management-system scale deployment reconciler-manager --replicas=0

    After this command, the existing Pod for Deployment reconciler-manager is terminated. The Config Sync Operator changes the replicas count back to one. A new Pod for this Deployment is created. So you don't need to manually change replicas back to 1. The new Pod uses the ConfigMap that has been patched.

What's next