Syncing from multiple repositories

Enabling multi-repo mode on Config Sync allows you to sync configuration from multiple repositories to the same set of clusters. You can utilize the following types of repositories:

  • Root repository: This repository allows you to sync cluster-scoped and namespace-scoped configs. The root repository uses admin-level credentials to enforce policies on application namespaces and override local changes that drift from the desired state. A central administrator governs the root repository. Only one root repository can exist for each cluster.

  • Namespace repositories: Namespace repositories are optional and can contain namespace-scoped configs synced to a particular namespace across clusters. You can delegate the setup and control of a namespace repository to non-administrative users.

Each repository has a single Git reference (a repository branch, commit or tag, and directory tuple) that you can manage separately. This configuration decouples the config deployment lifecycle for different teams. It also provides you with more autonomy in choosing where you want to house the repository and how to structure it.

The following diagram shows you an overview of how teams might use a root repository and namespace repositories:

A central administrator controlling multiple configs and app operators
controlling their own namespace configs.

In this diagram, a central administrator manages the centralized infrastructure for the organization and enforces policies on the cluster and on all namespaces in the organization.

The application operators, who are responsible for managing live deployments, apply configurations to the applications in the namespaces that they work on.

Before you begin

Complete the following tasks before enabling multi-repo mode:

  • Use Config Sync version 1.5.1 or later.

Limitations

Consider the following limitations when you use multi-repo mode:

  • Thespec.enableMultiRepo field is not compatible with spec.git field. If you still need to use spec.git, see the Using legacy spec.git fields section.
  • ClusterSelectors and namespaceSelectors (including annotations pointing to Selectors) only work in the root repository.
  • You cannot use gcenode in the auth field of your RootSync or RepoSync configurations.

Configuring syncing from the root repository

To configure the root repository, complete the following tasks:

  1. Open your config-management.yaml file.
  2. Set the spec.enableMultiRepo field to true:

    # config-management.yaml
    apiVersion: configmanagement.gke.io/v1
    kind: ConfigManagement
    metadata:
      name: config-management
    spec:
      enableMultiRepo: true
      # ...other fields...
    
  3. Apply the changes:

    kubectl apply -f config-management.yaml

  4. To define specific features for the root repository, create the root-sync.yaml file. For example:

    # root-sync.yaml
    apiVersion: configsync.gke.io/v1alpha1
    kind: RootSync
    metadata:
      name: root-sync
      namespace: config-management-system
    spec:
      git:
        repo: REPOSITORY
        revision: REVISION
        branch: BRANCH
        dir: "DIRECTORY"
        auth: AUTH_TYPE
        secretRef:
          name: SECRET_NAME
    

    Replace the following:

    • REPOSITORY: add the URL of the Git repository to use as the root repository. This field is required.
    • REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
    • BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
    • DIRECTORY: add the path within the Git repository that represents the top level of the repository to sync to. This field is optional and the default is the root directory of the repository.
    • AUTH_TYPE: add one of the following authentication types:

      • none
      • ssh
      • cookiefile
      • token

      This field is required.

    • SECRET_NAME: add the name you intend to give your Secret. This field is required.

  5. Create a Secret based on your preferred authentication method.

    The Secret must meet following requirements:

    • The Secret's name must match the spec.git.secretRef name you defined in repo-sync.yaml.
    • You must add the Secret's public key to the Git provider.
  6. Apply the configuration:

    kubectl apply -f root-sync.yaml
    
  7. Verify the installation:

    kubectl -n config-management-system get pods | grep reconciler-manager
    

    You should see output similar to the following example:

    reconciler-manager-6f988f5fdd-4r7tr 1/1 Running 0 26s
    

Configuring syncing from namespace repositories

There are two options for configuring namespace repositories:

  • Control namespace repositories in the root repository. This method centralizes all configuration of namespace repositories in the root repository, allowing a central administrator complete control of the setup.

  • Control namespace repositories with the Kubernetes API. This method delegates control of the namespace repository to the namespace owners.

Controlling namespace repositories in the root repository

In this method, the central administrator manages the setup of namespace repositories directly from the root repository. Because Config Sync manages the namespace repository resources, this method prevents any local changes to the namespace repository definitions.

To use this method, complete the following tasks:

  1. In the root repository, declare a namespace configuration:

    # ROOT_REPO/namespaces/NAMESPACE/namespace.yaml
    apiVersion: v1
    kind: Namespace
    metadata:
      name: NAMESPACE
    

    Replace NAMESPACE with a name for your namespace.

  2. In the root repository, declare a RepoSync configuration in the same namespace:

    # ROOT_REPO/namespaces/NAMESPACE/repo-sync.yaml
    apiVersion: configsync.gke.io/v1alpha1
    kind: RepoSync
    metadata:
      name: repo-sync
      namespace: NAMESPACE
    spec:
      git:
       repo: REPOSITORY
       revision: REVISION
       branch: BRANCH
       dir: "DIRECTORY"
       auth: AUTH_TYPE
       secretRef:
         name: SECRET_NAME
    

    Replace the following:

    • NAMESPACE: add the name of your namespace.
    • REPOSITORY: add the URL of the Git repository to use as the root repository. This field is required.
    • REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
    • BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
    • DIRECTORY: add the path within the Git repository that represents the top level of the repository to sync to. This field is optional and the default is the root directory of the repository.
    • AUTH_TYPE: add one of the following authentication types:

      • none
      • ssh
      • cookiefile
      • token

      This field is required.

    • SECRET_NAME: add the name you intend to give your Secret. This field is required.

    There can be at most one RepoSync resource per namespace. To enforce this, the object name must be repo-sync. All configs contained in the directory referenced by the RepoSync must also be in the same namespace as the RepoSync resource.

  3. In the root repository, declare a RoleBinding configuration that grants the ns-reconciler-NAMESPACE service account permission to manage resources in the namespace. Config Sync automatically creates the ns-reconciler-NAMESPACE service account when the RepoSync config is synced to the cluster.

    To declare the RoleBinding, create the following manifest:

    # ROOT_REPO/namespaces/NAMESPACE/sync-rolebinding.yaml
     kind: RoleBinding
     apiVersion: rbac.authorization.k8s.io/v1
     metadata:
       name: syncs-repo
       namespace: NAMESPACE
     subjects:
     - kind: ServiceAccount
       name: ns-reconciler-NAMESPACE
       namespace: config-management-system
     roleRef:
       kind: ClusterRole
       name: RECONCILER_ROLE
       apiGroup: rbac.authorization.k8s.io
    

    Replace the following:

    • NAMESPACE: add the name of your namespace.
    • RECONCILER_ROLE: as the central administrator, you can set the RECONCILER_ROLE to enforce what kinds of configuration can be synced from the namespace repository. You can choose one of the following roles:

      • A default ClusterRole:

        • admin
        • edit

        To learn more, see User-facing roles.

      • A user-defined ClusterRole or Role declared in the root repository. This role allows for fine grained permissions.

  4. Commit the previous changes to the root repository:

     git add .
     git commit -m 'Setting up new namespace repository.'
     git push
    
  5. Create a Secret based on your preferred authentication method.

    The Secret must meet following requirements:

    • Create the Secret in the same namespace as the RepoSync.
    • The Secret's name must match the spec.git.secretRef name you defined in repo-sync.yaml.
    • You must add the Secret's public key to the Git provider.
  6. To verify the configuration, use kubectl get on one of the objects in the namespace repository. For example:

    kubectl get rolebindings -n NAMESPACE
    

Controlling namespace repositories with the Kubernetes API

In this method, the central administrator only declares the namespace in the root repository and delegates declaration of the RepoSyncfile to the application operator.

Central administrator tasks

The central administrator completes the following tasks:

  1. In the root repository, declare a namespace configuration:

    # ROOT_REPO/namespaces/NAMESPACE/namespace.yaml
     apiVersion: v1
     kind: Namespace
     metadata:
       name: NAMESPACE
    

    Replace NAMESPACE with a name for your namespace.

  2. In the root repository, declare a RoleBinding configuration to give the application operators permissions. Use RBAC escalation prevention to ensure that the application operator cannot later apply a role binding with permissions not granted by this role binding.

    To declare the RoleBinding, create the following manifest:

    # ROOT_REPO/namespaces/NAMESPACE/operator-rolebinding.yaml
     kind: RoleBinding
     # Add RBAC escalation prevention
     apiVersion: rbac.authorization.k8s.io/v1
     metadata:
       name: operator
       namespace: NAMESPACE
     subjects:
     - kind: User
       name: USER_NAME
       apiGroup: rbac.authorization.k8s.io
     roleRef:
       kind: ClusterRole
       name: OPERATOR_ROLE
       apiGroup: rbac.authorization.k8s.io
    

    Replace the following:

    • NAMESPACE: add the namespace you created in the root repository.
    • USER_NAME: add the username of the application operator.
    • OPERATOR_ROLE: as the central administrator, you can set OPERATOR_ROLE to enforce what kinds of configurations can be synced from the namespace repository. You can choose one of the following roles:

      • A default ClusterRole:

        • admin
        • edit

        To learn more, see User-facing roles.

      • A user-defined ClusterRole or Role declared in the root repository. This allows for fine grained permissions.

  3. Commit the previous changes to the root repository:

     git add .
     git commit -m 'Setting up new namespace repository.'
     git push
    

Application operator tasks

The application operator completes the following tasks:

  1. Declare a RoleBinding configuration that grants the auto-provisioned ns-reconciler-NAMESPACE service account permission to manage resources in the namespace. Config Sync automatically creates the ns-reconciler-NAMESPACE service account when the RepoSync config is synced to the cluster.

    To declare the RoleBinding, create the following manifest:

    # sync-rolebinding.yaml
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: syncs-repo
      namespace: NAMESPACE
    subjects:
    - kind: ServiceAccount
      name: ns-reconciler-NAMESPACE
      namespace: config-management-system
    roleRef:
      kind: ClusterRole
      name: RECONCILER_ROLE
      apiGroup: rbac.authorization.k8s.io
    

    Replace the following:

    • NAMESPACE: add the namespace you created in the root repository.
    • RECONCILER_ROLE: as the application operator you can set RECONCILER_ROLE to enforce what kinds of configuration can be synced from the namespace repository. You can only further restrict the set of permissions the central administrator has granted you. As a result, this role cannot be more permissive than the OPERATOR_ROLE that the central administrator declared in the previous section.
  2. Apply the RoleBinding configuration:

    kubectl apply -f sync-rolebinding.yaml
    
  3. Create a Secret based on your preferred authentication method.

    The Secret must meet the following requirements:

    • Create the Secret in the same namespace as the RepoSync.
    • The Secret's name must match the spec.git.secretRef name you defined in repo-sync.yaml.
    • You must add the Secret's public key to the Git provider.
  4. Declare a RepoSync configuration:

    # repo-sync.yaml
    apiVersion: configsync.gke.io/v1alpha1
    kind: RepoSync
    metadata:
      name: repo-sync
      namespace: NAMESPACE
    spec:
      git:
       repo: REPOSITORY
       revision: REVISION
       branch: BRANCH
       dir: "DIRECTORY"
       auth: AUTH_TYPE
       secretRef:
         name: SECRET_NAME
    

    Replace the following:

    • REPOSITORY: add the URL of the Git repository to use as the root repository. This field is required.
    • REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
    • BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
    • DIRECTORY: add the path within the Git repository that represents the top level of the repository to sync to. This field is optional and the default is the root directory of the repository.
    • AUTH_TYPE: add one of the following authentication types:

      • none
      • ssh
      • cookiefile
      • token

      This field is required.

    • SECRET_NAME: add the name you intend to give your Secret. This field is required.

    There can be at most one RepoSync resource per namespace. To enforce this, the object's name must be repo-sync. All configs contained in the directory referenced by the RepoSync must also be in the same namespace as the RepoSync resource.

  5. Apply the RepoSync configuration:

    kubectl apply -f repo-sync.yaml
    
  6. To verify the configuration, use kubectl get on one of the objects in the namespace repository. For example:

    kubectl get rolebindings -n NAMESPACE
    

Using legacy spec.git fields

To make migration to RootSync resources easier, set the spec.enableLegacyFields to true in your config-management.yaml file. Adding this field lets you use the deprecated spec.git fields in the ConfigManagement resource. Setting this field also automatically generates the RootSync resource on the cluster. Do not change the RootSync configuration.

The following example shows you how to use the spec.git field:


# config-management.yaml

apiVersion: configmanagement.gke.io/v1
kind: ConfigManagement
metadata:
  name: config-management
spec:
  enableMultiRepo: true
  enableLegacyFields: true
  git:
    syncRepo: REPO
    syncBranch: BRANCH
    secretType: TYPE
    policyDir: "DIRECTORY"
  # ...other fields...

To learn more about these fields, see config-management.yaml fields.

Stopping and resuming syncing

This section shows you how to temporarily stop and resume syncing. You might need to do this if an incorrect config is accidentally committed to your repository.

Only a central administrator can stop syncing in the root repository.

The ability to stop syncing in namespace repositories depends on which configuration method was used for your namespace repositories.

  • If the Control namespace repositories in the root repository method was used, a central administrator is the only one who can stop and resume syncing.

  • If the Control namespace repositories with the Kubernetes API method was used, application operators can stop and resume syncing from the namespace repositories that they work on.

Stopping syncing

This section shows you how to stop syncing for the root repository and for namespace repositories.

Stopping syncing from the root repository

To stop syncing from the root repository, a central administrator can run the following command:

kubectl -n config-management-system scale deployment root-reconciler --replicas=0

This command reduces the replicas count in the root-reconciler Deployment to 0.

Stopping syncing from namespace repositories

Select the Root repository method or Kubernetes API method tab to view the relevant instructions.

Root repository method

If the Control namespace repositories in the root repository method was used, central administrators can run the following commands to stop syncing from a namespace repository:

kubectl -n config-management-system scale deployment ns-reconciler-NAMESPACE --replicas=0

The command reduces the replica count in the ns-reconciler-NAMESPACE Deployment to 0.

Kubernetes API method

If the Control namespace repositories with the Kubernetes API method was used, application operators can stop syncing by running the following commands:

  1. Retrieve the RepoSync configuration and save it to use it later when you want to resume syncing:

    kubectl -n NAMESPACE get reposyncs repo-sync -oyaml > repo-sync.yaml
    

    replace NAMESPACE with the namespace of your RepoSync.

  2. Delete the RepoSync configuration:

    kubectl -n NAMESPACE delete reposyncs repo-sync
    

    This command triggers the Reconciler Manager to remove the namespace reconciler (ns-reconciler-NAMESPACE) from NAMESPACE and stops syncing.

Resuming syncing

This section shows you how to resume syncing for the root repository and for namespace repositories.

Resuming syncing from the root repository

To resume syncing from a root repository, a central administrator can run the following command:

kubectl -n config-management-system scale deployment root-reconciler --replicas=1

This command scales the root-reconciler Deployment to 1 replica.

Resuming syncing from a namespace repository

Select the Root repository method or Kubernetes API method tab to view the relevant instructions.

Root repository method

If you used the Control namespace repositories in the root repository method, a central administrator can run the following command:

kubectl -n config-management-system scale deployment ns-reconciler-NAMESPACE --replicas=1

This command scales the ns-reconciler-NAMESPACE Deployment to 1 replica.

Kubernetes API method

If you used the Control namespace repositories with the Kubernetes API method, application operators can resume syncing by reapplying repo-sync.yaml which contains the RepoSync configuration:

kubectl apply -f repo-sync.yaml

This command triggers tne Reconciler Manager to create a namespace reconciler process and create a ns-reconciler-NAMESPACE Deployment.

Removing the root and namespace repositories

To remove the root repository, delete the RootSync file. For example,

kubectl delete -f root-sync.yaml

To uninstall a namespace repository, delete a RepoSync file. This action uninstalls the matching namespace repository.

Conflict resolution

When you are working with two repositories, conflicts might arise when the same resource (matching groups, kinds, names, and namespaces) is declared both in the root and the namespace repository. When this happens, only the declaration in the root repository is applied to the cluster.

Since the root repository always takes precedence, you can resolve the conflict by updating the root repository to match the namespace repository, or by deleting the conflicting object in the namespace repository.

What's next