Syncing from multiple repositories

Preview

This feature is covered by the Pre-GA Offerings Terms of the Google Cloud Platform Terms of Service. Pre-GA features may have limited support, and changes to pre-GA features may not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

Enabling multi-repo mode on Config Sync allows you to sync configuration from multiple repositories to the same set of clusters. You can utilize the following types of repositories:

• Root repository: This repository allows you to sync cluster-scoped and namespace-scoped configs. The root repository uses admin-level credentials to enforce policies on application namespaces and override local changes that drift from the desired state. A central administrator governs the root repository. Only one root repository can exist for each cluster.

• Namespace repositories: Namespace repositories are optional and can contain namespace-scoped configs synced to a particular namespace across clusters. You can delegate the setup and control of a namespace repository to non-administrative users.

Each repository has a single Git reference (a repository branch, commit or tag, and directory tuple) that you can manage separately. This configuration decouples the config deployment lifecycle for different teams. It also provides you with more autonomy in choosing where you want to house the repository and how to structure it.

The following diagram shows you an overview of how teams might use a root repository and namespace repositories:

In this diagram, a central administrator manages the centralized infrastructure for the organization and enforces policies on the cluster and on all namespaces in the organization.

The application operators, who are responsible for managing live deployments, apply configurations to the applications in the namespaces that they work on.

Before you begin

Complete the following tasks before enabling multi-repo mode:

• Deploy the Config Sync Operator.

• Use Config Sync version 1.5.1 or later.

Limitations

Consider the following limitations when you use multi-repo mode:

• Thespec.enableMultiRepo field is not compatible with spec.git field. If you still need to use spec.git, see the Using legacy spec.git fields section.
• ClusterSelectors and namespaceSelectors (including annotations pointing to Selectors) only work in the root repository.
• You cannot use gcenode in the auth field of your RootSync or RepoSync configurations.

Configuring syncing from the root repository

To configure the root repository, complete the following tasks:

1. Open your config-management.yaml file.

2. Set the spec.enableMultiRepo field to true:

   # config-management.yaml
   apiVersion: configmanagement.gke.io/v1
   kind: ConfigManagement
   metadata:
     name: config-management
   spec:
     enableMultiRepo: true
     # ...other fields...
   

3. Apply the changes:

   kubectl apply -f config-management.yaml

4. To define specific features for the root repository, create the root-sync.yaml file. For example:

   # root-sync.yaml
   apiVersion: configsync.gke.io/v1alpha1
   kind: RootSync
   metadata:
     name: root-sync
     namespace: config-management-system
   spec:
     git:
       repo: REPOSITORY
       revision: REVISION
       branch: BRANCH
       dir: "DIRECTORY"
       auth: AUTH_TYPE
       secretRef:
         name: SECRET_NAME
   

   Replace the following:

   • REPOSITORY: add the URL of the Git repository to use as the root repository. This field is required.
   • REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
   • BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
   • DIRECTORY: add the path within the Git repository that represents the top level of the repository to sync to. This field is optional and the default is the root directory of the repository.

   • AUTH_TYPE: add one of the following authentication types:

     • none
     • ssh
     • cookiefile
     • token

     This field is required.

   • SECRET_NAME: add the name you intend to give your Secret. This field is required.

   Note: You can only create one root repository per cluster.

5. Create a Secret based on your preferred authentication method.

   The Secret must meet following requirements:

   • The Secret's name must match the spec.git.secretRef name you defined in repo-sync.yaml.
   • You must add the Secret's public key to the Git provider.

6. Apply the configuration:

   kubectl apply -f root-sync.yaml
   

7. Verify the installation:

   kubectl -n config-management-system get pods | grep reconciler-manager
   

   You should see output similar to the following example:

   reconciler-manager-6f988f5fdd-4r7tr 1/1 Running 0 26s
   

Configuring syncing from namespace repositories

There are two options for configuring namespace repositories:

• Control namespace repositories in the root repository. This method centralizes all configuration of namespace repositories in the root repository, allowing a central administrator complete control of the setup.

• Control namespace repositories with the Kubernetes API. This method delegates control of the namespace repository to the namespace owners.

Controlling namespace repositories in the root repository

In this method, the central administrator manages the setup of namespace repositories directly from the root repository. Because Config Sync manages the namespace repository resources, this method prevents any local changes to the namespace repository definitions.

To use this method, complete the following tasks:

1. In the root repository, declare a namespace configuration:

   # ROOT_REPO/namespaces/NAMESPACE/namespace.yaml
   apiVersion: v1
   kind: Namespace
   metadata:
     name: NAMESPACE
   

   Replace NAMESPACE with a name for your namespace.

2. In the root repository, declare a RepoSync configuration in the same namespace:

   # ROOT_REPO/namespaces/NAMESPACE/repo-sync.yaml
   apiVersion: configsync.gke.io/v1alpha1
   kind: RepoSync
   metadata:
     name: repo-sync
     namespace: NAMESPACE
   spec:
     git:
      repo: REPOSITORY
      revision: REVISION
      branch: BRANCH
      dir: "DIRECTORY"
      auth: AUTH_TYPE
      secretRef:
        name: SECRET_NAME
   

   Replace the following:

   • NAMESPACE: add the name of your namespace.
   • REPOSITORY: add the URL of the Git repository to use as the root repository. This field is required.
   • REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
   • BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
   • DIRECTORY: add the path within the Git repository that represents the top level of the repository to sync to. This field is optional and the default is the root directory of the repository.

   • AUTH_TYPE: add one of the following authentication types:

     • none
     • ssh
     • cookiefile
     • token

     This field is required.

   • SECRET_NAME: add the name you intend to give your Secret. This field is required.

   There can be at most one RepoSync resource per namespace. To enforce this, the object name must be repo-sync. All configs contained in the directory referenced by the RepoSync must also be in the same namespace as the RepoSync resource.

3. In the root repository, declare a RoleBinding configuration that grants the ns-reconciler-NAMESPACE service account permission to manage resources in the namespace. Config Sync automatically creates the ns-reconciler-NAMESPACE service account when the RepoSync config is synced to the cluster.

   To declare the RoleBinding, create the following manifest:

   # ROOT_REPO/namespaces/NAMESPACE/sync-rolebinding.yaml
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: syncs-repo
      namespace: NAMESPACE
    subjects:
    - kind: ServiceAccount
      name: ns-reconciler-NAMESPACE
      namespace: config-management-system
    roleRef:
      kind: ClusterRole
      name: RECONCILER_ROLE
      apiGroup: rbac.authorization.k8s.io
   

   Replace the following:

   • NAMESPACE: add the name of your namespace.

   • RECONCILER_ROLE: as the central administrator, you can set the RECONCILER_ROLE to enforce what kinds of configuration can be synced from the namespace repository. You can choose one of the following roles:

     • A default ClusterRole:

       • admin
       • edit

       To learn more, see User-facing roles.

     • A user-defined ClusterRole or Role declared in the root repository. This role allows for fine grained permissions.

4. Commit the previous changes to the root repository:

    git add .
    git commit -m 'Setting up new namespace repository.'
    git push
   

5. Create a Secret based on your preferred authentication method.

   The Secret must meet following requirements:

   • Create the Secret in the same namespace as the RepoSync.
   • The Secret's name must match the spec.git.secretRef name you defined in repo-sync.yaml.
   • You must add the Secret's public key to the Git provider.

6. To verify the configuration, use kubectl get on one of the objects in the namespace repository. For example:

   kubectl get rolebindings -n NAMESPACE
   

Controlling namespace repositories with the Kubernetes API

In this method, the central administrator only declares the namespace in the root repository and delegates declaration of the RepoSyncfile to the application operator.

Central administrator tasks

The central administrator completes the following tasks:

1. In the root repository, declare a namespace configuration:

   # ROOT_REPO/namespaces/NAMESPACE/namespace.yaml
    apiVersion: v1
    kind: Namespace
    metadata:
      name: NAMESPACE
   

   Replace NAMESPACE with a name for your namespace.

2. In the root repository, declare a RoleBinding configuration to give the application operators permissions. Use RBAC escalation prevention to ensure that the application operator cannot later apply a role binding with permissions not granted by this role binding.

   To declare the RoleBinding, create the following manifest:

   # ROOT_REPO/namespaces/NAMESPACE/operator-rolebinding.yaml
    kind: RoleBinding
    # Add RBAC escalation prevention
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: operator
      namespace: NAMESPACE
    subjects:
    - kind: User
      name: USER_NAME
      apiGroup: rbac.authorization.k8s.io
    roleRef:
      kind: ClusterRole
      name: OPERATOR_ROLE
      apiGroup: rbac.authorization.k8s.io
   

   Replace the following:

   • NAMESPACE: add the namespace you created in the root repository.
   • USER_NAME: add the username of the application operator.

   • OPERATOR_ROLE: as the central administrator, you can set OPERATOR_ROLE to enforce what kinds of configurations can be synced from the namespace repository. You can choose one of the following roles:

     • A default ClusterRole:

       • admin
       • edit

       To learn more, see User-facing roles.

     • A user-defined ClusterRole or Role declared in the root repository. This allows for fine grained permissions.

3. Commit the previous changes to the root repository:

    git add .
    git commit -m 'Setting up new namespace repository.'
    git push
   

Application operator tasks

The application operator completes the following tasks:

1. Declare a RoleBinding configuration that grants the auto-provisioned ns-reconciler-NAMESPACE service account permission to manage resources in the namespace. Config Sync automatically creates the ns-reconciler-NAMESPACE service account when the RepoSync config is synced to the cluster.

   To declare the RoleBinding, create the following manifest:

   # sync-rolebinding.yaml
   kind: RoleBinding
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     name: syncs-repo
     namespace: NAMESPACE
   subjects:
   - kind: ServiceAccount
     name: ns-reconciler-NAMESPACE
     namespace: config-management-system
   roleRef:
     kind: ClusterRole
     name: RECONCILER_ROLE
     apiGroup: rbac.authorization.k8s.io
   

   Replace the following:

   • NAMESPACE: add the namespace you created in the root repository.
   • RECONCILER_ROLE: as the application operator you can set RECONCILER_ROLE to enforce what kinds of configuration can be synced from the namespace repository. You can only further restrict the set of permissions the central administrator has granted you. As a result, this role cannot be more permissive than the OPERATOR_ROLE that the central administrator declared in the previous section.

2. Apply the RoleBinding configuration:

   kubectl apply -f sync-rolebinding.yaml
   

3. Create a Secret based on your preferred authentication method.

   The Secret must meet the following requirements:

   • Create the Secret in the same namespace as the RepoSync.
   • The Secret's name must match the spec.git.secretRef name you defined in repo-sync.yaml.
   • You must add the Secret's public key to the Git provider.

4. Declare a RepoSync configuration:

   # repo-sync.yaml
   apiVersion: configsync.gke.io/v1alpha1
   kind: RepoSync
   metadata:
     name: repo-sync
     namespace: NAMESPACE
   spec:
     git:
      repo: REPOSITORY
      revision: REVISION
      branch: BRANCH
      dir: "DIRECTORY"
      auth: AUTH_TYPE
      secretRef:
        name: SECRET_NAME
   

   Replace the following:

   • REPOSITORY: add the URL of the Git repository to use as the root repository. This field is required.
   • REVISION: add the Git revision (tag or hash) to check out. This field is optional and the default value is HEAD.
   • BRANCH: add the branch of the repository to sync from. This field is optional and the default value is master.
   • DIRECTORY: add the path within the Git repository that represents the top level of the repository to sync to. This field is optional and the default is the root directory of the repository.

   • AUTH_TYPE: add one of the following authentication types:

     • none
     • ssh
     • cookiefile
     • token

     This field is required.

   • SECRET_NAME: add the name you intend to give your Secret. This field is required.

   There can be at most one RepoSync resource per namespace. To enforce this, the object's name must be repo-sync. All configs contained in the directory referenced by the RepoSync must also be in the same namespace as the RepoSync resource.

5. Apply the RepoSync configuration:

   kubectl apply -f repo-sync.yaml
   

6. To verify the configuration, use kubectl get on one of the objects in the namespace repository. For example:

   kubectl get rolebindings -n NAMESPACE
   

Using legacy spec.git fields

To make migration to RootSync resources easier, set the spec.enableLegacyFields to true in your config-management.yaml file. Adding this field lets you use the deprecated spec.git fields in the ConfigManagement resource. Setting this field also automatically generates the RootSync resource on the cluster. Do not change the RootSync configuration.

The following example shows you how to use the spec.git field:

# config-management.yaml

apiVersion: configmanagement.gke.io/v1
kind: ConfigManagement
metadata:
  name: config-management
spec:
  enableMultiRepo: true
  enableLegacyFields: true
  git:
    syncRepo: REPO
    syncBranch: BRANCH
    secretType: TYPE
    policyDir: "DIRECTORY"
  # ...other fields...

To learn more about these fields, see config-management.yaml fields.

Stopping and resuming syncing

This section shows you how to temporarily stop and resume syncing. You might need to do this if an incorrect config is accidentally committed to your repository.

Only a central administrator can stop syncing in the root repository.

The ability to stop syncing in namespace repositories depends on which configuration method was used for your namespace repositories.

• If the Control namespace repositories in the root repository method was used, a central administrator is the only one who can stop and resume syncing.

• If the Control namespace repositories with the Kubernetes API method was used, application operators can stop and resume syncing from the namespace repositories that they work on.

Stopping syncing

This section shows you how to stop syncing for the root repository and for namespace repositories.

Stopping syncing from the root repository

To stop syncing from the root repository, a central administrator can run the following command:

kubectl -n config-management-system scale deployment root-reconciler --replicas=0

This command reduces the replicas count in the root-reconciler Deployment to 0.

Stopping syncing from namespace repositories

Select the Root repository method or Kubernetes API method tab to view the relevant instructions.

Root repository method

If the Control namespace repositories in the root repository method was used, central administrators can run the following commands to stop syncing from a namespace repository:

kubectl -n config-management-system scale deployment ns-reconciler-NAMESPACE --replicas=0

The command reduces the replica count in the ns-reconciler-NAMESPACE Deployment to 0.

Kubernetes API method

If the Control namespace repositories with the Kubernetes API method was used, application operators can stop syncing by running the following commands:

1. Retrieve the RepoSync configuration and save it to use it later when you want to resume syncing:

   kubectl -n NAMESPACE get reposyncs repo-sync -oyaml > repo-sync.yaml
   

   replace NAMESPACE with the namespace of your RepoSync.

2. Delete the RepoSync configuration:

   kubectl -n NAMESPACE delete reposyncs repo-sync
   

   This command triggers the Reconciler Manager to remove the namespace reconciler (ns-reconciler-NAMESPACE) from NAMESPACE and stops syncing.

   Note: Deleting the RepoSync configurations does not affect resources from the namespace repositories that have already synced.

Resuming syncing

This section shows you how to resume syncing for the root repository and for namespace repositories.

Resuming syncing from the root repository

To resume syncing from a root repository, a central administrator can run the following command:

kubectl -n config-management-system scale deployment root-reconciler --replicas=1

This command scales the root-reconciler Deployment to 1 replica.

Resuming syncing from a namespace repository

Select the Root repository method or Kubernetes API method tab to view the relevant instructions.

Root repository method

If you used the Control namespace repositories in the root repository method, a central administrator can run the following command:

kubectl -n config-management-system scale deployment ns-reconciler-NAMESPACE --replicas=1

This command scales the ns-reconciler-NAMESPACE Deployment to 1 replica.

Kubernetes API method

If you used the Control namespace repositories with the Kubernetes API method, application operators can resume syncing by reapplying repo-sync.yaml which contains the RepoSync configuration:

kubectl apply -f repo-sync.yaml

This command triggers tne Reconciler Manager to create a namespace reconciler process and create a ns-reconciler-NAMESPACE Deployment.

Removing the root and namespace repositories

To remove the root repository, delete the RootSync file. For example,

kubectl delete -f root-sync.yaml

To uninstall a namespace repository, delete a RepoSync file. This action uninstalls the matching namespace repository.

Conflict resolution

When you are working with two repositories, conflicts might arise when the same resource (matching groups, kinds, names, and namespaces) is declared both in the root and the namespace repository. When this happens, only the declaration in the root repository is applied to the cluster.

Since the root repository always takes precedence, you can resolve the conflict by updating the root repository to match the namespace repository, or by deleting the conflicting object in the namespace repository.

What's next

• Learn more repositories on Config Sync.
• Learn more about writing configs.