Enabling multi-repo mode on Config Sync allows you to sync configuration from multiple repositories to the same set of clusters. You can utilize the following types of repositories:
Root repository: This repository allows you to sync cluster-scoped and namespace-scoped configs. The root repository uses admin-level credentials to enforce policies on application namespaces and override local changes that drift from the desired state. A central administrator governs the root repository. Only one root repository can exist for each cluster.
Namespace repositories: Namespace repositories are optional and can contain namespace-scoped configs synced to a particular namespace across clusters. You can delegate the setup and control of a namespace repository to non-administrative users.
Each repository has a single Git reference (a repository branch, commit or tag, and directory tuple) that you can manage separately. This configuration decouples the config deployment lifecycle for different teams. It also provides you with more autonomy in choosing where you want to house the repository and how to structure it.
The following diagram shows you an overview of how teams might use a root repository and namespace repositories:
In this diagram, a central administrator manages the centralized infrastructure for the organization and enforces policies on the cluster and on all namespaces in the organization.
The application operators, who are responsible for managing live deployments, apply configurations to the applications in the namespaces that they work on.
Before you begin
Complete the following tasks before enabling multi-repo mode:
- Use Config Sync version 1.5.1 or later.
Limitations
Consider the following limitations when you use multi-repo mode:
- The
spec.enableMultiRepo
field is not compatible withspec.git
field. If you still need to usespec.git
, see the Using legacy spec.git fields section. ClusterSelectors
andnamespaceSelectors
(including annotations pointing to Selectors) only work in the root repository.- You cannot use
gcenode
in theauth
field of your RootSync or RepoSync configurations.
Configuring syncing from the root repository
To configure the root repository, complete the following tasks:
- Open your
config-management.yaml
file. Set the
spec.enableMultiRepo
field totrue
:# config-management.yaml apiVersion: configmanagement.gke.io/v1 kind: ConfigManagement metadata: name: config-management spec: enableMultiRepo: true # ...other fields...
Apply the changes:
kubectl apply -f config-management.yaml
To define specific features for the root repository, create the
root-sync.yaml file
. For example:# root-sync.yaml apiVersion: configsync.gke.io/v1alpha1 kind: RootSync metadata: name: root-sync namespace: config-management-system spec: git: repo: REPOSITORY revision: REVISION branch: BRANCH dir: "DIRECTORY" auth: AUTH_TYPE secretRef: name: SECRET_NAME
Replace the following:
- REPOSITORY: add the URL of the Git repository to use as the root repository. This field is required.
- REVISION: add the Git revision (tag or hash) to check out. This
field is optional and the default value is
HEAD
. - BRANCH: add the branch of the repository to sync from. This
field is optional and the default value is
master
. - DIRECTORY: add the path within the Git repository that represents the top level of the repository to sync to. This field is optional and the default is the root directory of the repository.
AUTH_TYPE: add one of the following authentication types:
none
ssh
cookiefile
token
This field is required.
SECRET_NAME: add the name you intend to give your Secret. This field is required.
Create a Secret based on your preferred authentication method.
The Secret must meet following requirements:
- The Secret's name must match the
spec.git.secretRef
name you defined inrepo-sync.yaml
. - You must add the Secret's public key to the Git provider.
- The Secret's name must match the
Apply the configuration:
kubectl apply -f root-sync.yaml
Verify the installation:
kubectl -n config-management-system get pods | grep reconciler-manager
You should see output similar to the following example:
reconciler-manager-6f988f5fdd-4r7tr 1/1 Running 0 26s
Configuring syncing from namespace repositories
There are two options for configuring namespace repositories:
Control namespace repositories in the root repository. This method centralizes all configuration of namespace repositories in the root repository, allowing a central administrator complete control of the setup.
Control namespace repositories with the Kubernetes API. This method delegates control of the namespace repository to the namespace owners.
Controlling namespace repositories in the root repository
In this method, the central administrator manages the setup of namespace repositories directly from the root repository. Because Config Sync manages the namespace repository resources, this method prevents any local changes to the namespace repository definitions.
To use this method, complete the following tasks:
In the root repository, declare a
namespace
configuration:# ROOT_REPO/namespaces/NAMESPACE/namespace.yaml apiVersion: v1 kind: Namespace metadata: name: NAMESPACE
Replace NAMESPACE with a name for your namespace.
In the root repository, declare a
RepoSync
configuration in the same namespace:# ROOT_REPO/namespaces/NAMESPACE/repo-sync.yaml apiVersion: configsync.gke.io/v1alpha1 kind: RepoSync metadata: name: repo-sync namespace: NAMESPACE spec: git: repo: REPOSITORY revision: REVISION branch: BRANCH dir: "DIRECTORY" auth: AUTH_TYPE secretRef: name: SECRET_NAME
Replace the following:
- NAMESPACE: add the name of your namespace.
- REPOSITORY: add the URL of the Git repository to use as the root repository. This field is required.
- REVISION: add the Git revision (tag or hash) to check out. This
field is optional and the default value is
HEAD
. - BRANCH: add the branch of the repository to sync from. This
field is optional and the default value is
master
. - DIRECTORY: add the path within the Git repository that represents the top level of the repository to sync to. This field is optional and the default is the root directory of the repository.
AUTH_TYPE: add one of the following authentication types:
none
ssh
cookiefile
token
This field is required.
SECRET_NAME: add the name you intend to give your Secret. This field is required.
There can be at most one RepoSync resource per namespace. To enforce this, the object name must be
repo-sync
. All configs contained in the directory referenced by theRepoSync
must also be in the same namespace as theRepoSync
resource.In the root repository, declare a
RoleBinding
configuration that grants thens-reconciler-NAMESPACE
service account permission to manage resources in the namespace. Config Sync automatically creates thens-reconciler-NAMESPACE
service account when theRepoSync
config is synced to the cluster.To declare the
RoleBinding
, create the following manifest:# ROOT_REPO/namespaces/NAMESPACE/sync-rolebinding.yaml kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: syncs-repo namespace: NAMESPACE subjects: - kind: ServiceAccount name: ns-reconciler-NAMESPACE namespace: config-management-system roleRef: kind: ClusterRole name: RECONCILER_ROLE apiGroup: rbac.authorization.k8s.io
Replace the following:
- NAMESPACE: add the name of your namespace.
RECONCILER_ROLE: as the central administrator, you can set the RECONCILER_ROLE to enforce what kinds of configuration can be synced from the namespace repository. You can choose one of the following roles:
A default ClusterRole:
admin
edit
To learn more, see User-facing roles.
A user-defined ClusterRole or Role declared in the root repository. This role allows for fine grained permissions.
Commit the previous changes to the root repository:
git add . git commit -m 'Setting up new namespace repository.' git push
Create a Secret based on your preferred authentication method.
The Secret must meet following requirements:
- Create the Secret in the same namespace as the RepoSync.
- The Secret's name must match the
spec.git.secretRef
name you defined inrepo-sync.yaml
. - You must add the Secret's public key to the Git provider.
To verify the configuration, use
kubectl get
on one of the objects in the namespace repository. For example:kubectl get rolebindings -n NAMESPACE
Controlling namespace repositories with the Kubernetes API
In this method, the central administrator only declares the namespace
in the root repository and delegates declaration of the RepoSync
file to the
application operator.
Central administrator tasks
The central administrator completes the following tasks:
In the root repository, declare a
namespace
configuration:# ROOT_REPO/namespaces/NAMESPACE/namespace.yaml apiVersion: v1 kind: Namespace metadata: name: NAMESPACE
Replace NAMESPACE with a name for your namespace.
In the root repository, declare a
RoleBinding
configuration to give the application operators permissions. Use RBAC escalation prevention to ensure that the application operator cannot later apply a role binding with permissions not granted by this role binding.To declare the RoleBinding, create the following manifest:
# ROOT_REPO/namespaces/NAMESPACE/operator-rolebinding.yaml kind: RoleBinding # Add RBAC escalation prevention apiVersion: rbac.authorization.k8s.io/v1 metadata: name: operator namespace: NAMESPACE subjects: - kind: User name: USER_NAME apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: OPERATOR_ROLE apiGroup: rbac.authorization.k8s.io
Replace the following:
- NAMESPACE: add the namespace you created in the root repository.
- USER_NAME: add the username of the application operator.
OPERATOR_ROLE: as the central administrator, you can set OPERATOR_ROLE to enforce what kinds of configurations can be synced from the namespace repository. You can choose one of the following roles:
A default ClusterRole:
admin
edit
To learn more, see User-facing roles.
A user-defined ClusterRole or Role declared in the root repository. This allows for fine grained permissions.
Commit the previous changes to the root repository:
git add . git commit -m 'Setting up new namespace repository.' git push
Application operator tasks
The application operator completes the following tasks:
Declare a
RoleBinding
configuration that grants the auto-provisionedns-reconciler-NAMESPACE
service account permission to manage resources in the namespace. Config Sync automatically creates thens-reconciler-NAMESPACE
service account when theRepoSync
config is synced to the cluster.To declare the RoleBinding, create the following manifest:
# sync-rolebinding.yaml kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: syncs-repo namespace: NAMESPACE subjects: - kind: ServiceAccount name: ns-reconciler-NAMESPACE namespace: config-management-system roleRef: kind: ClusterRole name: RECONCILER_ROLE apiGroup: rbac.authorization.k8s.io
Replace the following:
- NAMESPACE: add the namespace you created in the root repository.
- RECONCILER_ROLE: as the application operator you can set RECONCILER_ROLE to enforce what kinds of configuration can be synced from the namespace repository. You can only further restrict the set of permissions the central administrator has granted you. As a result, this role cannot be more permissive than the OPERATOR_ROLE that the central administrator declared in the previous section.
Apply the RoleBinding configuration:
kubectl apply -f sync-rolebinding.yaml
Create a Secret based on your preferred authentication method.
The Secret must meet the following requirements:
- Create the Secret in the same namespace as the RepoSync.
- The Secret's name must match the
spec.git.secretRef
name you defined inrepo-sync.yaml
. - You must add the Secret's public key to the Git provider.
Declare a
RepoSync
configuration:# repo-sync.yaml apiVersion: configsync.gke.io/v1alpha1 kind: RepoSync metadata: name: repo-sync namespace: NAMESPACE spec: git: repo: REPOSITORY revision: REVISION branch: BRANCH dir: "DIRECTORY" auth: AUTH_TYPE secretRef: name: SECRET_NAME
Replace the following:
- REPOSITORY: add the URL of the Git repository to use as the root repository. This field is required.
- REVISION: add the Git revision (tag or hash) to check out. This
field is optional and the default value is
HEAD
. - BRANCH: add the branch of the repository to sync from. This
field is optional and the default value is
master
. - DIRECTORY: add the path within the Git repository that represents the top level of the repository to sync to. This field is optional and the default is the root directory of the repository.
AUTH_TYPE: add one of the following authentication types:
none
ssh
cookiefile
token
This field is required.
SECRET_NAME: add the name you intend to give your Secret. This field is required.
There can be at most one RepoSync resource per namespace. To enforce this, the object's name must be
repo-sync
. All configs contained in the directory referenced by theRepoSync
must also be in the same namespace as theRepoSync
resource.Apply the
RepoSync
configuration:kubectl apply -f repo-sync.yaml
To verify the configuration, use
kubectl get
on one of the objects in the namespace repository. For example:kubectl get rolebindings -n NAMESPACE
Using legacy spec.git fields
To make migration to RootSync
resources easier, set the
spec.enableLegacyFields
to true
in your config-management.yaml
file. Adding
this field lets you use the deprecated spec.git
fields in the ConfigManagement
resource. Setting this field also automatically generates the RootSync
resource on
the cluster. Do not change the RootSync
configuration.
The following example shows you how to use the spec.git
field:
# config-management.yaml
apiVersion: configmanagement.gke.io/v1
kind: ConfigManagement
metadata:
name: config-management
spec:
enableMultiRepo: true
enableLegacyFields: true
git:
syncRepo: REPO
syncBranch: BRANCH
secretType: TYPE
policyDir: "DIRECTORY"
# ...other fields...
To learn more about these fields, see config-management.yaml fields.
Stopping and resuming syncing
This section shows you how to temporarily stop and resume syncing. You might need to do this if an incorrect config is accidentally committed to your repository.
Only a central administrator can stop syncing in the root repository.
The ability to stop syncing in namespace repositories depends on which configuration method was used for your namespace repositories.
If the Control namespace repositories in the root repository method was used, a central administrator is the only one who can stop and resume syncing.
If the Control namespace repositories with the Kubernetes API method was used, application operators can stop and resume syncing from the namespace repositories that they work on.
Stopping syncing
This section shows you how to stop syncing for the root repository and for namespace repositories.
Stopping syncing from the root repository
To stop syncing from the root repository, a central administrator can run the following command:
kubectl -n config-management-system scale deployment root-reconciler --replicas=0
This command reduces the replicas
count in the root-reconciler
Deployment to 0.
Stopping syncing from namespace repositories
Select the Root repository method or Kubernetes API method tab to view the relevant instructions.
Root repository method
If the Control namespace repositories in the root repository method was used, central administrators can run the following commands to stop syncing from a namespace repository:
kubectl -n config-management-system scale deployment ns-reconciler-NAMESPACE --replicas=0
The command reduces the replica count in the
ns-reconciler-NAMESPACE
Deployment to 0.
Kubernetes API method
If the Control namespace repositories with the Kubernetes API method was used, application operators can stop syncing by running the following commands:
Retrieve the
RepoSync
configuration and save it to use it later when you want to resume syncing:kubectl -n NAMESPACE get reposyncs repo-sync -oyaml > repo-sync.yaml
replace NAMESPACE with the namespace of your RepoSync.
Delete the
RepoSync
configuration:kubectl -n NAMESPACE delete reposyncs repo-sync
This command triggers the Reconciler Manager to remove the namespace reconciler (
ns-reconciler-NAMESPACE
) from NAMESPACE and stops syncing.
Resuming syncing
This section shows you how to resume syncing for the root repository and for namespace repositories.
Resuming syncing from the root repository
To resume syncing from a root repository, a central administrator can run the following command:
kubectl -n config-management-system scale deployment root-reconciler --replicas=1
This command scales the root-reconciler
Deployment to 1 replica.
Resuming syncing from a namespace repository
Select the Root repository method or Kubernetes API method tab to view the relevant instructions.
Root repository method
If you used the Control namespace repositories in the root repository method, a central administrator can run the following command:
kubectl -n config-management-system scale deployment ns-reconciler-NAMESPACE --replicas=1
This command scales the ns-reconciler-NAMESPACE Deployment to 1 replica.
Kubernetes API method
If you used the Control namespace repositories with the Kubernetes API
method, application operators can resume syncing by reapplying
repo-sync.yaml
which contains the RepoSync
configuration:
kubectl apply -f repo-sync.yaml
This command triggers tne Reconciler Manager to create a namespace reconciler
process and create a ns-reconciler-NAMESPACE
Deployment.
Removing the root and namespace repositories
To remove the root repository, delete the RootSync
file. For
example,
kubectl delete -f root-sync.yaml
To uninstall a namespace repository, delete a RepoSync
file. This action
uninstalls the matching namespace repository.
Conflict resolution
When you are working with two repositories, conflicts might arise when the same resource (matching groups, kinds, names, and namespaces) is declared both in the root and the namespace repository. When this happens, only the declaration in the root repository is applied to the cluster.
Since the root repository always takes precedence, you can resolve the conflict by updating the root repository to match the namespace repository, or by deleting the conflicting object in the namespace repository.