Stop and resume syncing configs

In some situations, you might need to quickly stop Config Sync from syncing configs from your repo. One such scenario is if someone commits a syntactically valid but incorrect config to the repo, and you want to limit its effects on your running clusters while the config is removed or fixed.

Prerequisites

The user running the commands discussed in this topic needs the following Kubernetes RBAC permissions in the config-management-system namespaces on all clusters where you want to stop syncing:

- apiGroups: ["extensions"]
  resources: ["deployments", "deployments/scale"]
  verbs: ["get", "update"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["list", "watch"]

Stop and resume syncing from single repositories

This section is for single repositories and it shows you how to quickly stop syncing, and how to resume syncing when the problem is fixed. To learn how to stop syncing for multiple repositories, see stopping and resuming syncing from multiple repositories.

Stop syncing

To stop syncing for a single cluster, run the following commands, which are provided as a single command for your convenience, but can also be run separately:

kubectl scale -n config-management-system deployment config-management-operator --replicas=0 \
&& kubectl wait -n config-management-system --for=delete pods -l k8s-app=config-management-operator \
&& kubectl scale -n config-management-system deployment --replicas=0 --all \
&& kubectl wait -n config-management-system --for=delete pods --all

The commands do the following, in sequence. If a command fails, the remaining commands do not run.

Reduce the replicas count in the Config Management Operator Deployment to 0.
Reduce the replicas count of all Pods running in the config-management-system namespace to 0. The exact set of Pods affected varies by product version.

All deployments are still in the cluster, but no replicas of the Operator or any of the processes responsible for syncing are available, so configs are not synced from the repo.

If you need to stop syncing on multiple clusters, run the preceding commands for each cluster.

Resume syncing

To resume syncing for a single cluster, run the following command:

kubectl -n config-management-system scale deployment config-management-operator --replicas=1

This command scales the Operator Deployment to 1 replica. The Operator then notices that the Pods in the config-management-system namespace Deployments are scaled incorrectly and scales them to their appropriate replica count.

If you need to resume syncing on multiple clusters, run the preceding command for each cluster.

Stop and resume syncing from multiple repositories

The section is for multiple repositories and shows you how to temporarily stop and resume syncing from multiple repositories. To learn how to stop syncing for single repositories, see stopping and resuming syncing from single repositories.

Only a central administrator can stop syncing in the root repository.

The ability to stop syncing in namespace repositories depends on which configuration method was used for your namespace repositories.

If the Control namespace repositories in the root repository method was used, a central administrator is the only one who can stop and resume syncing.
If the Control namespace repositories with the Kubernetes API method was used, application operators can stop and resume syncing from the namespace repositories that they work on.

Stop syncing

The following sections show you how to stop syncing for the root repository and namespace repositories. To stop the Reconciler Manager from reverting the change, you need to stop the Reconciler Manager first by running the following commands:

kubectl -n config-management-system scale deployment config-management-operator --replicas=0 \
&& kubectl wait -n config-management-system --for=delete pods -l k8s-app=config-management-operator --timeout=60s \
&& kubectl scale deployment -n config-management-system reconciler-manager --replicas=0 \
&& kubectl wait -n config-management-system --for=delete pods -l app=reconciler-manager

The commands do the following, in sequence. If a command fails, the remaining commands do not run.

Reduce the replicas count in the Config Management Operator Deployment to 0.
Reduce the replicas count in the Reconciler Manager Deployment to 0.

Stop syncing from the root repository

To stop a cluster syncing from the root repository, a central administrator can run the following command:

kubectl -n config-management-system scale deployment root-reconciler --replicas=0

This command reduces the replicas count in the root-reconciler Deployment to 0.

If you need to stop syncing on multiple clusters, run the preceding command for each cluster.

Stop syncing from namespace repositories

Select the Root repository method or Kubernetes API method tab to view the relevant instructions.

Root repository method

If the Control namespace repositories in the root repository method was used, central administrators can run the following commands to stop a cluster syncing from a namespace repository:

kubectl -n config-management-system scale deployment ns-reconciler-NAMESPACE --replicas=0

The command reduces the replica count in the ns-reconciler-NAMESPACE Deployment to 0.

If you need to stop syncing on multiple clusters, run the preceding command for each cluster.

Kubernetes API method

If the Control namespace repositories with the Kubernetes API method was used, application operators can stop a cluster syncing by running the following commands:

Retrieve the RepoSync configuration and save it to use it later when you want to resume syncing:
```
kubectl -n NAMESPACE get reposyncs repo-sync -oyaml > repo-sync.yaml
```
Replace NAMESPACE with the namespace of your RepoSync object.
Delete the RepoSync configuration:
```
kubectl -n NAMESPACE delete reposyncs repo-sync
```
This command triggers the Reconciler Manager to remove the namespace reconciler (ns-reconciler-NAMESPACE) from NAMESPACE and stops syncing.

Note: Deleting the RepoSync configurations does not affect objects from the namespace repositories that have already synced.

If you need to stop syncing on multiple clusters, run the preceding commands for each cluster.

Stop the admission webhook from blocking drifts (optional)

The Config Sync admission webhook is disabled by default, and you can enable and disable the drift prevention functionality provided by the admission webhook using Google Cloud CLI or kubectl.

Stop syncing from all repositories

To stop syncing from all repositories, including the root repository and namespace repositories, run the following commands:

kubectl scale deployment -n config-management-system -l app=reconciler --replicas=0 \
&& kubectl wait -n config-management-system --for=delete pods -l app=reconciler

The commands reduce the replicas count of all Reconciler Pods running in the config-management-system namespace to 0, and wait until all Reconciler Pods are deleted.

All Reconciler Deployments are still in the cluster, but no replicas of the Reconcilers or any of the processes responsible for syncing are available, so configs are not synced from the repo.

Resume syncing

This section shows you how to resume syncing for the root repository and for namespace repositories.

Resume syncing from the root repository

To resume syncing from a root repository, a central administrator can run the following command:

kubectl -n config-management-system scale deployment root-reconciler --replicas=1

This command scales the root-reconciler Deployment to 1 replica.

Resume syncing from a namespace repository

Select the Root repository method or Kubernetes API method tab to view the relevant instructions.

Root repository method

If you used the Control namespace repositories in the root repository method, a central administrator can run the following command:

kubectl -n config-management-system scale deployment ns-reconciler-NAMESPACE --replicas=1

This command scales the ns-reconciler-NAMESPACE Deployment to 1 replica.

Kubernetes API method

If you used the Control namespace repositories with the Kubernetes API method, application operators can resume syncing by reapplying repo-sync.yaml which contains the RepoSync configuration:

kubectl apply -f repo-sync.yaml

This command triggers tne Reconciler Manager to create a namespace reconciler process and create a ns-reconciler-NAMESPACE Deployment.

Enable and disable the admission webhook

The Config Sync admission webhook is disabled by default, and you can enable and disable the drift prevention functionality provided by the admission webhook using Google Cloud CLI or kubectl.

Resume syncing from all repositories

To resume syncing from all repositories, run the following command:

kubectl -n config-management-system scale deployment config-management-operator --replicas=1

This command scales the Operator Deployment to 1 replica. The Operator then notices that the Reconciler Manager Deployment is scaled incorrectly and scales it to the appropriate replica count. In the end, the Reconciler Manager scales the root reconciler and namespace reconcilers to their corresponding replica count.

What's next

Use the nomos command