Stop and resume syncing configs
In some situations, you might need to quickly stop Config Sync from syncing configs from your repo. One such scenario is if someone commits a syntactically valid but incorrect config to the repo, and you want to limit its effects on your running clusters while the config is removed or fixed.
Prerequisites
The user running the commands discussed in this topic needs the following
Kubernetes RBAC permissions in the config-management-system
namespaces on all
clusters where you want to stop syncing:
- apiGroups: ["extensions"]
resources: ["deployments", "deployments/scale"]
verbs: ["get", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["list", "watch"]
Stop and resume syncing from single repositories
This section is for single repositories and it shows you how to quickly stop syncing, and how to resume syncing when the problem is fixed. To learn how to stop syncing for multiple repositories, see stopping and resuming syncing from multiple repositories.
Stop syncing
To stop syncing for a single cluster, run the following commands, which are provided as a single command for your convenience, but can also be run separately:
kubectl scale -n config-management-system deployment config-management-operator --replicas=0 \ && kubectl wait -n config-management-system --for=delete pods -l k8s-app=config-management-operator \ && kubectl scale -n config-management-system deployment --replicas=0 --all \ && kubectl wait -n config-management-system --for=delete pods --all
The commands do the following, in sequence. If a command fails, the remaining commands do not run.
- Reduce the
replicas
count in the Config Management Operator Deployment to 0. - Reduce the
replicas
count of all Pods running in theconfig-management-system
namespace to 0. The exact set of Pods affected varies by product version.
All deployments are still in the cluster, but no replicas of the Operator or any of the processes responsible for syncing are available, so configs are not synced from the repo.
If you need to stop syncing on multiple clusters, run the preceding commands for each cluster.
Resume syncing
To resume syncing for a single cluster, run the following command:
kubectl -n config-management-system scale deployment config-management-operator --replicas=1
This command scales the Operator Deployment to 1 replica. The
Operator then notices that the Pods in the
config-management-system
namespace Deployments are scaled incorrectly and
scales them to their appropriate replica count.
If you need to resume syncing on multiple clusters, run the preceding command for each cluster.
Stop and resume syncing from multiple repositories
The section is for multiple repositories and shows you how to temporarily stop and resume syncing from multiple repositories. To learn how to stop syncing for single repositories, see stopping and resuming syncing from single repositories.
Only a central administrator can stop syncing in the root repository.
The ability to stop syncing in namespace repositories depends on which configuration method was used for your namespace repositories.
If the Control namespace repositories in the root repository method was used, a central administrator is the only one who can stop and resume syncing.
If the Control namespace repositories with the Kubernetes API method was used, application operators can stop and resume syncing from the namespace repositories that they work on.
Stop syncing
The following sections show you how to stop syncing for the root repository and namespace repositories. To stop the Reconciler Manager from reverting the change, you need to stop the Reconciler Manager first by running the following commands:
kubectl -n config-management-system scale deployment config-management-operator --replicas=0 \ && kubectl wait -n config-management-system --for=delete pods -l k8s-app=config-management-operator --timeout=60s \ && kubectl scale deployment -n config-management-system reconciler-manager --replicas=0 \ && kubectl wait -n config-management-system --for=delete pods -l app=reconciler-manager
The commands do the following, in sequence. If a command fails, the remaining commands do not run.
- Reduce the
replicas
count in the Config Management Operator Deployment to 0. - Reduce the
replicas
count in the Reconciler Manager Deployment to 0.
Stop syncing from the root repository
To stop a cluster syncing from the root repository, a central administrator can run the following command:
kubectl -n config-management-system scale deployment root-reconciler --replicas=0
This command reduces the replicas
count in the root-reconciler
Deployment
to 0.
If you need to stop syncing on multiple clusters, run the preceding command for each cluster.
Stop syncing from namespace repositories
Select the Root repository method or Kubernetes API method tab to view the relevant instructions.
Root repository method
If the Control namespace repositories in the root repository method was used, central administrators can run the following commands to stop a cluster syncing from a namespace repository:
kubectl -n config-management-system scale deployment ns-reconciler-NAMESPACE --replicas=0
The command reduces the replica count in the
ns-reconciler-NAMESPACE
Deployment to 0.
If you need to stop syncing on multiple clusters, run the preceding command for each cluster.
Kubernetes API method
If the Control namespace repositories with the Kubernetes API method was used, application operators can stop a cluster syncing by running the following commands:
Retrieve the
RepoSync
configuration and save it to use it later when you want to resume syncing:kubectl -n NAMESPACE get reposyncs repo-sync -oyaml > repo-sync.yaml
Replace
NAMESPACE
with the namespace of your RepoSync object.Delete the
RepoSync
configuration:kubectl -n NAMESPACE delete reposyncs repo-sync
This command triggers the Reconciler Manager to remove the namespace reconciler (
ns-reconciler-NAMESPACE
) from NAMESPACE and stops syncing.If you need to stop syncing on multiple clusters, run the preceding commands for each cluster.
Stop the admission webhook from blocking drifts (optional)
The Config Sync admission webhook is disabled by default, and you can
enable and
disable the drift
prevention functionality provided by the admission webhook using Google Cloud CLI
or kubectl
.
Stop syncing from all repositories
To stop syncing from all repositories, including the root repository and namespace repositories, run the following commands:
kubectl scale deployment -n config-management-system -l app=reconciler --replicas=0 \ && kubectl wait -n config-management-system --for=delete pods -l app=reconciler
The commands reduce the replicas
count of all Reconciler Pods running in the
config-management-system
namespace to 0, and wait until all Reconciler Pods are deleted.
All Reconciler Deployments are still in the cluster, but no replicas of the Reconcilers or any of the processes responsible for syncing are available, so configs are not synced from the repo.
Resume syncing
This section shows you how to resume syncing for the root repository and for namespace repositories.
Resume syncing from the root repository
To resume syncing from a root repository, a central administrator can run the following command:
kubectl -n config-management-system scale deployment root-reconciler --replicas=1
This command scales the root-reconciler
Deployment to 1 replica.
Resume syncing from a namespace repository
Select the Root repository method or Kubernetes API method tab to view the relevant instructions.
Root repository method
If you used the Control namespace repositories in the root repository method, a central administrator can run the following command:
kubectl -n config-management-system scale deployment ns-reconciler-NAMESPACE --replicas=1
This command scales the ns-reconciler-NAMESPACE Deployment to 1 replica.
Kubernetes API method
If you used the Control namespace repositories with the Kubernetes API
method, application operators can resume syncing by reapplying
repo-sync.yaml
which contains the RepoSync
configuration:
kubectl apply -f repo-sync.yaml
This command triggers tne Reconciler Manager to create a namespace reconciler
process and create a ns-reconciler-NAMESPACE
Deployment.
Enable and disable the admission webhook
The Config Sync admission webhook is disabled by default, and you can
enable and
disable the drift
prevention functionality provided by the admission webhook using Google Cloud CLI
or kubectl
.
Resume syncing from all repositories
To resume syncing from all repositories, run the following command:
kubectl -n config-management-system scale deployment config-management-operator --replicas=1
This command scales the Operator Deployment to 1 replica. The Operator then notices that the Reconciler Manager Deployment is scaled incorrectly and scales it to the appropriate replica count. In the end, the Reconciler Manager scales the root reconciler and namespace reconcilers to their corresponding replica count.