When Config Sync manages a cluster object, it watches the object and the set of configs in the repo that affect the object, and ensures that they are in sync. This topic describes how to start managing an existing object and how to stop managing an object that is currently managed without deleting the object.
An object in a cluster is managed by Config Sync if it has the
configmanagement.gke.io/managed: enabled
annotation, and its
configsync.gke.io/resource-id
annotation, which tracks the group, kind,
namespace, and name information of the object, is correct.
The format of the configsync.gke.io/resource-id
annotation is
GROUP_KIND_NAMESPACE_NAME
for a namespace-scoped object, and
GROUP_KIND_NAME
for a cluster-scoped object.
The following flow chart describes some situations that cause an object to become managed or unmanaged:
The chart contains three separate flows: 1) start managing an object, 2) stop managing an object, and 3) delete a managed object.
- I want to start managing an object. Does the object have a manifest? Or in other words,
does the object have a config in the repo?
- No: Create a config for
the object. Config Sync sets the annotation
configmanagement.gke.io/managed: enabled
, sets theconfigsync.gke.io/resource-id
annotation to match the object, and begins managing the object. - Yes: Does the config set the following annotation?
configmanagement.gke.io/managed: disabled
- No: The object is managed by default.
- Yes: Edit the config to remove the
configmanagement.gke.io/managed: disabled
annotation. When the change is pushed to the source repo, Config Sync notices the change, applies the annotationconfigmanagement.gke.io/managed: enabled
and the annotationconfigsync.gke.io/resource-id
, and applies the config.
- No: Create a config for
the object. Config Sync sets the annotation
- I want to stop managing an object but not delete it.
- Edit the config for the object in the repo, and set the annotation
configmanagement.gke.io/managed: disabled
. When the config change is detected, Config Sync stops managing the object.
- Edit the config for the object in the repo, and set the annotation
- I want to stop managing an object and delete it.
- Delete the object's config from the repo. When you delete a config for an object that was previously managed, Config Sync deletes the object from all clusters or namespaces the config applies to.
In addition to the configmanagement.gke.io/managed: enabled
annotation and
the configsync.gke.io/resource-id
annotation,
Config Sync applies the label
app.kubernetes.io/managed-by: configmanagement.gke.io
to all objects it
manages. This label allows you to easily list
all objects by Config Sync.
Why not apply the annotation manually?
Config Sync uses a declarative model to apply configuration changes
to your clusters by reading your desired configuration from your repo.
If you attempt to apply the annotation manually (either using the kubectl
command or the Kubernetes API), Config Sync overrides the manual
automatically with the contents of your repo.
Before you begin
The following examples build upon Get started with Config Sync. Before you begin the following steps, follow the quickstart and complete all of the steps before Explore and test the Config Sync installation.
List all managed objects
To list all objects managed by Config Sync on a given cluster or namespace, use a label selector like the following:
kubectl get object-type -n namespace -l "app.kubernetes.io/managed-by=configmanagement.gke.io"
To list all objects not managed by Config Sync, use a label selector like this:
kubectl get object-type -n namespace -l "app.kubernetes.io/managed-by!=configmanagement.gke.io"
For example, this command lists RoleBindings in the gamestore
namespace that
are managed by Config Sync:
kubectl get rolebindings -n gamestore \ -l "app.kubernetes.io/managed-by=configmanagement.gke.io"
The output is similar to the following:
NAME ROLE AGE
configsync.gke.io:ns-reconciler ClusterRole/configsync.gke.io:ns-reconciler 34h
gamestore-admin ClusterRole/admin 34h
gamestore-webstore-admin ClusterRole/webstore-admin 34h
This command lists RoleBindings in the kube-system
namespace that are not
managed by Config Sync:
kubectl get rolebindings -n kube-system \ -l "app.kubernetes.io/managed-by!=configmanagement.gke.io"
The output is similar to the following:
NAME AGE
fluentd-gcp-scaler-binding 2d21h
gce:cloud-provider 2d21h
heapster-binding 2d21h
metrics-server-auth-reader 2d21h
system::leader-locking-kube-controller-manager 2d21h
system::leader-locking-kube-scheduler 2d21h
system:controller:bootstrap-signer 2d21h
system:controller:cloud-provider 2d21h
system:controller:token-cleaner 2d21h
Start managing an existing object
You can create a config for an existing Kubernetes object, such as a namespace
that already exists in your cluster before you install Config Sync.
However, this config is ignored unless the object has the annotation
configmanagement.gke.io/managed: enabled
and has the correct configsync.gke.io/resource-id
annotation. For an existing object, you need to apply the annotation manually.
For namespaces specifically, Config Sync does apply configs
that create new objects within an unannotated namespace, and applies the
configmanagement.gke.io/managed: enabled
and configsync.gke.io/resource-id
annotations to those objects. However,
Config Sync refuses to modify or remove any unannotated
cluster-scoped object from a cluster. This is illustrated in the diagram in
Working with configs over time.
The following example demonstrates how to manage an existing Role object. First, you create a Role manually, then begin managing it with Config Sync.
Create the
myrole
Role in thegamestore
namespace:kubectl create role -n gamestore myrole --verb=get --resource=pods
View the permissions granted by the
myrole
Role:kubectl describe role -n gamestore myrole
Name: myrole Labels: <none> Annotations: <none> PolicyRule: Resources Non-Resource URLs Resource Names Verbs --------- ----------------- -------------- ----- pods [] [] [get]
The Role only has permission to
get
Pods.At this point, the Role exists in the cluster, but Config Sync does not know about it.
- In a terminal, go to the local clone of your repo.
Use the following command to create a YAML manifest for
myrole
and save the manifest to a new file calledgamestore-myrole.yaml
.kubectl get role myrole -n gamestore -o yaml > gamestore-myrole.yaml
Edit the
gamestore-myrole.yaml
file.- Remove all fields under the
metadata
key except forname
andnamespace
. - Add the
list
verb afterget
in therules.verbs
list field.
Save the changes. The resulting file has the following contents:
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: myrole namespace: gamestore rules: - apiGroups: - "" resources: - pods verbs: - get - list
- Remove all fields under the
Commit the change to the repo.
Wait a few moments for the ConfigManagement Operator to notice the commit. To verify that the
myrole
Role is now managed by Config Sync, runkubectl describe
again.kubectl describe role myrole -n gamestore
Notice the annotation configmanagement.gke.io/managed: enabled
, which
indicates that the object is managed by Config Sync, and the annotation
configsync.gke.io/resource-id
, which tracks the group, kind, namespace, and name information. Also notice the
annotations that show the path and file name in the repo that caused the most
recent configuration change to the object, and the Git hash that represents the
commit.
Name: myrole
Labels: app.kubernetes.io/managed-by=configmanagement.gke.io
configsync.gke.io/declared-version=v1
Annotations: config.k8s.io/owning-inventory: config-management-system_root-sync
configmanagement.gke.io/cluster-name: my-cluster
configmanagement.gke.io/managed: enabled
configmanagement.gke.io/source-path: config-sync-quickstart/multirepo/root/gamestore-myrole.yaml
configmanagement.gke.io/token: 747b843a7ddbd945c0616034a935cf648b58e7b5
configsync.gke.io/declared-fields: {"f:rules":{}}
configsync.gke.io/git-context: {"repo":"https://github.com/GoogleCloudPlatform/anthos-config-management-samples","branch":"main","rev":"HEAD"}
configsync.gke.io/manager: :root
configsync.gke.io/resource-id: rbac.authorization.k8s.io_role_gamestore_myrole
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
pods [] [] [get list]
Stop managing a managed object
This example shows how to stop managing an object that Config Sync is
currently managing, such as the myrole
Role in
Start managing an existing object.
Edit the
config-sync-quickstart/multirepo/root/rolebinding-gamestore-webstore-admin.yaml
file in the local clone of your repo, and add anannotations:
section that matches the bold text below:kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: annotations: configmanagement.gke.io/managed: disabled name: gamestore-webstore-admin namespace: gamestore subjects: - kind: ServiceAccount name: ns-reconciler-gamestore namespace: config-management-system roleRef: kind: ClusterRole name: webstore-admin apiGroup: rbac.authorization.k8s.io
Save the file.
Create a Git commit with your changes, and push the commit to your repo.
Wait a few moments for Config Sync to notice and apply the new commit.
Use the following command to verify the annotations and labels of the
gamestore-webstore-admin
RoleBinding are both empty. Config Sync does not set theconfigmanagement.gke.io/managed
annotation todisabled
on the object.kubectl get rolebinding gamestore-webstore-admin -n gamestore -o yaml
apiVersion: rbac.authorization.k8s.io/v1 metadata: annotations: name: gamestore-webstore-admin namespace: gamestore subjects: - kind: ServiceAccount name: ns-reconciler-gamestore namespace: config-management-system roleRef: kind: ClusterRole name: webstore-admin apiGroup: rbac.authorization.k8s.io
After verifying that the object is now disabled, you could remove the config from your repository and verify that the now-unmanaged object is not deleted from the namespace. If you want to manage the object again, you must add its config back into your repository. Because of this, you might want to unmanage objects and leave their configs in the repo.
Now that the object is not managed, it is not created or recreated on new or existing clusters, and it is not removed even if it exists. To resume managing an object that you previously stopped managing, see the next example, Resume managing a previously unmanaged object.
Resume managing a previously unmanaged object
This example shows how to resume managing an object you previously removed from
management, as in Stop managing an existing object. It
assumes that you have not removed the config for the
gamestore-webstore-admin
RoleBinding.
If you deleted the
gamestore-webstore-admin
RoleBinding from your repo in the last commit, perform the following steps.Use
git revert
to revert the last commit:git revert HEAD~1
You are asked to confirm the revert operation.
Push the revert commit to your repo.
git push
Edit the
config-sync-quickstart/multirepo/root/rolebinding-gamestore-webstore-admin.yaml
file in the local clone of your repo and remove theconfigmanagement.gke.io/managed: disabled
annotation. Save the file.Commit and push your change. Config Sync does the following:
- Notices the change
- Applies the
configmanagement.gke.io/managed: enabled
annotation and theconfigsync.gke.io/resource-id
annotation; the object is now managed. - Applies the config, as would happen with any managed object.
To verify that the object is now managed, list its annotations:
kubectl get rolebinding gamestore-webstore-admin -n gamestore -o yaml
apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: annotations: configmanagement.gke.io/cluster-name: my-cluster configmanagement.gke.io/managed: enabled configsync.gke.io/resource-id: rbac.authorization.k8s.io_rolebinding_gamestore_gamestore-webstore-admin ...
Stop managing a namespace
You can stop managing a namespace in the same way as you stop managing any type of object. If you want to stop managing other resources within the namespace, perform the following steps:
Add the
configmanagement.gke.io/managed:disabled
annotation to the namespace config and all configs in the same namespace. All objects in the namespace must have this annotation.Commit and push your changes to the repo. Wait for the Operator to sync with the repo.
Delete the unmanaged resources from the repository.
If configs for managed configs exist within an unmanaged namespace directory, the reconciler logs errors, but other configs continue to sync normally.
Delete managed resources
When you remove an individual resource from a source of truth, that object is deleted from the cluster when Config Sync next syncs from the source of truth. Alternatively, you can enable deletion propagation which lets you bulk delete objects.
Delete individual objects
With Config Sync's default behaviour, when you remove an object from a source of truth, that object is deleted from the cluster when Config Sync syncs from the source of truth.
There are multiple ways to check the status of Config Sync or the status of specific objects:
- Use
nomos status
. - Use
kubectl
to examine resources. - Use the
gcloud alpha anthos config sync resources
command. - Use the Config Sync dashboard.
Bulk delete objects
By default, deleting a RootSync or RepoSync causes Config Sync to abandon the objects that were previously applied from the source of truth. Alternatively, you can enable deletion propagation to delete all previously-applied objects.
When you enable deletion propagation on a RootSync or RepoSync object, and then delete that object, Config Sync automatically deletes every object that was managed by that RootSync or RepoSync.
Deletion propagation can help make it easier to clean up resources, for example if you're migrating to a new namespace or cluster, cleaning up after a demo or experiment, or uninstalling an application.
Deletion propagation options
Delation propagation is disabled by default. To enable deletion propagation, you
add the configsync.gke.io/deletion-propagation-policy: Foreground
annotation
to your RootSync or RepoSync object, such as in the following example:
# example-rootsync.yaml
apiVersion: configsync.gke.io/v1beta1
kind: RootSync
metadata:
name: example-rootsync
namespace: config-management-system
annotations:
configsync.gke.io/deletion-propagation-policy: Foreground
spec:
sourceType: git
sourceFormat: unstructured
git:
repo: https://github.com/GoogleCloudPlatform/anthos-config-management-samples
branch: init
dir: config-sync-quickstart/multirepo/root
auth: none
period: 30s
Alternatively, you can update an existing RootSync or RepoSync to use deletion propagation by running the following command:
RootSync
kubectl patch RootSync ROOTSYNC_NAME \
--namespace config-management-system \
--type merge \
--patch '{"metadata":{"annotations":{"configsync.gke.io/deletion-propagation-policy":"Foreground"}}}'
Replace ROOTSYNC_NAME
with the name of the RootSync that
you want to update.
RepoSync
kubectl patch RepoSync REPOSYNC_NAME \
--namespace config-management-system \
--type merge \
--patch '{"metadata":{"annotations":{"configsync.gke.io/deletion-propagation-policy":"Foreground"}}}'
Replace REPOSYNC_NAME
with the name of the RepoSync that
you want to update.
To disable deletion propagation, remove the annotation or change the
value to configsync.gke.io/deletion-propagation-policy: Orphan
:
RootSync
kubectl patch RootSync ROOTSYNC_NAME \
--namespace config-management-system \
--type merge \
--patch '{"metadata":{"annotations":{"configsync.gke.io/deletion-propagation-policy":"Orphan"}}}'
Replace ROOTSYNC_NAME
with the name of the RootSync that
you want to update.
RepoSync
kubectl patch RepoSync REPOSYNC_NAME \
--namespace config-management-system \
--type merge \
--patch '{"metadata":{"annotations":{"configsync.gke.io/deletion-propagation-policy":"Orphan"}}}'
Propagate object deletion
This example shows how to apply deletion propagation to a RootSync or RepoSync object and then delete the RootSync or RepoSync to delete all objects that were managed by the RootSync or RepoSync.
RootSync
Apply the annotation to a RootSync object to enable deletion propagation:
kubectl patch RootSync example-rootsync \ --namespace config-management-system \ --type merge \ --patch '{"metadata":{"annotations":{"configsync.gke.io/deletion-propagation-policy":"Foreground"}}}'
Delete the RootSync object and wait for Config Sync to delete it:
kubectl delete RootSync example-rootsync --namespace config-management-system --wait
Deleting the RootSync can take a few minutes to complete.
RepoSync
Apply the annotation to a RepoSync object to enable deletion propagation:
kubectl patch RepoSync example-reposync \ --namespace example-namespace \ --type merge \ --patch '{"metadata":{"annotations":{"configsync.gke.io/deletion-propagation-policy":"Foreground"}}}'
Delete the RepoSync object and wait for Config Sync to delete it:
kubectl delete RepoSync example-reposync --namespace example-namespace --wait
Deleting the RepoSync can take a few minutes to complete.
Prevent deletion for Kubernetes objects
After you remove a Kubernetes object from a Git repository that is managed by Config Sync, this object is also deleted from the cluster when the new commit is synced to the cluster.
If you want to prevent Config Sync from deleting the object when its configuration is removed from the Git repository, you can take the following steps:
Add the annotation
client.lifecycle.config.k8s.io/deletion: detach
to the object configuration in Git repository.Commit and push the change in Git repository.
Wait for the change being synced to the cluster.
After you complete these steps, Config Sync won't delete this object from the cluster when its configuration is removed from the Git repository, but it can still be deleted by other clients.
Ignore an object in the source of truth
You might want Config Sync to ignore an object in your source of truth. For example, a kpt function configuration should never be applied to the cluster.
For objects you want Config Sync to ignore,
add the
config.kubernetes.io/local-config: "true"
annotation to the object. After you add this annotation, Config Sync ignores
this object as if it is removed from the source of truth. Resources with the
local-config
annotation set to any value other than "false"
are treated as
if it's set to "true"
and are ignored.