Managing existing cluster objects

When Anthos Config Management manages a cluster object, it watches the object and the set of configs in the repo that affect the object, and ensures that they are in sync. This topic describes how to start managing an existing object and how to stop managing an object that is currently managed without deleting the object.

Overview

An object in a cluster is managed by Anthos Config Management if it has the annotation configmanagement.gke.io/managed: enabled.

If an object does not have the label configmanagement.gke.io/managed at all, or if it is set to anything other than enabled, the object is unmanaged.

The following flow chart describes some situations that cause an object to become managed or unmanaged:

How to manage or unmanage a Kubernetes object using Anthos Config Management

The chart contains three separate flows: beginning to manage an object, stopping managing an object, and deleting a managed object.

  1. I want to manage an object. a. Does the object have a config in the repo?
    • No: Create a config for the object. Anthos Config Management sets the annotation configmanagement.gke.io/managed: enabled and begins managing the object.
    • Yes: Does the config set the annotation configmanagement.gke.io/managed: disabled?
      • No: The object is already managed.
      • Yes: Change the config to remove the configmanagement.gke.io/managed: disabled annotation. When the change is committed and pushed, Anthos Config Management notices the change, applies the annotation configmanagement.gke.io/managed: enabled, and applies the config.
  2. I want to stop managing an object but not delete it.
    • Edit the config for the object in the repo, and set the annotation configmanagement.gke.io/managed: disabled. When the config change is detected, Anthos Config Management stops managing the object.
  3. I want to stop managing an object and delete it.
    • Delete the object's config from the repo. When you delete a config for an object that was previously managed, the object is deleted from all clusters or Namespaces the config applies to.

In addition to the configmanagement.gke.io/managed: enabled annotation, Anthos Config Management applies the label app.kubernetes.io/managed-by: configmanagement.gke.io to all objects it manages. This label allows you to easily list all objects by Anthos Config Management.

Why not apply the annotation manually?

Anthos Config Management uses a declarative model to apply configuration changes to your clusters by reading your desired configuration from files in your repo. If you attempt to apply the annotation manually (either using the kubectl command or the Kubernetes API), the manual change is quickly and automatically overridden by the contents of your repo.

Stopping managing Namespaces

You can stop managing a Namespace in the same way as you stop managing any type of object. If you do, Anthos Config Management stops managing the Namespace itself, but continues to manage other managed objects in the Namespace, unless you add the configmanagement.gke.io/managed: disabled annotation to each of them.

If configs for managed configs exist within an unmanaged Namespace directory, the syncer logs errors, but other configs continue to sync normally.

Examples

The following examples build upon the quickstart. Follow the quickstart, and stop before Examine your cluster and repo.

List all managed objects

To list all objects managed by Anthos Config Management on a given cluster or Namespace, use a label selector like the following:

kubectl get -l "app.kubernetes.io/managed-by = configmanagement.gke.io"

To list all objects not managed by Anthos Config Management, use a label selector like this:

kubectl get -l "app.kubernetes.io/managed-by != configmanagement.gke.io"

For example, this command lists RoleBindings in the shipping-dev Namespace that are managed by Anthos Config Management:

kubectl get rolebindings -n shipping-dev \
    -l "app.kubernetes.io/managed-by = configmanagement.gke.io"
NAME           AGE
job-creators   12m
pod-creators   12m
sre-admin      12m
viewers        12m

This command lists RoleBindings in the kube-system Namespace that are not managed by Anthos Config Management:

kubectl get rolebindings -n kube-system \
    -l "app.kubernetes.io/managed-by != configmanagement.gke.io"
NAME                                             AGE
fluentd-gcp-scaler-binding                       2d21h
gce:cloud-provider                               2d21h
heapster-binding                                 2d21h
metrics-server-auth-reader                       2d21h
system::leader-locking-kube-controller-manager   2d21h
system::leader-locking-kube-scheduler            2d21h
system:controller:bootstrap-signer               2d21h
system:controller:cloud-provider                 2d21h
system:controller:token-cleaner                  2d21h

Start managing an existing object

In this example, you create a Role manually, then begin managing it with Anthos Config Management.

  1. Create the myrole Role in the audit Namespace:

    kubectl create role -n audit myrole --verb=get --resource=pods
  2. View the permissions granted by the myrole Role:

    kubectl describe role -n audit myrole
    Name:         myrole
    Labels:       <none>
    Annotations:  <none>
    PolicyRule:
      Resources  Non-Resource URLs  Resource Names  Verbs
      ---------  -----------------  --------------  -----
      pods       []                 []              [get]
    

    The Role only has permission to get Pods.

  3. At this point, the Role exists in the cluster, but Anthos Config Management does not know about it.

    1. In a terminal, go to the local clone of your repo.
    2. Use the following command to create a YAML manifest for myrole and save the manifest to a new file called namespaces/audit/myrole.yaml.

      kubectl describe -n audit role myrole -o yaml > namespaces/audit/myrole.yaml
      
    3. Edit the myrole.yaml file.

      1. Remove all fields under the metadata key except for name and namespace.
      2. Add the list verb after get in the rules.verbs list field.

      Save the changes. The resulting file has the following contents:

      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: myrole
        namespace: audit
      rules:
      - apiGroups:
        - ""
        resources:
        - pods
        verbs:
        - get
        - list
      
    4. Commit the change to the repo.

Wait a few moments for the Config Management Operator to notice the commit. To verify that the myrole Role is now managed by Anthos Config Management, use the kubectl describe command again.

Notice the annotation configmanagement.gke.io/managed: enabled, which indicates that the object is managed by Anthos Config Management. Also notice the annotations that show the path and file name in the repo that caused the most recent configuration change to the object, and the Git hash that represents the commit.

Name:         myrole
Labels:       app.kubernetes.io/managed-by=configmanagement.gke.io
Annotations:  configmanagement.gke.io/cluster-name: example-cluster-name
              configmanagement.gke.io/managed: enabled
              configmanagement.gke.io/source-path: namespaces/audit/myrole.yaml
              configmanagement.gke.io/token: 0836805a09f160f12aa934b9042527e7283c7030
PolicyRule:
Resources  Non-Resource URLs  Resource Names  Verbs
---------  -----------------  --------------  -----
pods       []                 []              [get list]

Stop managing a managed object

This example shows how to stop managing an object that Anthos Config Management is currently managing, such as the myrole Role in Start managing an existing object.

  1. Edit the namespaces/online/shipping-app-backend/shipping-dev/job-creator-rolebinding.yaml file in the local clone of your repo, and add an annotations: section that matches the bold text below:

     kind: RoleBinding
     apiVersion: rbac.authorization.k8s.io/v1
     metadata:
       name: job-creators
     subjects:
     - kind: User
       name: sam@foo-corp.com
       apiGroup: rbac.authorization.k8s.io
     roleRef:
       kind: Role
       name: job-creator
       apiGroup: rbac.authorization.k8s.io
     annotations:
       configmanagement.gke.io/managed: disabled
    

    Save the file.

  2. Create a Git commit with your changes, and push the commit to your repo.

  3. Wait a few moments for Anthos Config Management to notice and apply the new commit.

  4. Use the following command to list all the annotations on the job-creators RoleBinding. Output is truncated for readability.

    kubectl get rolebinding job-creators -n audit -o yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      annotations:
        configmanagement.gke.io/cluster-name: my-cluster
        configmanagement.gke.io/managed: disabled
        configmanagement.gke.io/source-path: namespaces/viewers-rolebinding.yaml
        configmanagement.gke.io/sync-token: fabdb51587d51a81c7e419eeb983aafcf293dc83
    ...
    

After verifying that the object is now disabled, you could remove the config for and verify that the now-unmanaged object is not deleted from the Namespace. However, if you decide to manage the object again, you need to re-create its config. It may make more sense to leave configs for unmanaged objects in the repo. For now, do not delete the config, because it is used in the next example.

Now that the object is not managed, it is not created or re-created on new or existing clusters, and it is not removed even if it exists. To resume managing an object that you previously stopped managing, see the next example, Resume managing a previously-unmanaged object.

Resume managing a previously-unmanaged object

This example shows how to resume managing an object you previously removed from management, as in Stop managing an existing object. It assumes that you have not removed the config for the job-creators RoleBinding.

  1. Edit the namespaces/online/shipping-app-backend/shipping-dev/job-creator-rolebinding.yaml file in the local clone of your repo and remove the configmanagement.gke.io/managed: disabled annotation. Save the file.

  2. Commit and push your change. Anthos Config Management does the following:

    • Notices the change
    • Applies the configmanagement.gke.io/managed: enabled annotation; the object is now managed.
    • Applies the config, as would happen with any managed object.
  3. To verify that the object is now managed, list its annotations:

    kubectl get rolebinding job-creators -n audit -o yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      annotations:
        configmanagement.gke.io/cluster-name: my-cluster
        configmanagement.gke.io/managed: enabled
    ...
    

What's next?