Viewing Config Sync logs

Config Sync follows the same logging conventions as Kubernetes. By default, logging verbosity is set to 2.

On each enrolled cluster, multiple Deployments run in Pods in the config-management-system namespace. Each of these Deployments represents a different subsystem, as indicated by the name of the Deployment.

  • reconciler-manager: controls the lifecycle of the root reconciler and each namespace reconciler process.
  • root-reconciler: imports configs found in the root repository and syncs the configs to the cluster.
  • ns-reconciler-NAMESPACE (one per namespace): imports configs found in the namespace repository and syncs the configs to the namespace in the cluster.
  • admission-webhook: rejects requests to modify a managed resource in a way that conflicts with its declaration in Git.
  • git-importer: imports configs found in the repo into the cluster as CustomResourceDefinitions (CRDs) and ensures that the cluster's configuration is kept in sync with the CRDs created.
  • monitor: Exposes monitoring and metrics endpoints

The reconciler (one per RootSync and RepoSync object) Pod has three different containers, git-sync, reconciler, and otel-agent. The git-sync container pulls configs from a Git repository to a local directory that the reconciler container can read from. The reconciler container applies those configs to the cluster. The otel-agent container is an OpenTelemetry agent for retrieving and exporting Config Sync metrics. To view the logs for only one of these containers, specify it with the -c option. The following example shows the logs for the root-reconciler container:

kubectl logs --namespace=config-management-system \
  -l configsync.gke.io/reconciler=root-reconciler \
  -c reconciler

To view the logs for a namespace-reconciler container, replace NAMESPACE with a name for your namespace:

kubectl logs --namespace=config-management-system \
  -l configsync.gke.io/reconciler=ns-reconciler-NAMESPACE \
  -c reconciler

Changing logging verbosity

The default logging verbosity is 2. If you need to increase the verbosity for debugging, follow these steps.

  1. Edit the config-management-operator Deployment:

    kubectl edit deployment -n=kube-system config-management-operator
    

    In the interactive editor, change the value of replicas to 0. This prevents the Operator from reverting the changes you make below.

  2. Get a list of all Deployments related to Config Sync. The names of the Deployments, and the number of Deployments, are both subject to change.

    kubectl get deployments -n=config-management-system
    
  3. For each Deployment you are debugging or monitoring, edit the object. Replace root-reconciler with the name of the Deployment you want to change.

    kubectl edit deployment root-reconciler
    
  4. After modifying all the relevant Deployments, edit the config-management-operator object again and set replicas to 1.

    kubectl get deployments -n=config-management-system
    

When you are finished debugging, you may want to reduce the verbosity of the logs back to 2, to conserve disk space on your nodes. To do that, follow the procedure again, setting the value to 2.

Finding the Git commit that updated an object

When the Operator applies a change to a Kubernetes object because of an update to the repo, the hash for the Git commit is stored in the configmanagement.gke.io/token annotation. To view this hash, use kubectl get:

kubectl get clusterrole namespace-reader  -oyaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    config.k8s.io/owning-inventory: config-management-system_root-sync
    configmanagement.gke.io/cluster-name: config-management-cluster
    configmanagement.gke.io/managed: enabled
    configmanagement.gke.io/source-path: quickstart/multirepo/root/clusterrole-namespace-reader.yaml
    configmanagement.gke.io/token: 67ae536807af8085adae0f31671ef255423c0244
    configsync.gke.io/declared-fields: '{"f:rules":{}}'
    configsync.gke.io/git-context: '{"repo":"https://github.com/GoogleCloudPlatform/anthos-config-management-samples","branch":"init","rev":"HEAD"}'
    configsync.gke.io/manager: :root
  creationTimestamp: "2021-04-23T18:28:55Z"
  labels:
    app.kubernetes.io/managed-by: configmanagement.gke.io
    configsync.gke.io/declared-version: v1
  name: namespace-reader
  resourceVersion: "915476"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/namespace-reader
  uid: 6751acd2-b58d-434a-aeee-2ee92a9020d2
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
  - watch
  - list

You can then use commands such as git show [HASH] to view information about the commit.

What's next