Introduction to troubleshooting

If you're encountering difficulties with Config Sync, this page introduces you to some common tools and procedures that can help you identify and resolve problems that you experience.

Upgrade to a supported version

Consider upgrading Config Sync to a supported version. Upgrading often resolves common problems and gives you access to the most current functionalities. To ensure that you're always on a supported version, you can enable auto-upgrade (Preview).

Use the nomos command-line tool

The nomos command-line tool provides essential insights into your Config Sync setup. The commands described in the following sections are particularly helpful when you're trying to determine the source of your problem or when you need to work with Cloud Customer Care.

View Config Sync status

The nomos status command provides you with aggregated data and errors to help you understand what's happening with your Config Sync installation. The following information is available with nomos status:

  • Installation status per cluster
  • Syncing errors (both reading from Git as well as reconciling the changes)

Create a bug report

If you have a problem with Config Sync that requires help from Cloud Customer Care, you can provide them with valuable debugging information by using the nomos bugreport command.

This command generates a timestamped zip file with information on the Kubernetes cluster set in your kubectl context. The file also contains logs from Config Sync Pods. It doesn't contain information from the resources synced with Config Sync.

View the overview dashboard

The Config Sync dashboard provides you with an overview of the status of the packages that Config Sync manages and the status of the resources in these packages. Exploring this dashboard can help you to get a quick overview of the status of your Config Sync installation and discover any packages that have issues.

  • To access the dashboard, in the Google Cloud console go to the Config page in the Features section:

    Go to Config

Use monitoring and log analysis

Monitoring Config Sync and exploring its logs can help you determine the source of bugs and to better understand any unexpected behavior.

Understand Config Sync metrics

Use Config Sync metrics to gain visibility into the health of Config Sync.

Monitor RootSync and RepoSync objects

When you install Config Sync using the Google Cloud console or Google Cloud CLI, Config Sync automatically creates a RootSync object for you. When you Configure syncing from multiple repositories, you can create RepoSync objects that contain configuration information about your namespace repositories.

Monitoring these objects can reveal valuable information about the state of Config Sync. To learn more, see Monitor RootSync and RepoSync objects.

Use service level indicators (SLIs)

To receive notifications when Config Sync isn't working as intended, use Config Sync SLIs.

Query logs

You can use the Logs Explorer to retrieve, view, and analyze log data for Config Sync. These logs can contain valuable historical data that isn't captured by nomos bugreport when the operator or reconciler Pods are restarted. For examples of queries that might help you diagnose your issue, see Query Config Sync logs.

Examine resources with the kubectl command-line tool

Config Sync is composed of multiple custom resources that you can query by using kubectl commands. These commands help you understand the status of each of Config Sync's objects.

You should know the following information about the Kubernetes resources that Config Sync manages:

  • config-management-system is the namespace we use to run all core system components of Config Sync.
  • configmanagement.gke.io and configsync.gke.io are the API groups we use for all custom resources.

Examples

The following sections show you how you might use kubectl commands to examine Config Sync.

List custom resources

  • You can get a full list of the custom resources by running the following command:

    kubectl api-resources | grep -E "configmanagement.gke.io|configsync.gke.io"

  • Individual custom resources can be consumed by running the following command:

    kubectl get RESOURCE -o yaml.

    Replace RESOURCE with the name of the resource that you want to query.

    For example, the output of the following command lets you check the status of a RootSync object:

    kubectl get rootsync -n config-management-system -o yaml

Check an object's token annotation

You might want to know when a managed Kubernetes object was last updated by Config Sync. Each managed object is annotated with the hash of the Git commit when it was last modified, and the path to the config that contained the modification.

For example, to get the annotation of a ClusterRoleBinding named namespace-readers, run the following command:

kubectl get clusterrolebinding namespace-readers

The output is similar to the following:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    configmanagement.gke.io/source-path: cluster/namespace-reader-clusterrolebinding.yaml
    configmanagement.gke.io/token: bbb6a1e2f3db692b17201da028daff0d38797771
  name: namespace-readers
...

For more information, see labels and annotations.

Read additional troubleshooting documentation

If you're still experiencing problems, the following resources might be helpful:

  • If you've received an error message, see the error reference page for advice on resolving the error.

  • Check to see if the problem you're having is caused by a known issue.

  • If you're having difficulties with a specific area, one of the targeted troubleshooting guides listed in the Troubleshoot by issue type section of the table of contents might help.

What's next