Using the nomos command

The nomos command (nomos.exe on Windows) is an optional command-line tool you can install locally, such as on a workstation or laptop. You can use the nomos command to interact with the Config Sync Operator, check the syntax of configs before you commit them to your repo, and debug problems with Config Sync, your cluster, or your repo.

Note: When the examples in this documentation refer to nomos, Windows users should substitute nomos.exe.

Prerequisites

Before you can use the nomos command to interact with a cluster, the Operator must already be installed on the target cluster, and the kubectl command must be configured to authenticate to the target cluster. See Installing Config Sync for more information.

Installing the nomos command

The nomos command is a binary compiled from Go code. It is optional and is not included in a default Config Sync installation.

To download the nomos command for each version of Config Sync, see Downloads.

Extra steps for macOS and Linux clients

After downloading the binary, configure it to be executable:

chmod +x /path/to/nomos

You can move the command to a location your system searches for binaries, such as /usr/local/bin, or you can run the command by using its fully-qualified path.

Basic usage

The nomos and nomos.exe commands include sub-commands to initialize your repo, check for syntax errors, get the status of each enrolled cluster, and view your repo as a series of CustomResourceDefinitions.

For basic command syntax, use the --help argument:

nomos --help

The nomos command reads from the local clone of your repo. Use the --path flag to specify the location of the top level of the repo. By default, --path is set to ., or the current directory.

nomos --path=path/to/your/repo status

Checking installation status

You can check if Config Sync is properly installed and configured on all of your clusters using the nomos status command. It reports any errors that prevent Config Sync from running. For example:

nomos status

my_managed_cluster-1
  --------------------
  NOT INSTALLED

my_managed_cluster-2
  --------------------
  <root>   [e-mail]:foo-corp/acme@main
  ERROR
  Error:   git-creds not found. Create git-creds secret in config-management-system namespace.

my_managed_cluster-3
  --------------------
  <root>   [e-mail]:foo-corp/acme@main
  SYNCED   f52a11e4

In this output, Config Sync is not installed on managed-cluster-1. It is installed but not correctly configured on managed-cluster-2. It is installed and running correctly on managed-cluster-3.

In addition, the reason managed-cluster-2 is not correctly configured is that the git-creds Secret is not present.

Initializing a new repo

To initialize a new Config Sync repo, create an empty directory, change to it, initialize a new Git repository, then run the nomos init command:

mkdir my-repo
cd my-repo
git init
nomos init

This creates the basic directory structure of your repo, including the system/, cluster/, and namespaces/ directories.

Checking for errors in the repo

Before you commit a config to the repo, use the nomos vet command to check the syntax and validity of the configs in your repo:

nomos vet

If syntax errors are found, the nomos vet command exits with a non-zero status and logs error messages to STDERR.

Server-side validation

If nomos vet is unable to determine if the type is namespaced, nomos connects to the API Server. Because nomos by default understands core Kubernetes types and Config Sync CRDs, it only tries to connect to the API Server if there are CRs which have no corresponding declared CRD. In this case, if the API Server does not have the CRD applied, nomos vet returns an error. To disable this check and suppress errors from missing CRDs, pass the --no-api-server-check flag.

Caching API server metadata

Note: This feature requires a nomos CLI version of v1.5.1-rc.12 or later.

Instead of suppressing API server checks, you can cache the data on the API server for nomos vet. To cache your api-resources, complete the following steps:

1. Connect to a cluster which has all CRDs that you need for your repository. The cluster does not need to have Config Sync enabled.
2. Go to the policyDir of your repository. This is the same directory specified in your ConfigManagement or RootSync resource.
3. Run the following command: kubectl api-resources > api-resources.txt This command creates a file called api-resources.txt that contains the exact output of kubectl api-resources.

From now on, runs of nomos vet within the repository are aware of those type definitions. If the api-resources.txt file is removed or renamed, nomos vet cannot find the file. nomos vet will still attempt to connect to the cluster if it finds manifests for types not declared in api-resources.txt (unless --no-api-server-check is passed).

The api-resources.txt file only impacts how the nomos CLI works. It does not modify the behavior of Config Sync in any way.

It's okay to have extra entries in the api-resources.txt file which are for types that are not in the repository being validated. nomos vet imports the definitions, but does nothing with them.

Caution: Custom types in a repository must either be namespaced or not-namespaced. The nomos vet command does not support types which are namespace-scoped in some clusters and cluster-scoped in others.

Updating api-resources.txt

After ensuring all the CRDs that you want are on the cluster, run the following command:

kubectl api-resources > api-resources.txt

Caution: Do not try to hand-modify api-resources.txt. It is easy to format incorrectly.

Automatically checking for syntax errors when committing

If you commit a file with JSON or YAML errors Config Sync does not apply the change. However, you can prevent these types of errors from ever getting into the repo by using client-side or server-side hooks.

Using nomos vet in a Git pre-commit hook

You can configure a pre-commit hook that runs nomos vet to check for syntax errors when you commit a change to the local Git clone of your repo. If a pre-commit hook exits with a non-zero status, the git commit operation fails.

Note: Pre-commit hooks run on the client, not the server. They must be added manually to each client, and the user can bypass or disable them at any time.

To run the nomos vet command as a pre-commit hook, edit the .git/hooks/pre-commit file in your repo (notice that .git starts with a . character). You may need to create the file manually. Add the nomos vet command to a new line in the script. The --path argument is optional.

nomos vet --path=/path/to/repo

Ensure that the pre-commit file is executable:

chmod +x .git/hooks/pre-commit

Now, when you run a git commit command in the clone of your repo, nomos vet runs automatically.

The contents of the repo's .git/ directory are not tracked by the repo itself, and cannot be committed to the repo in the same location. You can create a directory in the repo for Git hooks, and people who use the repo can copy the hooks into the appropriate place in their local clone.

Using nomos vet in a server-side hook

Git provides a mechanism for running checks at the server, rather than the client, during a git push operation. If the check fails, the git push also fails. These server-side hooks cannot be bypassed by the client. The method for configuring server-side hooks depends on how your Git server is hosted. See one of the following links for more information, or check the documentation for your Git hosting service.

• GitHub
• Cloud Source Repositories
• Bitbucket
• GitLab
• Self-hosted Git server

Viewing the result of all configs in the repo

You can use the nomos hydrate command to view the combined contents of your repo on each enrolled cluster.

Note: nomos hydrate replaces the nomos view command, which is deprecated in Config Sync v1.1 and higher. To continue to use nomos view, use v1.0 of the nomos command.

If you run nomos hydrate with no options, it creates a compiled/ directory in the current working directory. Within that directory, a subdirectory is created for each enrolled cluster, with the fully-resolved configs the Operator would apply to the cluster.

You can specify the name of the output directory by providing a directory path as the command's argument:

nomos hydrate [/path/to/directory]

To limit the output to a single cluster or a list of clusters, use the --clusters flag and supply a comma-separated list of cluster names.

To emulate the behavior of nomos view and save the effective configuration to a single file, use the --flat flag.

For more information, use the --help flag:

nomos hydrate --flat

nomos view

Note: This command is deprecated in Config Sync v1.1 and higher. To continue to use nomos view, use v1.0 of the nomos command.

When Config Sync imports configs from the repo, it converts them to CustomResourceDefinitions (CRDs) of type ClusterConfig or NamespaceConfig. The nomos view command allows you to view the CRDs resulting from the current state of your repo in JSON format. This can be useful before you commit your change, or to debug issues with configs that are not obvious by using the nomos vet command.

nomos view --path=/path/to/your/repo

Note: To prevent an invalid config from propagating further while you debug a problem, you can temporarily prevent the Operator from syncing configs.

Check for errors on your clusters

Whenever you push a Git commit to the repo, the Operator detects the change and applies the new configurations to all enrolled clusters. You can monitor the status of Config Sync on all enrolled clusters using the nomos status command. For each cluster it reports the hash of the Git commit that was last applied to the cluster as well as any errors that have occurred while trying to apply any recent changes. For example:

nomos status

my_managed_cluster-1
  --------------------
  <root>   [e-mail]:foo-corp/acme@main
  SYNCED   f52a11e4

my_managed_cluster-2
  --------------------
  <root>   [e-mail]:foo-corp/acme@main
  PENDING  9edf8444

my_managed_cluster-3
  --------------------
  <root>   [e-mail]:foo-corp/acme@main
  ERROR    f52a11e4
  Error:   KNV1021: No CustomResourceDefinition is defined for the resource in the cluster.

my_managed_cluster-4
  --------------------
  NOT INSTALLED

my_managed_cluster-5
  --------------------
  <root>   [e-mail]:foo-corp/acme@main
  SYNCED   f52a11e4

You can see that two of the clusters have synced the most recent change, one cluster is still syncing, and one cluster has an error that has prevented the change from being applied. In this case it appears that managed-cluster-3 is missing a CRD that the other clusters have installed.

By default the nomos status command prints the status for each cluster and then exits. However you can use the poll flag to run the command continuously and have it reprint the status table at a regular interval:

nomos status --poll 2s

Check for errors on your clusters (multi-repo)

If you have enabled the multi-repo option for your cluster, you can sync from multiple Git repositories. The nomos status command prints the status for each repository, grouped by cluster. For example:

nomos status

my_managed_cluster-1
  --------------------
  <root>   [e-mail]:foo-corp/acme/admin@main
  SYNCED   f52a11e4
  --------------------
  bookstore  [e-mail]:foo-corp/acme/bookstore@v1
  SYNCED     34d1a8c8

my_managed_cluster-2
  --------------------
  <root>   [e-mail]:foo-corp/acme@main
  ERROR    f52a11e4
  Error:   KNV1021: No CustomResourceDefinition is defined for the resource in the cluster.
  --------------------
  bookstore  [e-mail]:foo-corp/acme/bookstore@v1
  SYNCED     34d1a8c8

You can see that each of the clusters are configured with two Git repositories. The <root> repository belongs to the cluster admin and the bookstore repository might belong to an application development team.

By default, the nomos status command prints the status for all repositories on the cluster. However you can use the namespace flag to limit the command to a specific namespaced repository. This lets the application team use nomos status for their repository:

nomos status --namespace bookstore

About Last Synced Token

nomos status displays the most recent Git commit hash that was applied to the cluster in its output under Last Synced Token. To obtain this value, query the Repo object and look at the status.sync field:

kubectl get repos.configmanagement.gke.io -o yaml

apiVersion: v1
kind: Repo
items:
- status:
    sync:
      lastUpdate: "2019-09-26T10:17:57Z"
      latestToken: f1739af550912034139aca51e382dc50c4036ae0

This commit represents the most recent commit against the cluster. However, not every namespace in the cluster is affected by each commit; to see the most recent commit in a particular namespace, query the particular NamespaceConfig object and look at the status field. For example:

kubectl get namespaceconfigs.configmanagement.gke.io my-namespace -o yaml

apiVersion: configmanagement.gke.io/v1
kind: NamespaceConfig
status:
  syncState: synced
      lastUpdate: "2019-09-26T08:24:32Z"
      latestToken: 1850bad9419d3baec8af220c15d5e952dbeb3b25

Even though a namespace may be affected by a commit, not every resource in the namespace may have been affected. To find the most recently-synced commit for a specific resource, query the specific resource and look at metadata.annotations.configmanagement.gke.io/token. For example:

kubectl get clusterrolebindings.rbac.authorization.k8s.io my-role-binding -o yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    configmanagement.gke.io/token: e56b09554ca8004a2c38185a4ed11364fd6986c4

Creating a bug report

If you have a problem with Config Sync that requires help from Google Cloud support, you can provide them valuable debug information using the nomos bugreport command.

nomos bugreport

This command generates a timestamped zip file with information on the Kubernetes cluster set in your kubectl context.

The file contains logs from Config Sync Pods. It does not contain information from the resources synced with Config Sync.

Troubleshooting

On Linux, you may see the following error when executing a nomos command:

failed to create client configs: while getting config path: failed to get current user: user: Current not implemented on linux/amd64

Create a USER environment variable to fix this problem:

export USER=$(whoami)

What's next

• Try the quickstart
• Learn more about configs
• Learn about viewing logs