Use the nomos command

The nomos command (nomos.exe on Windows) is an optional command-line tool you can install locally, for example, on a workstation or laptop. You can use the nomos command to check the syntax of configs before you commit them to your repo, and debug problems with Config Sync, your cluster, and your repo.

Prerequisites

Before you can use the nomos command to interact with a cluster, the Operator must already be installed on the target cluster, and you must configure the kubectl command to authenticate to the target cluster by Generating a kubeconfig entry.

Install the nomos command

The nomos command is a binary compiled from Go code. It is optional and is not included when you install Config Sync. You can install the nomos command by installing the Cloud SDK. If you use Cloud Shell, the Cloud SDK comes pre-installed.

If you do not have the Cloud SDK, we recommend that you use gcloud components install nomos to install the nomos command-line tool. Installing nomos with the gcloud command-line tool lets you use gcloud components update to update nomos to the latest version.

For information on alternate ways the install the nomos command-line tool, see Anthos Config Management downloads.

Basic usage

For basic command syntax, use the --help argument:

nomos --help

The nomos command reads from the local clone of your repo. Use the --path flag to specify the location of the top level of the repo. By default, --path is set to ., or the current directory. For example:

nomos --path=path/to/your/repo vet

Check installation status

You can monitor the status of Config Sync on all enrolled clusters by using the nomos status command. For each cluster, nomos status reports the hash of the Git commit that was last applied to the cluster as well as any errors that have occurred while trying to apply any recent changes. For example:

nomos status

Example output:

my_managed_cluster-1
  --------------------
  <root>   git@github.com:foo-corp/acme@main
  SYNCED   f52a11e4

my_managed_cluster-2
  --------------------
  <root>   git@github.com:foo-corp/acme@main
  PENDING  9edf8444

my_managed_cluster-3
  --------------------
  <root>   git@github.com:foo-corp/acme@main
  ERROR    f52a11e4
  Error:   KNV1021: No CustomResourceDefinition is defined for the resource in the cluster.

my_managed_cluster-4
  --------------------
  NOT INSTALLED

my_managed_cluster-5
  --------------------
  <root>   git@github.com:foo-corp/acme@main
  SYNCED   f52a11e4

In this output:

  • my_managed_cluster-1 and my_managed_cluster-5 have synced the most recent change.
  • my_managed_cluster-2 is still syncing.
  • my_managed_cluster-3 has an error that has prevented the change from being applied. In this case, managed-cluster-3 is missing a CRD that the other clusters have installed.
  • my-managed-cluster-4 does not have Config Sync installed.

Check installation status (multiple-repositories)

When you sync from multiple Git repositories. The nomos status command prints the status for each repository, grouped by cluster. For example:

nomos status

Example output:

my_managed_cluster-1
  --------------------
  <root>   git@github.com:foo-corp/acme/admin@main
  SYNCED   f52a11e4
  --------------------
  bookstore  git@github.com:foo-corp/acme/bookstore@v1
  SYNCED     34d1a8c8

my_managed_cluster-2
  --------------------
  <root>   git@github.com:foo-corp/acme@main
  ERROR    f52a11e4
  Error:   KNV1021: No CustomResourceDefinition is defined for the resource in the cluster.
  --------------------
  bookstore  git@github.com:foo-corp/acme/bookstore@v1
  SYNCED     34d1a8c8

You can see that each of the clusters are configured with two Git repositories. The <root> repository belongs to the cluster admin and the bookstore repository might belong to an application development team.

About last synced commit

nomos status displays the most recent Git commit hash that was applied to the cluster in its output under status.sync.commit. To obtain this value, query the RootSync or RepoSync object and look at the status.sync field.

For example, to query a RootSync object, run the following command:

kubectl get rootsyncs.configsync.gke.io -n config-management-system root-sync -o yaml

Example output:

apiVersion: configsync.gke.io/v1beta1
kind: RootSync
status:
  sync:
    commit: f1739af550912034139aca51e382dc50c4036ae0
    lastUpdate: "2021-04-20T00:25:01Z"

To query a RepoSync object, run the following command:

kubectl get reposync.configsync.gke.io -n NAMESPACE repo-sync -o yaml

Replace NAMESPACE with the namespace that you created your namespace repository in.

Example output:

apiVersion: configsync.gke.io/v1beta1
kind: RepoSync
status:
  sync:
    commit: ed95b50dd918cf65d8908f7561cb8d8d1f179c2f
    lastUpdate: "2021-04-20T00:25:20Z"

This commit represents the most recent commit against the cluster. However, not every resource in the cluster is affected by each commit; to see the most recent commit for a specific resource, query the specific resource and look at metadata.annotations.configmanagement.gke.io/token. For example:

kubectl get clusterroles CLUSTER_ROLE_NAME -o yaml

Replace CLUSTER_ROLE_NAME with the name of the clusterrole you want to query.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    configmanagement.gke.io/token: ed95b50dd918cf65d8908f7561cb8d8d1f179c2f

nomos status flags

To customize nomos status, add the following flags:

Flag Description
--contexts Accepts a comma-separated list of contexts to use in multi-cluster commands. Defaults to all contexts. Use "" for no contexts.
-h or --help Help for the nomos status command.
--namespace Accepts a string. Use the namespace flag to limit the command to a specific namespace repository. Leave unset to get all repositories. This flag is only available if you enabled syncing from multiple repositories.
--poll Use the poll flag to run nomos status continuously and have it reprint the status table at a regular interval. For example 3s. Leave this flag unset to run nomos status once
--resources Accepts true or false. If true, nomos status shows the resource level status for your root or namespace repository when syncing from multiple repositories. The default value is true.
--timeout Timeout for connecting to each cluster. The default value is 3s.

Check the status of each resource

In nomos 1.8.0 or later, you can check if the resources managed by Config Sync are ready by using the nomos status command. This command reports the status for each individual resource from your Git repository. For example:

nomos status --contexts my_managed_cluster_4

Example output:

my_managed_cluster_4
  --------------------
  <root>   https:/github.com/GITHUB_USERNAME/anthos-config-management-samples/namespace-specific-policy/configsync@main
  SYNCED   bf8655aa
  Managed resources:
     NAMESPACE   NAME                                                             STATUS
                 namespace/foo                                                    Current
                 namespace/istio-system                                           Current
                 namespace/tenant-a                                               Current
                 namespace/tenant-b                                               Current
                 namespace/tenant-c                                               Current
     tenant-a    networkpolicy.networking.k8s.io/deny-all                         Current
     tenant-a    role.rbac.authorization.k8s.io/tenant-admin                      Current
     tenant-a    rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding   Current
     tenant-b    networkpolicy.networking.k8s.io/deny-all                         Current
     tenant-b    role.rbac.authorization.k8s.io/tenant-admin                      Current
     tenant-b    rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding   Current
     tenant-c    networkpolicy.networking.k8s.io/deny-all                         Current
     tenant-c    role.rbac.authorization.k8s.io/tenant-admin                      Current
     tenant-c    rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding   Current

In this example, all resources have a status of Current, which means the state of the resource matches the state that you want.

Status can be one of the following values:

  • InProgress: The actual state of the resource has not yet reached the state that you specified in the resource manifest. This means that the resource reconciliation is not complete yet. Newly created resources usually start with this status, although some resources like ConfigMaps are Current immediately.

  • Failed: The process of reconciling the actual state with the state that you want has encountered an error or it has made insufficient progress.

  • Current: The actual state of the resource matches the state that you want. The reconcile process is considered complete until there are changes to either the wanted or the actual state.

  • Terminating: The resource is in the process of being deleted.

  • NotFound: The resource does not exist in the cluster.

  • Unknown: Config Sync is unable to determine the status of the resource.

To turn off showing the resource level status, add --resources=false to the nomos status command.

Check for errors in the repo

Before you commit a config to the repo, use the nomos vet command to check the syntax and validity of the configs in your repo:

nomos vet

If syntax errors are found, the nomos vet command exits with a non-zero status and logs error messages to STDERR.

nomos vet flags

To customize nomos vet, add the following flags:

Flag Description
--clusters Accepts a comma-separated list of cluster names to use in multi-cluster commands. Defaults to all clusters. Use "" for no clusters.
-h or --help Help for the nomos vet command.
--namespace Accepts a string. If set, validates the repository as a namespace repository with the provided name. Automatically sets --source-format=unstructured.
--no-api-server-check Accepts a Boolean. If true, disables talking to the API server for discovery. For more information on this flag, see the Server-side validation section.
--path Accepts a string. The path to the root directory of your Config Sync repository. The default is "."
--source-format Accepts hierarchy or unstructured. If hierarchy or unset, validates the repository as a hierarchical repository. If unstructured, validates the repository as an unstructured repository. This flag is required if you are using an unstructured repository.
--keep-output Accepts a Boolean. If true, the rendered output is saved to the location that you can specify with the --output flag. This flag is available in versions of 1.9.0 and later.
--output Accepts a string. The path to the rendered output. Defaults to the compiled directory. If --keep-output is set to false, this flag is ignored. This flag is available in versions of 1.9.0 and later.
--format Accepts yaml or json. The format of the output. The default value is yaml. This flag is available in versions of 1.9.0 and later.

Server-side validation

If nomos vet is unable to determine if the type is namespaced, nomos connects to the API Server. Because nomos by default understands core Kubernetes types and Config Sync CRDs, it only tries to connect to the API Server if there are CRs which have no corresponding declared CRD. In this case, if the API Server does not have the CRD applied, nomos vet returns an error. To disable this check and suppress errors from missing CRDs, pass the --no-api-server-check flag.

Caching API server metadata

Instead of suppressing API server checks, you can cache the data on the API server for nomos vet. To cache your api-resources, complete the following steps:

  1. Connect to a cluster which has all CRDs that you need for your repository. The cluster does not need to have Config Sync enabled.
  2. Go to the policyDir of your repository. This is the same directory specified in your ConfigManagement or RootSync resource.
  3. Run the following command: kubectl api-resources > api-resources.txt This command creates a file called api-resources.txt that contains the exact output of kubectl api-resources.

From now on, runs of nomos vet within the repository are aware of those type definitions. If the api-resources.txt file is removed or renamed, nomos vet cannot find the file. nomos vet will still attempt to connect to the cluster if it finds manifests for types not declared in api-resources.txt (unless --no-api-server-check is passed).

The api-resources.txt file only impacts how the nomos CLI works. It does not modify the behavior of Config Sync in any way.

It's okay to have extra entries in the api-resources.txt file which are for types that are not in the repository being validated. nomos vet imports the definitions, but does nothing with them.

Update api-resources.txt

After ensuring all the CRDs that you want are on the cluster, run the following command:

kubectl api-resources > api-resources.txt

Automatically check for syntax errors when committing

If you commit a file with JSON or YAML errors Config Sync does not apply the change. However, you can prevent these types of errors from ever getting into the repo by using client-side or server-side hooks.

Use nomos vet in a Git pre-commit hook

You can configure a pre-commit hook that runs nomos vet to check for syntax errors when you commit a change to the local Git clone of your repo. If a pre-commit hook exits with a non-zero status, the git commit operation fails.

To run the nomos vet command as a pre-commit hook, edit the .git/hooks/pre-commit file in your repo (notice that .git starts with a . character). You may need to create the file manually. Add the nomos vet command to a new line in the script. The --path argument is optional.

nomos vet --path=/path/to/repo

Ensure that the pre-commit file is executable:

chmod +x .git/hooks/pre-commit

Now, when you run a git commit command in the clone of your repo, nomos vet runs automatically.

The contents of the repo's .git/ directory are not tracked by the repo itself, and cannot be committed to the repo in the same location. You can create a directory in the repo for Git hooks, and people who use the repo can copy the hooks into the appropriate place in their local clone.

Use nomos vet in a server-side hook

Git provides a mechanism for running checks at the server, rather than the client, during a git push operation. If the check fails, the git push also fails. These server-side hooks cannot be bypassed by the client. The method for configuring server-side hooks depends on how your Git server is hosted. See one of the following links for more information, or check the documentation for your Git hosting service.

View all configs in the repo

You can use the nomos hydrate command to view the combined contents of your repo on each enrolled cluster.

If you run nomos hydrate with no options, it creates a compiled/ directory in the current working directory. Within that directory, a subdirectory is created for each enrolled cluster, with the fully-resolved configs the Operator would apply to the cluster.

This command can also be used to convert a hierarchical repo to one or more unstructured repos, using the content in the compiled/ directory.

nomos hydrate flags

To customize nomos hydrate, add the following flags:

Flag Description
--clusters Accepts a comma-separated list of cluster names. Use this flag to limit the output to a single cluster or a list of clusters. Defaults to all clusters. Use "" for no clusters.
--flat If enabled, print all output to a single file. Use this flag if you want to emulate the behavior of nomos view.
-h or --help Help for the nomos hydrate command.
--format Accepts a yaml or json. The format of the output. The default value is yaml. This flag is available in versions of 1.9.0 and later.
--no-api-server-check Accepts a Boolean. If true, disables talking to the API server for discovery. For more information on this flag, see the Server-side validation section.
--output Accepts a string. The location to write hydrated configuration to. The default is the compiled directory. If --flat is not enabled, writes each resource manifest as a separate file. If --flat is enabled, writes to the, writes a single file holding all resource manifests.
--path Accepts a string. The path to the root directory of your Config Sync repository. The default is "."
--source-format Accepts hierarchy or unstructured. If hierarchy or unset, validates the repository as a hierarchical repository. If unstructured, validates the repository as an unstructured repository. This flag is required if you are using an unstructured repository and is available in versions 1.9.0 and later.

nomos view

When Config Sync imports configs from the repo, it converts them to CustomResourceDefinitions (CRDs) of type ClusterConfig or NamespaceConfig. The nomos view command allows you to view the CRDs resulting from the current state of your repo in JSON format. This can be useful before you commit your change, or to debug issues with configs that are not obvious by using the nomos vet command.

nomos view --path=/path/to/your/repo

Create a bug report

If you have a problem with Config Sync that requires help from Google Cloud support, you can provide them with valuable debugging information by using the nomos bugreport command. You can use this command for single repositories and multiple repositories.

nomos bugreport

This command generates a timestamped zip file with information on the Kubernetes cluster set in your kubectl context. The file also contains logs from Config Sync Pods. It does not contain information from the resources synced with Config Sync.

Initialize a hierarchical repo

You can organize your repo arbitrarily if you are using an unstructured repository. If you are using a hierarchical repository, you need to run the nomos init command to initialize a hierarchical directory:

nomos init

This creates the basic directory structure of a hierarchical repo, including the system/, cluster/, and namespaces/ directories.

nomos init flags

To customize nomos init, add the following flags:

Flag Description
--force Write to directory even if nonempty, overwriting conflicting files
-h or --help Help for the nomos init command.
--path Accepts a string. The root directory to use for your Anthos Config Management repository. The default value is "."

Troubleshooting

On Linux, you might see the following error when executing a nomos command:

failed to create client configs: while getting config path: failed to get current user: user: Current not implemented on linux/amd64

To fix this problem, create a USER environment variable:

export USER=$(whoami)

What's next