Error reference

Anthos Config Management error messages consist of an error ID in the format KNV1234 where 1234 is a unique number, followed by a description of the problem and a suggestion for how to fix it. This topic documents each of those error messages.

KNV1000: InternalError

KNV1000 indicates a problem with the nomos command itself. Please file a bug report with the exact command you ran and the message you received.

KNV1001: ReservedDirectoryNameError

Deprecated in Anthos Config Management 1.3.

KNV1002: DuplicateDirectoryNameError

Deprecated in Anthos Config Management 1.3.

KNV1003: IllegalNamespaceSubdirectoryError

When using the default repo structure, a directory that contains a namespace config must not contain any subdirectories.

A directory without a namespace config is an abstract namespace directory and has directories inheriting from it, and consequently must have subdirectories. A directory containing a namespace config is a namespace directory and cannot be inherited from, so it must not have any subdirectories.

To fix, either remove the namespace config from the parent directory, or move the subdirectory somewhere else.

This can happen if a directory containing a namespace has a subdirectory.

namespaces/
└── prod/
    ├── namespace.yaml
    └── us_west_1/
# namespaces/prod/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: prod

That directory structure and the contents of namespace.yaml produce this error:

KNV1003: A Namespace directory MUST NOT have subdirectories. Remove the
         Namespace policy from "prod", or move "us_west_1" to an Abstract
         Namespace:

path: namespaces/prod/us_west_1
name: us_west_1

KNV1004: IllegalSelectorAnnotationError

A cluster-scoped object must not declare the annotation configmanagement.gke.io/namespace-selector. NamespaceSelectors can only be declared for namespace scoped objects.

A cluster object must not declare the annotation configmanagement.gke.io/cluster-selector. To fix the error, remove configmanagement.gke.io/cluster-selector from metadata.annotations.

To fix the error, remove configmanagement.gke.io/namespace-selector from the metadata.annotations field.

The following ClusterRole config produces this error:

# cluster/namespace-reader-clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: namespace-reader
  annotations: {
    "configmanagement.gke.io/namespace-selector" : "shipping-dev",
  }
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "watch"]

If you attempt to include this in your cluster, nomos vet returns the following error:

KNV1004: Cluster-scoped objects may not be namespace-selected, and so MUST NOT declare the annotation 'configmanagement.gke.io/namespace-selector'. To fix, remove `metadata.annotations.configmanagement.gke.io/namespace-selector` from:
source: cluster/namespace-reader-clusterrole.yaml
metadata.name: namespace-reader
group: rbac.authorization.k8s.io
version: v1
kind: ClusterRole

If a cluster object declares configmanagement.gke.io/cluster-selector, nomos vet returns the following error:

KNV1004: Clusters may not be cluster-selected, and so MUST NOT declare the annotation 'configmanagement.gke.io/cluster-selector'. To fix, remove `metadata.annotations.configmanagement.gke.io/cluster-selector` from:

source: clusterregistry/cluster.yaml
metadata.name: default-name
group: clusterregistry.k8s.io
version: v1alpha1
kind: Cluster

KNV1005: IllegalManagementAnnotationError

The only valid setting for the management annotation is configmanagement.gke.io/managed=disabled. This setting is used to explicitly unmanage a resource in the Git repository while leaving the config checked in. The annotation configmanagement.gke.io/managed=enabled is not necessary. For more information, see Managing objects.

Setting a different annotation will result in an error like the following:

KNV1005: Config has invalid management annotation configmanagement.gke.io/managed=invalid. If set, the value must be "disabled".

source: namespaces/foo/role.yaml
metadata.name: default-name
group: rbac.authorization.k8s.io
version: v1
kind: Role

KNV1006: ObjectParseError

This error occurs when an object declared in the repository could not be parsed. To fix, validate your yaml format with a tool such as kubectl --validate.

Example:

KNV1006: The following config could not be parsed as a rbac.authorization.k8s.io/v1, Kind=Role:

source: namespaces/foo/role.yaml
metadata.name: default-name
group: rbac.authorization.k8s.io
version: v1
kind: Role

KNV1007: IllegalAbstractNamespaceObjectKindError

When using an unstructured repo, configs must not be declared in an abstract namespace directory. For more information about using unstructured repos, see Using an unstructured repo.

KNV1007: Config "default-name" illegally declared in an abstract namespace directory. Move this config to a namespace directory:

source: namespaces/foo/bar/role.yaml
metadata.name: default-name
group: rbac.authorization.k8s.io
version: v1
kind: Role

KNV1009: IllegalMetadataNamespaceDeclarationError

When using the default repo structure, configs either declare namespace which match the namespace directory containing them or omit the field.

The following is an example of a Role config that triggers the error:

# namespaces/shipping-prod/pod-reader-role.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pod-reader
  namespace: shipping-dev
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "watch", "list"]

If you declare a config with such a namespace, this error occurs:

KNV1009: A config MUST either declare a `namespace` field exactly matching the directory containing the config, "shipping-prod", or leave the field blank:

source: namespaces/shipping-prod/pod-reader-role.yaml
namespace: shipping-dev
metadata.name: pod-reader
group: rbac.authorization.k8s.io
version: v1
kind: Role

For more information about the default repo structure, see Structure of the repo.

KNV1010: IllegalAnnotationDefinitionError

Configs must not declare unsupported annotations starting with configmanagement.gke.io.

Supported annotations are:

  • configmanagement.gke.io/managed: For more information about use, see Managing objects.

  • configmanagement.gke.io/namespace-selector: For more information about use, see Namespace-scoped objects.

  • configmanagement.gke.io/cluster-selector: For more information about use, see ClusterSelectors.

Example error:

KNV1010: Configs MUST NOT declare unsupported annotations starting with
         "configmanagement.gke.io/". The config has invalid annotations:
         "configmanagement.gke.io/invalid", "configmanagement.gke.io/sync-token"

source: namespaces/foo/role.yaml
metadata.name: role
group: rbac.authorization.k8s.io
version: v1
kind: Role

KNV1011: IllegalLabelDefinition

Configs must not have labels with keys that begin with configmanagement.gke.io/. This label key prefix is reserved for use by Anthos Config Management.

The following is an example of a ConfigMap that triggers this error:

# namespaces/prod/mymap.yaml
kind: ConfigMap
apiVersion: v1
metadata:
  name: my-map
  labels:
    configmanagement.gke.io/bad-label: label-value
data:
  mydata: moredata

If you declare a config with such a label, this error occurs:

KNV1011: Configs MUST NOT declare labels starting with "configmanagement.gke.io/". The config has disallowed labels: "configmanagement.gke.io/bad-label"

source: namespaces/prod/mymap.yaml
metadata.name: my-map
group:
version: v1
kind: ConfigMap

KNV1012: NamespaceSelectorMayNotHaveAnnotation

Deprecated in Anthos Config Management 1.3.

KNV1013: ObjectHasUnknownSelector

The config refers to a ClusterSelector or NamespaceSelector that does not exist. Before you can use a selector in an annotation for a config, the selector must exist.

If the selector is removed, remove any configs that refer to it as well. In this example, assume that there is no unknown-cluster-selector ClusterSelector in the clusterregistry/ directory of the repo.

# namespaces/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: foo
  annotations:
    configmanagement.gke.io/cluster-selector: unknown-cluster-selector

That causes this error:

KNV1013: Config "foo" MUST refer to an existing ClusterSelector, but has
         annotation
         “configmanagement.gke.io/cluster-selector=unknown-cluster-selector",
         which maps to no declared ClusterSelector

NamespaceSelector annotations have the additional requirement that the referenced NamespaceSelector be defined in either the same directory or a parent directory of the config reference. Failure to do so will result in this error:

KNV1013: Config "default-name" MUST refer to a NamespaceSelector in its directory or a parent directory. Either remove the annotation "configmanagement.gke.io/namespace-selector=default-ns-selector" from "default-name" or move NamespaceSelector "default-ns-selector" to a parent directory of "default-name".

    source: namespaces/bar/selector.yaml
    metadata.name: default-ns-selector
    group: configmanagement.gke.io
    version: v1
    kind: NamespaceSelector

    source: namespaces/foo/role.yaml
    metadata.name: default-name
    group: rbac.authorization.k8s.io
    version: v1
    kind: Role

KNV1014: InvalidSelectorError

ClusterSelector and NamespaceSelector configs use correct syntax, but a syntax error was found. To fix, ensure that you specify the config according to the appropriate data schema:

For example, this invalid ClusterSelector:

kind: ClusterSelector
apiVersion: configmanagement.gke.io/v1
metadata:
  name: selector-1
spec:
  selector:
    someUnknownField:  # This field is not defined for a LabelSelector
      foo: bar

Causes the following error:

KNV1014: ClusterSelector has validation errors that must be corrected: invalid field "someUnknownField"
    source: clusterregistry/cs.yaml
    metadata.name: selector-1
    group: configmanagement.gke.io
    version: v1
    kind: ClusterSelector

In particular, ClusterSelector and NamespaceSelector definitions define the spec.selector field. Failure to do so will cause the following error:

   KNV1014: NamespaceSelectors MUST define `spec.selector`

    source: namespaces/ns.yaml
    metadata.name: ns-selector-1
    group: configmanagement.gke.io
    version: v1
    kind: NamespaceSelector

KNV1016: PolicyManagementNotInstalledError

Deprecated in Anthos Config Management 1.3.2.

KNV1017: MissingRepoError

When using the default repo structure, a Repo config must exist in the system/ directory of the repo and must include required information such as the repo's semantic version.

If a Repo config doesn't exist, the following error occurs:

KNV1017: The system/ directory must declare a Repo Resource.
path: system/

To fix, define at least a minimal Repo config.
# system/repo.yaml
kind: Repo
apiVersion: configmanagement.gke.io/v1
metadata:
  name: repo
spec:
  version: "0.1.0"

For more information about the default repo structure, see Structure of the repo.

KNV1018: IllegalSubdirectoryError

Deprecated in Anthos Config Management 1.3.

KNV1019: IllegalTopLevelNamespaceError

When using the default repo structure, Namespaces must not be declared directly in namespaces/.

The following is a config that triggers the error:

# namespaces/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: namespaces
source: namespaces/namespace.yaml
metadata.name: namespaces
group:
version: v1
kind: Namespace
   KNV1019: Namespaces MUST be declared in subdirectories of 'namespaces/'. Create a subdirectory for the following Namespace configs:

    source: namespaces/namespace.yaml
    metadata.name: namespaces
    group:
    version: v1
    kind: Namespace

For more information about the default repo structure, see Structure of the repo.

KNV1020: InvalidNamespaceNameError

When using the default repo structure, a namespace config declare metadata.name, and its value must match the name of the namespace's directory. To fix, correct the namespace's metadata.name or its directory.

The following is a config that triggers the error:

# namespaces/prod/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: dev
KNV1020: A Namespace MUST declare `metadata.name` that matches the name of its
         directory.

expected `metadata.name`: prod

source: namespaces/prod/namespace.yaml
metadata.name: dev
group:
version: v1
kind: Namespace

For more information about the default repo structure, see Structure of the repo.

KNV1021: UnknownObjectError

KNV1021: No CustomResourceDefinition is defined for the resource in the cluster.
         Resource types that are not native Kubernetes objects must have a
         CustomResourceDefinition.

source: namespaces/foo/role.yaml
metadata.name: role
group: rbac.authorization.k8s.io
version: v1
kind: Role

KNV1024: IllegalKindInSystemError

KNV1024: Configs of this Kind may not be declared in the `system/` directory of
         the repo:

source: namespaces/foo/role.yaml
metadata.name: role
group: rbac.authorization.k8s.io
version: v1
kind: Role

KNV1027: UnsupportedRepoSpecVersion

The spec.version field in the Repo config represents the semantic version of the repo. This error indicates that you are using an unsupported version.

If your repo's format is compatible with the supported version, update the spec.version field.

If you need to upgrade, follow the instructions in the release notes.

# system/repo.yaml
kind: Repo
apiVersion: configmanagement.gke.io/v1
metadata:
  name: repo
spec:
  version: "0.0.0"

That produces this error:

KNV1027: Unsupported Repo spec.version: "0.0.0". Must use version "1.0.0"

source: system/repo.yaml
name: repo
group: configmanagement.gke.io
version: v1
kind: Repo

KNV1028: InvalidDirectoryNameError

KNV1028: Directory names have fewer than 64 characters, consist of lower case
         alphanumeric characters or '-', and must start and end with an
         alphanumeric character. Rename or remove directory:

path: namespaces/a.b`c
name: a.b`c

KNV1029: MetadataNameCollisionError

KNV1029: Configs of the same Kind MUST have unique names in the same Namespace
         and their parent abstract namespaces:

source: namespaces/foo/r1.yaml
metadata.name: role
group: rbac.authorization.k8s.io
version: v1
kind: Role

source: namespaces/foo/r2.yaml
metadata.name: role
group: rbac.authorization.k8s.io
version: v1
kind: Role

KNV1030: MultipleSingletonsError

KNV1030: Multiple Namespace resources cannot exist in the same directory. To fix, remove the duplicate config(s) such that no more than 1 remains:

source: namespaces/foo/namespace.yaml
metadata.name: foo
group:
version: v1
kind: Namespace

source: namespaces/foo/namespace.yaml
metadata.name: foo
group:
version: v1
kind: Namespace

KNV1031: MissingObjectNameError

All configs must declare metadata.name. To fix, add the metadata.name field to the problematic configs.

KNV1031: A config must declare metadata.name:

source: namespaces/foo/role.yaml
metadata.name:
group: rbac.authorization.k8s.io
version: v1
kind: Role

KNV1032: IllegalHierarchicalKindErrorCode

KNV1032: The type Repo.configmanagement.gke.io is not allowed if `sourceFormat` is set to `unstructured`. To fix, remove the problematic config, or convert your repo to use `sourceFormat: hierarchy`.

source: system/repo.yaml
metadata.name: repo
group: configmanagement.gke.io
version: v1
kind: Repo

KNV1033: IllegalSystemResourcePlacementError

Some Kinds can only be declared inside the system/ directory. The following is a list of Kinds that can exist exclusively in the system/ directory: - HierarchyConfig - Repo

KNV1033: A config of the below Kind MUST NOT be declared outside system/:

source: namespaces/foo/repo.yaml
metadata.name: repo
group: configmanagement.gke.io
version: v1
kind: Repo

KNV1034: IllegalNamespaceError

It is forbidden to declare the config-management-system namespace, or resources within it. To fix, remove the config-management-system namespace and any configs in that namespace.

KNV1034: Configs must not be declared in the "config-management-system" namespace

source: namespaces/config-management-system/role.yaml
namespace: namespaces/config-management-system
metadata.name: default-name
group: rbac.authorization.k8s.io
version: v1
kind: Role
KNV1034: The "config-management-system" namespace must not be declared

source: namespaces/config-management-system/namespace.yaml
metadata.name: config-management-system
group:
version: v1
kind: Namespace

KNV1036: InvalidMetadataNameError

The metadata.name supplied is of invalid format. A valid metadata.name must: - Be shorter than 254 characters. - Consist of lower case alphanumeric characters, ‘-’, or ‘.’. - Start and end with an alphanumeric character.

To fix, change the metadata.name to satisfy the above.

KNV1036: Configs MUST define a metadata.name that is shorter than 254
        characters, consists of lower case alphanumeric characters, '-' or '.',
         and must start and end with an alphanumeric character. Rename or remove
         the config:

source: namespaces/foo/role.yaml
metadata.name: a`b.c
group: rbac.authorization.k8s.io
version: v1
kind: Role

KNV1037: IllegalKindInClusterregistryError

Deprecated in Anthos Config Management 1.3.

KNV1038: IllegalKindInNamespacesError

KNV1038: Configs of the below Kind may not be declared in `namespaces/`:

source: cluster/cr.yaml
metadata.name: role
group: rbac.authorization.k8s.io
version: v1
kind: ClusterRole

KNV1039: IllegalKindInClusterError

It is forbidden to declare a namespace-scoped object outside of namespaces/ or a cluster-scoped object outside of cluster/. To fix, relocate the problematic configs such that they are in a legal directory.

For more information about cluster-scoped objects, see Cluster-scoped objects.

For more information about namespace-scoped objects, see Namespace-scoped objects.

KNV1039: Namespace-scoped configs of the below Kind must not be declared in
         cluster/:

source: namespaces/foo/role.yaml
metadata.name: role
group: rbac.authorization.k8s.io
version: v1
kind: Role

KNV1040: UnknownResourceInHierarchyConfigError

Deprecated in Anthos Config Management 1.3.

KNV1041: UnsupportedResourceInHierarchyConfigError

KNV1041: This Resource Kind MUST NOT be declared in a HierarchyConfig:

source: system/hc.yaml
group: configmanagement.gke.io
kind: Repo

KNV1042: IllegalHierarchyModeError

An illegal value for HierarchyMode was detected on a HierarchyConfig. HierarchyMode must be either none or inherit.

To read more about HierarchyConfigs see Disabling Inheritance for an Object Type.

KNV1042: HierarchyMode invalid is not a valid value for the APIResource Role.rbac.authorization.k8s.io. Allowed values are [none,inherit].

source: system/hc.yaml
metadata.name: default-name
group: configmanagement.gke.io
version: v1
kind: HierarchyConfig

KNV1043: UnsupportedObjectError

KNV1043: Anthos Config Management cannot configure this object. To fix, remove this
         config from the repo.

source: namespaces/foo/role.yaml
metadata.name: role
group: rbac.authorization.k8s.io
version: v1
kind: Role

KNV1044: UnsyncableResourcesErrorCode

KNV1044: An Abstract Namespace directory with configs MUST have at least one
Namespace subdirectory. To fix, do one of the following: add a Namespace
directory below "bar", add a Namespace config to "bar", or remove the configs in
"bar":

path: namespaces/foo/bar/

KNV1045: IllegalFieldsInConfigError

KNV1045: Configs with "metadata.ownerReference" specified are not allowed. To
fix, either remove the config or remove the "metadata.ownerReference" field in
the config:

source: namespaces/foo/replicaset.yaml
metadata.name: replicaSet
group: apps
version: v1
kind: ReplicaSet

KNV1046: ClusterScopedResourceInHierarchyConfigError

KNV1046: This HierarchyConfig references the APIResource "ClusterSelector.configmanagement.gke.io" which has cluster scope. Cluster scoped objects are not permitted in HierarchyConfig.

source: system/hc.yaml
metadata.name: hierarchyconfig
group: configmanagement.gke.io
version: v1
kind: HierarchyConfig

KNV1047: UnsupportedCRDRemovalError

KNV1047: Removing a CRD and leaving the corresponding Custom Resources in the
repo is disallowed. To fix, remove the CRD along with the Custom Resources.

source: cluster/crd.yaml
metadata.name: customResourceDefinition
group: apiextensions.k8s.io
version: v1beta1
kind: CustomResourceDefinition

KNV1048: InvalidCRDNameError

KNV1048: The CustomResourceDefinition has an invalid name. To fix, change the
name to `spec.names.plural+"."+spec.group`.

source: cluster/crd.yaml
metadata.name: customResourceDefinition
group: apiextensions.k8s.io
version: v1beta1
kind: CustomResourceDefinition

KNV1050: DeprecatedGroupKindError

KNV1050: The config is using a deprecated Group and Kind. To fix, set the Group and Kind to "Deployment.apps"

source: namespaces/deployment.yaml
metadata.name: default-name
group: extensions
version: v1beta1
kind: Deployment

KNV1062: InvalidKptfileError

This type of error occues when the Kptfile doesn't have a valid inventory field. A Kptfile should have a non empty inventory field with both identifier and namespace specified. To fix it, you need to specify the valus for .inventory.identifier and .inventory.namespace in the Kptfile.

Example errors:

KNV1062: Invalid inventory invalid name

metadata.name: default-name
group: kpt.dev
version: v1alpha1
kind: Kptfile

KNV2001: pathError

This occurs when an OS-level system call accessing a filesystem resource fails.

KNV2002: apiServerError

This occurs when a request accessing the API Server fails.

KNV2003: osError

This occurs when a generic OS-level system call fails.

KNV2004: SourceError

This error indicates that Anthos Config Management cannot read from the repo. It is usually caused by one of the following:

  • Misconfigured ConfigManagement object

    Check for mistakes such as an incorrect value for policyDir. The value of policyDir is included in the error; verify the value against your Git repository.

  • Invalid Git credentials

    Check the logs for the git-sync process for an error such as Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

    Verify that the Git credentials and the git-creds secret are configured correctly.

The error message might not include the full details for what caused the error, but it does provide a command which prints the logs from the git-sync process which might have more information.

kubectl logs -n config-management-system  -l app=git-importer -c git-sync

KNV2005: ResourceFightWarning

This error indicates that Anthos Config Management is fighting with another controller over a resource. Such fights consume a high amount of resources and can degrade your performance. Fights are also known as resource contention.

To detect fights, check the Anthos Config Management syncer logs by running the following command:

kubectl logs --namespace config-management-system syncer

If you see KNV2005 in the results, then there is a resource fight.

To find more information about any resource conflicts, watch updates to the resource's YAML file by running the following command:

 kubectl get resource --watch -o yaml

Replace resource with the kind of resource that is being fought over. You can see which resource to add based on the log results.

This command returns a stream of the state of the resource after updates are applied to the API server. You can use a file comparison tool to compare the output.

Some resources should belong to other controllers (for example, some operators install or maintain CRDs). These other controllers automatically remove any metadata specific to Anthos Config Management. If another component in your Kubernetes cluster removes Anthos Config Management metadata, stop managing the resource. For information about how to do this, see Stop managing a managed object.

KNV2006: Config Management Errors

In order to help prevent accidental deletion, Anthos Config Management does not allow you to remove all namespaces or cluster-scoped resources in a single commit. If you want to delete the full set of resources under management, it requires two steps:

  1. Remove all but one namespace or cluster-scoped resource in a first commit and allow Anthos Config Management to sync those changes.
  2. Remove the final resource in a second commit.

KNV2010: resourceError

This is a generic error indicating a problem with a resource or set of resources. The message includes the specific resources which caused the error.

KNV2010: Resources were improperly formatted.

Affected resources:
source: system/hc.yaml
group: configmanagement.gke.io
kind: Repo

KNV2011: MissingResourceError

This indicates a specific resource is required to proceed, but the resource was not found. For example, Config Management Operator tried to update a resource, but the resource was deleted while calculating the update.

KNV2012: MultipleSingletonsError

This error reports that more than one instance of an APIResource was found in a context where exactly one of that APIResource is allowed. For example, only one Repo resource may exist on a cluster.

KNV9999: UndocumentedError

You've encountered an error with no documented error message. We haven't yet written documentation specific to the error you encountered.

What's next?