Release notes

This page documents production updates to Anthos Config Management. Check this page for announcements about new or updated features, bug fixes, known issues, and deprecated functionality.

You can see the latest product updates for all of Google Cloud on the Google Cloud release notes page.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/anthosconfig-release-notes.xml

October 29, 2020

1.5.1

Anthos Config Management now includes the ability to sync from multiple Git repositories. This is a preview feature. To learn more, see Syncing from multiple repositories.

The installed product version was being misreported as "anthos-config-management" in ACM 1.5.0. The correct product version is now being reported.

When the enableLegacyFields is set to true, the ACM operator will create a RootSync resource automatically, but any subsequent changes to the RootSync resource will not be noticed by the operator. This will be fixed in a subsequent release. As a workaround, if the RootSync resource resource is modified, add or modify an unused annotation on the ConfigManagement resource to cause the operator to reconcile changes in the RootSync resource.

The nomos status output has been modified significantly to provide a consistent experience for both mono-repo and multi-repo clusters.

(Fixed on October 30, 2020) The version of Anthos Configuration Management included in the Anthos On-Prem release 1.5.1-gke.8 had initially referenced a version of the nomos image that had not be moved into the gcr.io/gke-on-prem-release repository, thus preventing a successful installation and/or upgrade of Anthos Configuration Management. This image has since been pushed to the repository to correct the issue for customers not using private registries. Customers using private registries will need to upgrade to 1.5.2 when it is available, or manually copy the nomos:v1.5.1-rc.7 image into their private repository.

September 24, 2020

1.5.0

Anthos Config Management now includes Config Connector v1.19.1.

Anthos Policy Controller has been updated to include a more recent build of OPA Gatekeeper (hash: 15d56e3).

Binary Authorization can now be enabled through the Anthos Config Management Operator. See Setting up with Anthos Config Management for details.

The syncer and importer Containers now both run in the git-importer Pod in the importer Container.

Anthos Config Management installs a resource-group-controller Deployment which fails on non-GKE clusters. This Deployment is unnecessary and does not cause any other issues.

The nomos CLI tool is now available via gcloud. Please see the downloads page for more information.

This release includes several logging and performance improvements.

August 27, 2020

1.4.2

Anthos Config Management now includes Config Connector v1.15.1.

Anthos Policy Controller has been updated to include a more recent build of OPA Gatekeeper (hash: 1de87b6).

This release includes several logging and performance improvements.

An issue with git submodule support is preventing syncing of configuration stored in submodule repositories. If this affects you, please contact support so we can suggest ways to handle your required use cases while we correct this.

July 30, 2020

1.3.3

Updated the git-sync image to fix security vulnerability CVE-2019-5482.

July 23, 2020

1.4.1

Config Connector has been updated in Anthos Config Management to version 1.13.1.

Anthos Config Management now includes Hierarchy Controller as a beta feature. For more information on this component, see the Hierarchy Controller overview.

Policy Controller users may now enable --log-denies to log all denies and dryrun failures. This is useful when trying to see what is being denied or fails dry-run and for keeping a log to debug cluster problems without looking through the status of all constraints. This is configured by setting spec.policyController.logDeniesEnabled: true in the configuration file for the Operator. There is an example in the section on Installing Policy Controller.

This release includes several logging and performance improvements.

This release includes several fixes and improvements for the nomos command line utility.

The use of unsecured HTTP for GitHub repo connections or in an http_proxy is now discouraged, and support for unsecured HTTP will be removed in a future release. HTTPS will continue to be supported for GitHub repo and local proxy connections.

This release improves the handling of GitHub repositories with very large histories.

Prior to this release, Config Sync and kubectl controllers and processes used the same annotation (kubectl.kubernetes.io/last-applied-configuration) to calculate three-way merge patches. The shared annotation sometimes resulted in resource fights, causing unnecessary removal of each other's fields. Config Sync now uses its own annotation, which prevents resource clashes.

In most cases, this change will be transparent to you. However, there are two cases where some previously unspecified behavior will change.

The first case is when you have run kubectl apply on an unmanaged resource in a cluster, and you later add that same resource to the GitHub repo. Previously, Config Sync would have pruned any fields that were previously applied but not declared in the GitHub repo. Now, Config Sync writes the declared fields to the resource and leaves undeclared fields in place. If you want to remove those fields, do one of the following:

  • Get a local copy of the resource from GitHub and kubectl apply it.
  • Use kubectl edit --save-config to remove the fields directly.

The second case is when you stop managing a resource on the cluster or even stop all of Config Sync on a cluster. In this case, if you want to prune fields from a previously managed resource, you will see different behavior. Previously, you could get a local copy of the resource from GitHub, remove the unwanted fields, and kubectl apply it. Now, kubectl apply no longer prunes the missing fields. If you want to remove those fields, do one of the following:

  • Call kubectl apply set-last-applied with the unmodified resource from GitHub, then remove unwanted fields and kubectl apply it again without the set-last-applied flag.
  • Use kubectl edit --save-config to remove the fields directly.

In error messages, links to error docs are now more concise.

June 25, 2020

1.4.0

Anthos Config Management is now Generally Available on AKS (Kubernetes v1.16 or higher) and EKS (Kubernetes v1.16 or higher).

Config Connector is not currently supported on EKS or AKS, as it is unable to run on these providers.

The following Policy Controller constraint templates have been added to the Default Template Library:

  • allowedserviceportname
  • destinationruletlsenabled
  • disallowedauthzprefix
  • policystrictonly
  • sourcenotallauthz

The following constraint templates have been updated:

  • k8sblockprocessnamespacesharing
  • k8sdisallowedrolebindingsubjects
  • k8semptydirhassizelimit
  • k8slocalstoragerequiresafetoevict
  • k8smemoryrequestequalslimit
  • k8snoexternalservices
  • k8spspallowedusers
  • k8spspallowprivilegeescalationcontainer
  • k8spspapparmor
  • k8spspcapabilities
  • k8spspflexvolumes
  • k8spspforbiddensysctls
  • k8spspfsgroup
  • k8spsphostfilesystem
  • k8spsphostnamespace
  • k8spsphostnetworkingports
  • k8spspprivilegedcontainer
  • k8spspprocmount
  • k8spspreadonlyrootfilesystem
  • k8spspseccomp
  • k8spspselinux
  • k8spspvolumetypes

See the Default Template Library documentation for more information.

Anthos Policy Controller has been updated to include a more recent build of OPA Gatekeeper (hash: 25ca799).

This new build of OPA Gatekeeper includes a number of bug fixes and performance improvements, and adds three new monitoring metrics:

  • gatekeeper_sync
  • gatekeeper_sync_duration_seconds
  • gatekeeper_sync_last_run_time

The nomos CLI tool now supports the KUBECONFIG environment variable in a way that matches the kubectl behavior with multiple delimited configuration files.

Anthos Config Management no longer gets into a continuous PATCH loop when encountering unmanaged resources with config-management annotations and a missing last-applied-configuration annotation.

Anthos Config Management is not issuing errors when it encounters certain types of malformed configurations in a resource definition. This may result in the Kubernetes API Server ignoring the malformed fields and applying the default value for the field instead.

Policy Controller may fail to start successfully when synced resources are marked for deletion.

This issue will be addressed in the upstream OPA Gatekeeper project in a future release. For more information see the relevant issue in the Gatekeeper project.

This release includes several logging and performance improvements.

June 15, 2020

1.3.2

A regression in Anthos Config Management 1.3.2 results in unnecessary patches to the API server for the gatekeeper-system namespace and spurious logging for error KNV2005. This "fight" results when the gatekeeper-system namespace is managed in the Git repo, and two Anthos Config Management components (the operator and syncer) are both trying to reconcile the state of the namespace with the API server. The only workaround at this time is to unmanage the gatekeeper-system namespace. The issue will be fixed in Anthos Config Management 1.4.1.

May 21, 2020

1.3.2

This release includes several performance and memory improvements.

In order to help prevent accidental deletion, Anthos Config Management will no longer allow a user to remove all namespaces or cluster-scoped resources in a single commit. If you wish to delete the full set of resources under management, it now requires two steps: remove all but one in a first commit, allow ACM to sync those changes, then remove the final resource in a second commit.

Error documentation has been updated to add more information on error codes. Errors that are no longer encountered in the product have been removed. Most error references have been embellished with examples and steps for remediation.

Anthos Config Management now supports a GKE-only authentication mechanism based on the service account of the cluster's node pool. Documentation on its use is here.

Anthos Config Management now includes Config Connector v1.8.0.

Anthos Config Management will now attempt to detect when resources that it manages are also managed by other controllers. Documentation on this behavior is available in error knv2005 which ACM will log in that case.

Policy Controller has been upgraded to include a newer version of Open Policy Agent Gatekeeper.

This version includes updates to improve the management of policy resources. As a consequence, finalizers are no longer used to manage Constraints and Constraint Templates.

The following metrics have been made obsolete due to these changes and have been removed:

  • gatekeeper_watch_manager_is_running

  • gatekeeper_watch_manager_last_restart_check_time

  • gatekeeper_watch_manager_last_restart_time

  • gatekeeper_watch_manager_restart_attempts

The following metrics were removed and will be re-implemented in a later version:

  • gatekeeper_watch_manager_intended_watch_gvk

  • gatekeeper_watch_manager_watched_gvk

April 23, 2020

1.3.1

Anthos Config Management images are now included in the Google-provided system images for Binary Authorization.

Policy Agent now allows configuration of namespaces that will bypass the admission controller. For more information, see Excluding Namespaces from Policy Controller

You can now exempt Namespaces from Policy Controller enforcement

Anthos Config Management v1.3.1 now supports Kubernetes v1.16 and higher. Earlier versions of Anthos Config Management relied on APIs that have been deprecated in Kubernetes v1.16.

The Anthos Config Management Syncer pod now reports when it detects that it is fighting with another process over a resource.

Anthos Config Management no longer allows managing resources in unmanaged Namespaces.

If you define a CRD with an integer field that has min/max values, Anthos Config Management will be unable to update the CRD.

Anthos Config Management no longer overwrites undeclared labels and annotations on Namespaces.

March 24, 2020

1.3.0

Anthos Policy Controller is now Generally Available

Anthos Config Management now includes the generally-available version of Config Connector.

Anthos Config Management now supports the use of a Personal Access Tokens for authentication against supported Git providers. More information can be found in Installing Anthos Config Management.

Anthos Config Management now supports the use of an HTTP or HTTPS proxy to connect with your Git host. More information can be found at Installing Anthos Config Management.

February 21, 2020

1.2.2

GKE On-prem 1.2.2 includes images for ACM 1.2.1. Upgrading from ACM 1.2.1 to ACM 1.3 is a valid, tested, safe upgrade path.

Minor updates and bug fixes.

February 10, 2020

1.2.1

Anthos Config Management v1.2.1 is generally available for use in production.

Git repos with submodules are now also supported by Anthos Configuration Management out of the box without additional configuration. This allows delegation of config authoring in a Git-idiomatic way. For more information, please see Git's documentation on submodules.

A new CLI subcommand is available. nomos bugreport bundles up Anthos Config Management log information into a Zip file, which can be attached to a Google support ticket.

Previously, adding an APIService to the repo will leave Anthos Config Management in a bad state, with the error message "KNV2002: failed to get server resources: unable to retrieve the complete list of server APIs." This issue has been mitigated; Anthos Config Management will now sync APIService objects correctly.

It is not currently possible to downgrade to v1.0.x after upgrading to a more recent version of Anthos Config Management.

Currently, Config Connector can only be installed on a single cluster. If multiple Config Connector instances attempt to create or mutate the same Google Cloud resource, conflicts or errors may occur, and you may exhaust quota for a given resource.

Anthos Config Management now can optionally support an unstructured repository, though some features that relied on hierarchical namespaces are disabled in this mode. More information can be found here.

December 20, 2019

1.2.0

Anthos Config Management v1.2.0 is generally available for use in production.

This release has minor bug fixes and performance improvements.

September 19, 2019

1.1.0

Anthos Config Management v1.1.0 is generally available for use in production.

Policy Controller (Beta) is a Kubernetes dynamic admission controller that checks, audits, and enforces your clusters' compliance with policies related to security, regulations, or business rules. It is built using Gatekeeper, an open source project.

You can now enable integration with Config Connector (beta), a Kubernetes addon that allows you to manage your Google Cloud Platform resources through Kubernetes configuration. You can sync configs for GCP resources with your Anthos Config Management repo and apply them automatically. For more information, see Installing Config Connector.

The apiVersion for the ConfigManagement CustomResource has changed. No action is required; the CustomResource is upgraded automatically when you upgrade to v1.1.0. You can read more about configuring Anthos Config Management.

Managed CRDs (CustomResourceDefinitions) are now Namespace-scoped by default, instead of cluster-scoped. This matches the semantics of the kubectl command.

If a CRD explicitly specifies a scope, Anthos Config Management honors that scope.

The nomos hydrate command is a replacement for the nomos view command, and reports your Anthos Config Management configuration in a human-readable way.

To use nomos hydrate, upgrade the nomos command to v1.1.

If you need to continue using nomos view, do not upgrade the nomos command from v1.0. It will remain forward-compatible for the foreseeable future.

You can read about a known issue with nomos view.

Anthos Config Management can now be installed on clusters using PodSecurityPolicies.

The nomos view command is deprecated and is not included in v1.1 or higher of the nomos command.

If you need to continue using nomos view, do not upgrade the nomos command from v1.0. It will remain forward-compatible for the foreseeable future.

It is not currently possible to downgrade to v1.0.x after upgrading to v1.1.0.

Currently, Config Connector can only be installed on a single cluster. If multiple Config Connector instances attempt to create or mutate the same GCP resource, conflicts or errors may occur, and you may exhaust quota for a given resource.

Adding an APIService to the repo will leave Anthos Config Management in a bad state, with the error message KNV2002: failed to get server resources: unable to retrieve the complete list of server APIs." This issue will affect both new and existing clusters syncing from this repo. To correct the issue:

  • find the name of the git-importer and syncer pods using kubectl get pods -n config-management-system
  • copy those names and restart the pods with kubectl delete -n config-management-system pods git-importer-xxxx-xxxx syncer-xxxx-xxxx
  • These steps are required once for each cluster.

Once the pods for a cluster are restarted, syncing will be back to normal.

nomos view can fail to parse CRDs (Custom Resource Definitions) that exist in the local clone of the repo but have not yet been synced to a cluster.

To work around this issue, use nomos hydrate instead of nomos view.

June 14, 2019

1.0.0

Anthos Config Management v1.0.0 is generally available for use in production.

To upgrade to this version, follow the instructions for upgrading.

You must update all nomos binaries when you upgrade to Anthos Config Management v1.0.0.

Versions older than v1.0.0 are no longer available. If you participated in the early-access program for Anthos Config Management, you must upgrade to v1.0.0.

You can now sync CustomResourceDefinitions (CRDs). Anthos Config Management can now sync any type of Kubernetes object. For more information, see Configuring CustomResourceDefinitions.

We document how to stop Anthos Config Management from syncing configs as quickly as possible. This allows you to mitigate the potential for propagating unintended configs to clusters.

The nomos status subcommand provides a top-level view of the state of Anthos Config Management on all enrolled clusters, including errors and sync status. It reports on all clusters that kubectl can access.

The product name is now Anthos Config Management.

The nomos version command now provides version details for the Config Management Operator on each configured cluster, as well as the version of the nomos command itself.

New metrics allow you to monitor counts, latencies, and timestamps of specific operations.

The following changes have been made to the canonical example repository:

  • The canonical example repo has moved. If you use this repo or a fork, update your Git repository's remotes or create a separate clone of the new repo to ensure that you receive updates.

  • The filesystem standard and the value of the Repo object's spec.version for this version of Anthos Config Management are both 1.0.0.

  • A new branch named 1.0.0 contains configs compatible with Anthos Config Management v1.0.0.

An example NetworkPolicy illustrates some methods for enforcing good security practices across your clusters.

We improved the instructions for setting up SSH keys for authenticating to a Git repository.

HierarchicalResourceQuotas are no longer supported.

March 29, 2019

0.13.1

Anthos Config Management v0.13.1 (beta) is a maintenance release, and is compatible with Anthos Config Management v0.13.0.

To upgrade from v0.13.0 on an existing cluster, deploy the new Config Management Operator. You may need to remove an object that is no longer used, to prevent spurious log messages. See the release note about the change below.

You can now manage the default Namespace as well as Namespaces with names beginning with kube-.

Previously, if a config change removed a controller object (for example, a Deployment that has a ReplicaSet), removing the controller object did not remove objects it controlled. All of a controller object's child objects are now correctly removed when the controller object itself is removed.

Previously, if your repo contained only configs for Namespaces and no other configs, the Namespace configs would fail to sync. Repos now sync correctly even if it only contains configs for Namespaces.

The git-policy-importer application has been renamed to git-importer.

The nomos-cluster-policy ClusterConfig has been renamed to config-management-cluster-config. After upgrading, both ClusterConfig objects both exist on the cluster. This does not impact the functionality of the cluster, but you may see spurious log messages if the older ClusterConfig is still present. You can remove the old ClusterConfig to avoid these log messages:

kubectl delete clusterconfig nomos-cluster-policy

Syncing of CustomResourceDefinitions is not currently supported. If CustomResourceDefinition has been applied to the cluster, you can sync associated CustomResources.

Update: This issue no longer exists in Anthos Config Management v1.0.0 and higher.

March 20, 2019

0.13.0

Anthos Config Management 13.0.0 is the second beta release of Anthos Config Management. It represents a major change from v0.11.6, is not backward-compatible with any prior release, and cannot be installed on a cluster with a previous installation of Anthos Config Management. Backward-incompatible releases will always use a new minor version number.

You can now share a ResourceQuota among multiple Namespaces with a common abstract namespace directory. See Aggregate ResourceQuotas.

Syncs are no longer required, and are now silently ignored by the Config Management Operator. You can now create a config for any object in your cluster except a CustomResourceDefinition or the Config Management Operator itself.

The nomos-system Namespace has been renamed to config-management-system.

The nomos.dev/ API group has been renamed to configmanagement.gke.io/.

The Nomos object has been renamed to the ConfigManagement object and is now cluster-scoped.

The nomos-resource-quota object, which combines all of a Namespace's effective ResourceQuotas into a single one that is more efficient for Kubernetes to check and enforce, has been renamed to config-management-resource-quota.

Prometheus now uses the gkeconfig Namespace.

Annotations, rather than labels, are now used to determine that Anthos Config Management is managing a Kubernetes object.

Syncing of CustomResourceDefinitions is not currently supported. If a CustomResourceDefinition has been applied to the cluster, you can sync associated CustomResources.

Update: This issue no longer exists in Anthos Config Management v1.0.0 and higher.

March 04, 2019

0.11.6

This is the beta release of Anthos Config Management. It represents a major change from v0.10.4, and cannot be installed on a cluster with a previous installation of Anthos Config Management. An existing installation of the alpha from v0.10.4 or earlier will conflict with a new installation of v0.11.6 due to changes in the repository structure.

Added support for syncing all Kubernetes resources generically. For current limitations, see the list of known issues for this release.

Added support for NamespaceSelectors.

Moved repository format to Filesystem Standard v0.1.0.

Moved installation to use the Config Management Operator.

Syncing of CustomResourceDefinitions is not currently supported. If a CustomResourceDefinition has been applied to the cluster, you can sync associated CustomResources.

Update: This issue no longer exists in Anthos Config Management v1.0.0 and higher.

In some cases, local changes to managed resources made by kubectl apply can result in the removal of the nomos.dev/managed: enabled label, causing the resource to become unmanaged. As a workaround, use kubectl edit instead, or include the label in the YAML manifest you are applying.

Update: This issue no longer exists in Anthos Config Management v1.0.0 and higher. If changes are manually applied to managed Kubernetes objects, Anthos Config Management effectively reverts those changes as soon as it notices a difference between the config in the repo and the object in the cluster.

For more information, see Managing and unmanaging objects.