The Config Management Operator is a controller that manages Anthos Config Management in a Kubernetes cluster. Follow these steps to install and configure the Operator in each cluster you want to manage using Anthos Config Management.
Before you begin
This section describes prerequisites you must meet before enrolling supported clusters in Anthos Config Management.
Preparing your local environment
Before you install Operator, make sure you have prepared your local environment by completing the following tasks:
Anthos Config Management requires an active Anthos entitlement. For more information, see Pricing for Anthos.
Install the Cloud SDK, which provides the
kubectlcommands used in these instructions.
kubectlis not installed by default by Cloud SDK. To install
kubectl, use the following command:
gcloud components install kubectl
To download components of Anthos Config Management, you must be authenticated to Google Cloud using the
gcloud auth logincommand.
You must configure the
kubectlcommand to connect to your clusters. The steps to do that are different for GKE and GKE on-prem clusters:
The Anthos API must be enabled for your project:
To enable the Anthos API, run the following command:
gcloud services enable anthos.googleapis.com
Your clusters must be on an Anthos supported platform and version.
Your clusters must be registered to an Anthos environ using Connect. Your project's environ provides a unified way to view and manage your clusters and their workloads as part of Anthos, including clusters outside Google Cloud. Anthos charges apply only to your registered clusters. You can find out how to register a cluster in Registering a cluster.
The Google Cloud user installing Anthos Config Management needs Identity and Access Management (IAM) permissions to create new Roles in your cluster.
Enrolling a cluster
To enroll a cluster in Anthos Config Management, complete the following steps:
Deploying the Operator
If you are installing the Config Management Operator using the Google Cloud Console, the
Operator is automatically deployed. Continue to
After ensuring that you meet all the prerequisites, you can deploy the Operator by downloading and applying a YAML manifest.
Download the latest version of the Operator CRD using the following command. To download a specific version instead, see Downloads.
gsutil cp gs://config-management-release/released/latest/config-management-operator.yaml config-management-operator.yaml
Apply the CRD:
kubectl apply -f config-management-operator.yaml
If this fails, see Troubleshooting.
Granting the Operator read-only access to Git
The Operator needs read-only access to your Git repository (the
repo) so it can read the configs committed to the repo and apply them to your
clusters. If credentials are required, they are stored in the
Secret on each enrolled cluster.
If the repo does not require any authentication or authorization for
read-only access, you do not need to create a
git-creds Secret. You can
Configuring Anthos Config Management
and set the
Most users need to create credentials because read access to their repository is restricted. Anthos Config Management supports the following mechanisms for authentication:
- SSH keypair
- Personal Access Token
- Google Service Account (Google Source Repositories only)
- no authentication
The mechanism you choose depends on what your repo supports. If all mechanisms are available, using an SSH keypair is recommended. At the time of writing GitHub, Cloud Source Repositories, and Bitbucket all support using an SSH keypair. If your repo is hosted by your organization and you don't know which authentication methods are supported, contact your administrator.
After you create the credential, you add it to the cluster as a Secret. How you create the secret depends on the method of authentication. For more information, see the section on your authentication method below.
Choose the best method to authorize to your repo from the options below. The
method you choose also determines the
secretType you use when
configuring the Operator.
Using an SSH keypair
If your repo supports using an SSH keypair for authorization, follow the steps in this section to create the Secret materials. Otherwise, follow the instructions in using a cookiefile instead.
An SSH keypair consists of two files, a public key and a private key. The public
key typically has a
Create an SSH keypair to allow the Operator to authenticate to your Git repository. This is necessary if you need to authenticate to the repository in order to clone it or read from it. Skip this step if a security administrator will provide you with a keypair. You can use a single keypair for all clusters, or a keypair per cluster, depending on your security and compliance requirements.
The following command creates 4096-bit RSA key. Lower values are not recommended. Replace
[GIT REPOSITORY USERNAME]and
/path/to/[KEYPAIR-FILENAME]with the values you want the Operator to use to authenticate to the repository. It is recommended to use a separate account if using a third-party Git repository host such as GitHub, or to use a service account if using Cloud Source Repositories.
ssh-keygen -t rsa -b 4096 \ -C "[GIT REPOSITORY USERNAME]" \ -N '' \ -f [/path/to/KEYPAIR-FILENAME]
Configure your repo to recognize the newly-created public key. Refer to the documentation for your Git hosting provider. Instructions for some popular Git hosting providers are included for convenience:
Add the private key to a new Secret in the cluster. Substitute the name of the private key (the one without the
.pubsuffix) where you see /path/to/[KEYPAIR-PRIVATE-KEY-FILENAME].
kubectl create secret generic git-creds \ --namespace=config-management-system \ --from-file=ssh=/path/to/[KEYPAIR-PRIVATE-KEY-FILENAME]
Delete the private key from the local disk or otherwise protect it.
Follow the steps in this section to create the cookiefile Secret materials.
The process for acquiring a cookiefile depends on the configuration of your
repo. For example, see the article about
generating static credentials on Cloud Source Repositories.
The credentials are usually stored in the
.gitcookies file in the user's home
directory, or they may be provided to you by a security administrator.
After you create and obtain the cookiefile, add it to a new Secret in the cluster. Replace /path/to/[COOKIEFILE] with the appropriate path and filename.
kubectl create secret generic git-creds \ --namespace=config-management-system \ --from-file=cookie_file=/path/to/[COOKIEFILE]
Make sure to protect the contents of the cookiefile if you still need it locally. Otherwise, delete it.
Using a token
If your organization does not permit the use of SSH keys, you might prefer to use a token. With Anthos Config Management you can use GitHub's Personal Access Tokens (PAT) or Bitbucket's App Password as your token.
To create a Secret using your token, complete the following steps:
Create a token using GitHub or Bitbucket.
After you create and obtain the token, add it to a new Secret in the cluster. Replace [USERNAME] with the desired username and [TOKEN] with the token you created in the previous step.
kubectl create secret generic git-creds \ --namespace="config-management-system" \ --from-literal=username=[USERNAME] \ --from-literal=token=[TOKEN]
Protect the token if you still need it locally. Otherwise, delete it.
Using a Google Service Account with a Google Source Repository
If your repo is in a Google Source Repository, you
secretType: gcenode to give Anthos Config Management access to a
repository in the same project as your managed cluster.
Before you begin, ensure the following prerequisites are met:
The Compute Engine default service account
PROJECT_NUMBERemail@example.com the cluster must have
source.readeraccess to the repository.
gcloud projects add-iam-policy-binding [PROJECT_ID] \ --member serviceAccount:PROJECT_NUMBERfirstname.lastname@example.org \ --role roles/source.reader
Access scopes for the nodes in the cluster must include
cloud-source-repos-ro. This can be achieved by including
--scopeslist specified at cluster creation time, or by using the
cloud-platformscope at cluster creation time:
gcloud container clusters create example-cluster --scopes=cloud-platform
Once these prerequisites are met, set
spec.git.syncRepo to the URL of the
desired Google Source Repository when you configure the
operator. For example:
gcloud source repos list REPO_NAME PROJECT_ID URL my-repo my-project https://source.developers.google.com/p/my-project/r/my-repo-csr
would require the following:
Using a Google Source Repository with Workload Identity
If Workload Identity is
enabled on your cluster, additional steps are required to use
gcenode. After completing the preceding steps, and
configuring the Operator in
the following section, create an
IAM policy binding
between the Kubernetes service account and the Google service account. The
Kubernetes service account is not created until you configure the
Operator to enable Anthos Config Management for the first time.
This binding allows the Anthos Config Management Kubernetes service account to act as the Compute Engine default service account:
gcloud iam service-accounts add-iam-policy-binding \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:[PROJECT_ID].svc.id.goog[config-management-system/importer]" \ [PROJECT_NUMBER]-email@example.com
Finally, add an
annotation to the Anthos Config Management Kubernetes service
account using the email address of the Compute Engine default service account:
kubectl annotate serviceaccount -n config-management-system importer \ iam.gke.io/gcp-service-account=[PROJECT_NUMBER]-firstname.lastname@example.org
Using no authentication
If your repo does not require authentication for read-only access, you can
continue to configure the Operator
Configuring the Operator
You can configure the Operator for your cluster using
kubectl or the Google Cloud Console.
To configure the behavior of the Operator, create a
configuration file for the ConfigManagement
then apply it using the
kubectl apply command.
For example, create a file
config-management.yaml and copy the following
YAML file into it.
# config-management.yaml apiVersion: configmanagement.gke.io/v1 kind: ConfigManagement metadata: name: config-management spec: # clusterName is required and must be unique among all managed clusters clusterName: my-cluster git: syncRepo: email@example.com:my-github-username/csp-config-management.git syncBranch: 1.0.0 secretType: ssh policyDir: "foo-corp"
For a complete list of fields that you can add to the
spec field, see the
following Configuration for the Git repository
section. Do not change configuration values outside the spec field.
To apply the configuration, use the
kubectl apply command:
kubectl apply -f config-management.yaml
You may need to create separate configuration files for each cluster, or each
type of cluster. You can specify the cluster using the
kubectl apply -f config-management1.yaml --context=cluster-1
Configuration for the Git repository
||The URL of the Git repository to use as the source of truth. Required.|
||The branch of the repository to sync from. Default:
||The path within the Git repository that represents the top level of the repo to sync. Default: the root directory of the repository.|
||Period in seconds between consecutive syncs. Default: 15.|
||Git revision (tag or hash) to check out. Default HEAD.|
||The type of secret configured for access to the Git repository. One of
||When set to
Proxy configuration for the Git repository
If your organization's security policies require you to route traffic through an HTTP(S) proxy, you can configure Anthos Config Management to communicate with your Git host using the proxy's URI.
||httpProxy defines a HTTP_PROXY env variable used to access the Git repo.|
||httpsProxy defines a HTTPS_PROXY env variable used to access the Git repo.|
If both the
httpsProxy fields are specified,
httpProxy will be ignored.
Configuration for behavior of the ConfigManagement object
||The user-defined name for the cluster used by ClusterSelectors to group clusters together. Unique within a Anthos Config Management installation.|
Configuration for integrations
Configuring Anthos Config Management on the Google Cloud Console has the following limitations:
- You cannot install or configure:
- Policy Controller
- Config Connector
- Hierarchy Controller
- Only a subset of Config Sync configuration options are supported. Unsupported
Config Sync configurations include:
If you want to use any of these options, you should not use the Cloud Console.
To configure the Operator on the Cloud Console, complete the following steps:
Visit the Anthos Config Management menu in Google Cloud Console.
Select your registered clusters and click Configure.
In the Git Repository Authentication for ACM section, complete the following:
In the ACM Settings for your clusters section, complete the following:
- In the URL field, add the URL of the Git repository to use as the source of truth. This field is required.
- In the Branch field, add the branch of the repository to sync from. The default is the master. This field is required.
- In the Tag/Commit filed, add the Git revision (tag or
hash) to check out. The default is
- Click Show advanced options.
- In the Policy directory field, add the path within the repository to the top of the policy hierarchy to sync. The default is the root directory of the repository.
- In the Sync wait field, add the period in seconds between consecutive syncs. The default is 15 seconds.
Click Done. You are taken back to the Anthos Config Management menu. After a few minutes, refresh the page and you should see
Syncedin the status column next to the clusters you configured.
Verifying the installation
You can use the
nomos status command to check if the Operator
is installed successfully. A valid installation with no problems has a status of
SYNCED. An invalid or incomplete installation has a status of
NOT INSTALLED OR
NOT CONFIGURED. The output also includes any reported
When the Operator is deployed successfully, it runs in a Pod
whose name begins with
config-management-operator, in the
namespace. The Pod may take a few moments to initialize. Verify that the Pod is
kubectl -n kube-system get pods | grep config-management
If the Pod is running, the command's response is similar (but not identical) to the following:
config-management-operator-6f988f5fdd-4r7tr 1/1 Running 0 26s
You can also verify that the
config-management-system namespace exists:
kubectl get ns | grep 'config-management-system'
The command's output is similar to the following:
config-management-system Active 1m
If the commands don't return output similar to that shown here, view the logs to see what went wrong:
kubectl -n kube-system logs -l k8s-app=config-management-operator
You can also use
kubectl get events to check if Anthos Config Management has
created any events.
kubectl get events -n kube-system
It is possible to have an invalid configuration that isn't detected right away,
such as a missing or invalid
git-creds Secret. For troubleshooting steps, see
Valid but incorrect ConfigManagement object
in the Troubleshooting section of this topic.
The following sections help you troubleshoot your Anthos Config Management installation.
The output of
kubectl get events might include an event with the type
FailedScheduling. The event looks like the following:
LAST SEEN TYPE REASON OBJECT MESSAGE 9s Warning FailedScheduling pod/config-management-operator-74594dc8f6 0/1 nodes are available: 1 Insufficient cpu.
To fix this error, you need to either:
- Add a node to an existing GKE node pool, or
- Create a node pool with larger nodes.
Error: attempt to grant extra privileges
kubectl apply -f config-management-operator.yaml
Error from server (Forbidden): error when creating "config-management-operator.yaml": clusterroles.rbac.authorization.k8s.io "config-management-operator" is forbidden: attempt to grant extra privileges: [...] ruleResolutionErrors=
This error indicates that the current user has fewer permissions than are required for the installation. See the Prerequisites section for role-based access control in GKE.
Valid but incorrect ConfigManagement object
If installation fails due to a problem with the ConfigManagement object that is
not due to YAML or JSON syntax error, the ConfigManagement object may be
instantiated in the cluster, but may not work correctly. In this situation, you
can use the
nomos status command to check for errors in the ConfigManagement
A valid installation with no problems has a status of
An invalid installation has a status of
NOT CONFIGURED and lists one of the
missing git-creds Secret
missing required syncRepo field
git-creds Secret is missing the key specified by secretType
Other errors may be added in the future.
To fix the problem, correct the configuration error. Depending on the type of error, you may need to re-apply the ConfigManagement manifest to the cluster.
If the problem is that you forgot to create the
git-creds Secret, the
Operator detects the Secret as soon as you create it, and you do
not need to re-apply the configuration.
This section provides general instructions for upgrading to the current version. Before upgrading, check the release notes for any specific instructions.
Run these commands for each enrolled cluster.
Download the Operator manifest and
nomoscommands for the new version.
Apply the Operator manifest:
kubectl apply -f config-management-operator.yaml
This command updates the Operator image. Kubernetes retrieves the new version and restarts the Operator Pod using the new version. When the Operator starts, it runs a reconcile loop that applies the set of manifests bundled in the new image. This updates and restarts each component Pod.
nomos.execommand on all clients with the new version. This ensures that the
nomoscommand can always get the status of all enrolled clusters and can validate configs for them.
Uninstalling the Operator from a cluster
Follow these instructions to uninstall the Operator from a cluster. You must follow these steps for each cluster that you no longer want to manage by using Anthos Config Management.
Delete the ConfigManagement object from the cluster:
kubectl delete configmanagement --all
The following things happen:
- Any ClusterRoles and ClusterRoleBindings created in the cluster by Anthos Config Management are deleted from the cluster.
- Any admission controller configurations installed by Anthos Config Management are deleted.
- The contents of the
config-management-systemnamespace are deleted, with the exception of the
git-credsSecret. Anthos Config Management cannot function without the
config-management-systemnamespace. Any CustomResourceDefinitions created or modified by Anthos Config Management are removed from the clusters where they were created or modified. The CustomResourceDefinitions (CRDs) required to run the Operator still exist, because from the point of view of Kubernetes, they were added by the user who installed the Operator. Information about removing these is covered in the next step.
At this point, the Operator still exists in your cluster, but does nothing. If you are using GKE on-prem, you cannot remove the Operator manually.
If you are using GKE and decide you no longer want to use Anthos Config Management at all, you can uninstall the Operator by following these steps:
Verify that the
config-management-systemnamespace is empty after deleting the ConfigManagement object in the previous step. Wait until the
kubectl -n config-management-system get allcommand returns
No resources found.
kubectl delete ns config-management-system
Delete the ConfigManagement CustomResourceDefinition:
kubectl delete crd configmanagements.configmanagement.gke.io
Delete all Anthos Config Management objects from the
kubectl -n kube-system delete all -l k8s-app=config-management-operator