Advanced installation options

This topic provides you with two additional installation options for installing Config Connector on your Google Kubernetes Engine (GKE) cluster:

• Manual installation: Installing with the manual method provides you with faster updates than using the add-on. This method also provides you with more configuration options. For example, you can increase the CPU limit of the Config Connector Operator.
• Namespaced mode: This method is an extension of the Config Connector installation. Namespaced mode supports managing multiple projects, each with their own Google Cloud identities.

For more information on these installation types, see Choosing an installation type.

Manually installing the Config Connector Operator

The following sections show you how to manually install the Config Connector Operator.

Before you begin

Before you manually install Config Connector Operator, complete the following steps:

• Create or identify a GKE cluster where Config Connector has not yet been installed and that has Workload Identity and Kubernetes Engine Monitoring enabled.
• Configure kubectl to connect to your cluster.

Installing the Config Connector Operator

Config Connector uses a Kubernetes Operator to keep its installation up to date. To install this Operator, complete the following steps:

1. Download the latest Config Connector Operator tar file:

   gsutil cp gs://configconnector-operator/latest/release-bundle.tar.gz release-bundle.tar.gz
   

   Note: Keep a copy of the installation bundle as you need this bundle to uninstall Config Connector.

2. Extract the tar file:

   tar zxvf release-bundle.tar.gz
   

3. Install the Config Connector operator on your cluster:

   kubectl apply -f operator-system/configconnector-operator.yaml
   

Creating an identity

Config Connector creates and manages Google Cloud resources by authenticating with a Identity and Access Management (IAM) service account and using GKE's Workload Identity to bind IAM service accounts with Kubernetes service accounts.

Note: You only need to perform these steps once per cluster.

To create the identity, complete the following steps:

1. Create an IAM service account. If you want to use an existing service account, you can use that account and skip this step.
   
   To create the service account, use the following command:

     gcloud iam service-accounts create SERVICE_ACCOUNT_NAME

   Replace SERVICE_ACCOUNT_NAME with a name for your service account.

To learn more about creating service accounts, see Creating and managing service accounts.

2. Give the IAM service account elevated permissions on your project:

   gcloud projects add-iam-policy-binding PROJECT_ID \
       --member="serviceAccount:SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com" \
       --role="roles/owner"

   Replace the following:
   • PROJECT_ID with your project ID.
   • SERVICE_ACCOUNT_NAME with your service account's name.
   Note: If you prefer to grant editor access to the project replace roles/owner with roles/editor. Granting the editor role allows most Config Connector functionality except project and organization wide configurations such as IAM modifications.
3. Create an IAM policy binding between the IAM service account and the predefined Kubernetes service account that Config Connector runs:

   gcloud iam service-accounts add-iam-policy-binding \
   SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com \
       --member="serviceAccount:PROJECT_ID.svc.id.goog[cnrm-system/cnrm-controller-manager]" \
       --role="roles/iam.workloadIdentityUser"

   Replace the following:
   • SERVICE_ACCOUNT_NAME with your service account's name.
   • PROJECT_ID with your project ID.

Configuring Config Connector

To complete the installation, create a configuration file for the ConfigConnector CustomResource, then apply it using the kubectl apply command. The Config Connector Operator installs Google Cloud Resource CRDs and Config Connector components in your cluster.

To configure the operator, complete the following steps:

1. Copy the following YAML file into a file named configconnector.yaml:

   # configconnector.yaml
   apiVersion: core.cnrm.cloud.google.com/v1beta1
   kind: ConfigConnector
   metadata:
     # the name is restricted to ensure that there is only one
     # ConfigConnector resource installed in your cluster
     name: configconnector.core.cnrm.cloud.google.com
   spec:
    mode: cluster
    googleServiceAccount: "SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com"
   

   Replace the following:
   • SERVICE_ACCOUNT_NAME with your service account's name.
   • PROJECT_ID with your project ID.
2. Apply the configuration to your cluster with kubectl apply:

     kubectl apply -f configconnector.yaml

Specifying where to create your resources

Config Connector can organize resources by project, folder, or organization, which is the same way you would organize resources with Google Cloud.

Before creating resources with Config Connector, you must configure where to create your resources. To determine where to create the resource, Config Connector uses an annotation on either the resource configuration or an existing Namespace. For more information, see Organizing resources.

If you do not have a Namespace for this purpose, create one with kubectl.

kubectl create namespace NAMESPACE

Replace NAMESPACE with your namespace name. For example config-connector.

Select a tab to choose where you want Config Connector to create resources.

Project

To create resources in a certain project, run the following command:

    kubectl annotate namespace \
    NAMESPACE cnrm.cloud.google.com/project-id=PROJECT_ID

Replace the following:

• NAMESPACE with your namespace name.
• PROJECT_ID with your Google Cloud project ID.

Folder

To create resources in a certain folder, run the following command:

    kubectl annotate namespace \
    NAMESPACE cnrm.cloud.google.com/folder-id=FOLDER_ID

Replace the following:

• NAMESPACE with your namespace name.
• FOLDER_ID with your Google Cloud folder ID.

Organization

To create resources in a certain organization, run the following command:

    kubectl annotate namespace \
    NAMESPACE cnrm.cloud.google.com/organization-id=ORGANIZATION_ID

Replace the following:

• NAMESPACE with your namespace name.
• ORGANIZATION_ID with your Google Cloud organization ID.

When you annotate your namespace, Config Connector creates resources in the corresponding project, folder or organization. To learn more about how Config Connector uses Kubernetes namespaces, see Kubernetes Namespaces and Google Cloud projects.

Verifying your installation

Config Connector runs all of its components in a namespace named cnrm-system. You can verify the Pods are ready by running the following command:

kubectl wait -n cnrm-system \
      --for=condition=Ready pod --all

If Config Connector is installed correctly, the output is similar to the following:

pod/cnrm-controller-manager-0 condition met

Note: The controller Pod can take several minutes to start.

Upgrading Config Connector

Important: Downgrading directly is not supported. To downgrade Config Connector, you must uninstall, reinstall Config Connector, then re-apply your resources.

To download and install the latest version of Config Connector operator:

gsutil cp gs://configconnector-operator/latest/release-bundle.tar.gz release-bundle.tar.gz
tar zxvf release-bundle.tar.gz
kubectl apply -f operator-system/configconnector-operator.yaml

Uninstalling Config Connector

Use kubectl delete to remove the Config Connector CRDs along with controller components:

kubectl delete ConfigConnector configconnector.core.cnrm.cloud.google.com \
    --wait=true

To uninstall the Config Connector operator, run the following command:

kubectl delete -f operator-system/configconnector-operator.yaml  --wait=true

Installing Config Connector using namespaced mode

The following sections show you how to enable namespaced mode.

Before you begin

Before configuring Config Connector to run in namespaced mode, ensure that you have enabled the Config Connector GKE add-on or manually installed Config Connector Operator.

Configure Config Connector to run in namespaced mode

To enable namespaced mode, complete the following steps::

1. Copy the following YAML manifest into a file named configconnector.yaml:

   apiVersion: core.cnrm.cloud.google.com/v1beta1
   kind: ConfigConnector
   metadata:
     # the name is restricted to ensure that there is only ConfigConnector resource installed in your cluster
     name: configconnector.core.cnrm.cloud.google.com
   spec:
    mode: namespaced
   

2. Apply the configuration to your cluster with kubectl apply:

   kubectl apply -f configconnector.yaml
   

Configure Config Connector to manage resources in your namespaces

Note: You must perform these steps for every namespace that you want Config Connector to create resources from.

In the following sections, the Google Cloud project where you install Config Connector is known as the host project, or HOST_PROJECT_ID. The other projects where you manage resources are known as the managed projects, or MANAGED_PROJECT_ID. These could be the same project if you only intend to use Config Connector to create Google Cloud resources in the same project as your cluster.

Creating a namespace

You can skip this step if you already have a namespace to use to organize Google Cloud resources.

Use kubectl to create a new namespace by running the following command:

kubectl create namespace NAMESPACE

Replace NAMESPACE with a name for the namespace.

Creating an Identity

Create an Identity and Access Management (IAM) service account and create a binding between the IAM Service Account and Config Connector's Kubernetes service account:

1. Create an IAM service account. If you have an existing service account, you can use it instead of creating a new service account. Use gcloud to create the service account by running the following command:

   gcloud iam service-accounts create NAMESPACE_GSA --project HOST_PROJECT_ID
   

   Replace the following:

   • NAMESPACE_GSA with the name of the Google service account (GSA) bound to your namespace.
   • HOST_PROJECT_ID with your host project's ID.

   To learn more about creating service accounts, see Creating and managing service accounts.

2. Give the IAM service account elevated permissions on your managed project.

   gcloud projects add-iam-policy-binding MANAGED_PROJECT_ID \
       --member="serviceAccount:NAMESPACE_GSA@HOST_PROJECT_ID.iam.gserviceaccount.com" \
       --role="roles/owner"
   

   Replace the following:

   • MANAGED_PROJECT_ID with your managed project's ID.
   • NAMESPACE_GSA with the name of the Google service account bound to your namespace.
   • HOST_PROJECT_ID with your host project's ID.
   Note: If you prefer to grant editor access to the project replace roles/owner with roles/editor. Granting the editor role allows most Config Connector functionality except Project or Organization wide configurations such as IAM modifications.

3. Create an IAM policy binding between the IAM service account and the Config Connector Kubernetes service account. You bind the service accounts by running the following gcloud command:

   gcloud iam service-accounts add-iam-policy-binding \
   NAMESPACE_GSA@HOST_PROJECT_ID.iam.gserviceaccount.com \
       --member="serviceAccount:HOST_PROJECT_ID.svc.id.goog[cnrm-system/cnrm-controller-manager-NAMESPACE]" \
       --role="roles/iam.workloadIdentityUser" \
       --project HOST_PROJECT_ID
   

   Replace the following:

   • HOST_PROJECT_ID with your host project's ID.
   • NAMESPACE_GSA with the name of the Google service account bound to your namespace.
   • NAMESPACE with your namespace.

4. Give the IAM service account permissions to publish Prometheus metrics to Google Cloud's operations suite on your host project.

   gcloud projects add-iam-policy-binding HOST_PROJECT_ID \
       --member="serviceAccount:NAMESPACE_GSA@HOST_PROJECT_ID.iam.gserviceaccount.com" \
       --role="roles/monitoring.metricWriter"
   

   Replace the following:

   • HOST_PROJECT_ID with your host project's ID.
   • NAMESPACE_GSA with the name of the Google service account bound to your namespace.

Creating a ConfigConnectorContext

To create Google Cloud resources, you need to configure Config Connector to watch your namespace by adding a ConfigConnectorContext object in the namespace you want to use.

To create a ConfigConnectorContext, complete the following steps:

1. Copy the following YAML manifest into a file named configconnectorcontext.yaml:

   apiVersion: core.cnrm.cloud.google.com/v1beta1
   kind: ConfigConnectorContext
   metadata:
     # you can only have one ConfigConnectorContext per namespace
     name: configconnectorcontext.core.cnrm.cloud.google.com
     namespace: NAMESPACE
   spec:
     googleServiceAccount: "NAMESPACE_GSA@HOST_PROJECT_ID.iam.gserviceaccount.com"
   

   Replace the following:

   • NAMESPACE with the name of your namespace.
   • NAMESPACE_GSA with the name of the Google service account bound to your namespace.
   • HOST_PROJECT_ID with your host project's ID.

2. Apply the file to your cluster with kubectl:

   kubectl apply -f configconnectorcontext.yaml
   

3. Verify that the Config Connector Operator created a Kubernetes service account for your namespace with kubectl by running the following command:

   kubectl get serviceaccount/cnrm-controller-manager-NAMESPACE  -n cnrm-system
   

   Replace NAMESPACE with the name of your namespace.

4. Verify that the Config Connector controller Pod is running for your namespace with kubectl by running the following command:

   kubectl wait -n cnrm-system \
       --for=condition=Ready pod \
       -l cnrm.cloud.google.com/component=cnrm-controller-manager \
       -l cnrm.cloud.google.com/scoped-namespace=NAMESPACE
   

   Replace NAMESPACE with the name of your namespace.

   If the Config Connector controller is running, the output is similar to:

   cnrm-controller-manager-abcdefghijk-0 condition met.
   

Configure Config Connector to no longer manage resources in your namespace

To configure Config Connector to no longer manage your namespace, you remove all Config Connector resources in your namespace and delete the ConfigConnectorContext in your namespace.

Remove the Config Connector resources in your namespace

To finalize the removal of ConfigConnectorContext, remove all Config Connector resources from your namespace.

1. To discover all Config Connector resources in your namespace, for each Config Connector Custom Resource Definition, list all resources.

   kubectl get crds --selector cnrm.cloud.google.com/managed-by-kcc=true \
   -o=jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | xargs -n 1 \
   kubectl get -o jsonpath='{range .items[*]}{" Kind: "}{@.kind}{"Name: "}{@.metadata.name}{"\n"}{end}' \
   --ignore-not-found -n NAMESPACE
   

   Replace NAMESPACE with the name of your namespace.

2. To remove all Config Connector resources, for each resource in the output of the previous step, issue a delete command.

   kubectl delete -n NAMESPACE KIND NAME
   

   Replace the following:

   • NAMESPACE: the name of your namespace
   • KIND: the kind of the resource discovered in the previous step
   • NAME: the name of the resource discovered in the previous step

Remove the ConfigConnectorContext

To configure Config Connector to no longer manage the Config Connector resources in your namespace, delete ConfigConnectorContext in your namespace.

  kubectl delete -n NAMESPACE ConfigConnectorContext configconnectorcontext.core.cnrm.cloud.google.com

Replace NAMESPACE with the name of your namespace.

The deletion of ConfigConnectorContext will not finalize until all Config Connector resources are removed from your namespace.

Upgrading from non-operator installations

Config Connector version 1.33.0 and higher only supports installation with the GKE add-on, or the operator.

To upgrade to the operator (and retain all Config Connector resources), you must remove all Config Connector system components except the CRDs, and then install the operator.

1. Run the following commands to remove Config Connector system non-CRD components:

   kubectl delete sts,deploy,po,svc,roles,clusterroles,clusterrolebindings --all-namespaces -l cnrm.cloud.google.com/system=true --wait=true
   kubectl delete validatingwebhookconfiguration abandon-on-uninstall.cnrm.cloud.google.com --ignore-not-found --wait=true
   kubectl delete validatingwebhookconfiguration validating-webhook.cnrm.cloud.google.com --ignore-not-found --wait=true
   kubectl delete mutatingwebhookconfiguration mutating-webhook.cnrm.cloud.google.com --ignore-not-found --wait=true
   

2. Install Config Connector with the GKE add-on, or the operator.

Switching from the add-on to a manual installation

When installed as an add-on, the version of Config Connector is directly tied to the GKE version that is installed.

The manual installation allows for faster updates at the cost of manual upgrades.

To switch while safely keeping all resources:

1. Disable the addon without deleting any ConfigConnector or ConfigConnectorContext objects:

   gcloud container clusters update CLUSTER_NAME --update-addons ConfigConnector=DISABLED
   

   Replace CLUSTER_NAME with the name of the cluster that you installed Config Connector on.

2. Follow the instructions to install the manual operator of the desired version.

Note: Because the namespace-id ConfigMap in the configconnector-operator-system namespace is deleted as part of disabling the addon, old cnrm-controller-manager StatefulSets are recreated.

Troubleshooting

The following section provides you with troubleshooting tips for your Config Connector installation.

Troubleshooting permissions for resource reconciliations

If Config Connector can't reconcile resources successfully and the logs contain the error message The caller does not have permission, forbidden., then Workload Identity might not be enabled on your GKE cluster and/or node pool.

To investigate, complete the following steps:

1. Save the following Pod configuration as wi-test.yaml:

   apiVersion: v1
   kind: Pod
   metadata:
     name: workload-identity-test
     namespace: cnrm-system
   spec:
     containers:
     - image: google/cloud-sdk:slim
       name: workload-identity-test
       command: ["sleep","infinity"]
     serviceAccountName: cnrm-controller-manager
   

2. Create the Pod in your GKE cluster:

   kubectl apply -f wi-test.yaml
   

3. Open an interactive session in the Pod:

   kubectl exec -it workload-identity-test \
     --namespace cnrm-system \
     -- /bin/bash
   

4. List your identity:

   gcloud auth list
   

5. Verify that the identity listed matches the Google service account bound to your resources.

   If you see the Compute Engine default service account instead, then that means that Workload Identity is not enabled on your GKE cluster and/or node pool.

6. Exit the interactive session, then delete the Pod from your GKE cluster:

   kubectl delete pod workload-identity-test \
   --namespace cnrm-system
   

What's next

• Get started with Config Connector.
• Learn how to install Config Connector as a GKE add-on.
• Learn how to install Config Connector on other Kubernetes distributions.