ContainerAttachedCluster

---

Property	Value
Google Cloud Service Name	Anthos Multi-Cloud
Google Cloud Service Documentation	/anthos/clusters/docs/multi-cloud/attached
Google Cloud REST Resource Name	v1.projects.locations.attachedClusters
Google Cloud REST Resource Documentation	/anthos/clusters/docs/multi-cloud/reference/rest/v1/projects.locations.attachedClusters
Config Connector Resource Short Names	gcpcontainerattachedcluster gcpcontainerattachedclusters containerattachedcluster
Config Connector Service Name	gkemulticloud.googleapis.com
Config Connector Resource Fully Qualified Name	containerattachedclusters.containerattached.cnrm.cloud.google.com
Can Be Referenced by IAMPolicy/IAMPolicyMember	No
Config Connector Default Average Reconcile Interval In Seconds	600

Prerequisites

Before you can use this resource, you must prepare the target cluster so that the multi-cloud service can connect to the target cluster. To prepare the cluster, follow the steps to deploy an install-agent into the target cluster:

1. Get the manifest yaml file for the install-agent:

gcloud container attached clusters generate-install-manifest my-cluster --location=GOOGLE_CLOUD_REGION --platform-version=PLATFORM_VERSION --output-file=manifest.yaml

Example command:

gcloud container attached clusters generate-install-manifest kcc-attached-cluster --location=us-west1 --platform-version=1.25.0-gke.5 --output-file=manifest.yaml

1. Check out the target cluster and get the kubeconfig context:

Amazon Elastic Kubernetes Service cluster: aws eks update-kubeconfig --region $AWS_REGION --name $CLUSTER

Azure Kubernetes Service cluster: az aks get-credentials -n $CLUSTER -g $AZURE_RESOURCE_GROUP

export KUBECONFIG_CONTEXT=$(kubectl config current-context)

1. Apply the manifest.yaml file to the target cluster:

(Optional if you used the previous command to switch context) kubectl use-context $KUBECONFIG_CONTEXT

kubectl apply -f manifest.yaml

You should see the following logs:

 namespace/gke-install created
 serviceaccount/gke-install-agent created
 clusterrolebinding.rbac.authorization.k8s.io/multicloud-install-agent-admin created
 deployment.apps/gke-multicloud-agent created

1. Switch back to the Google Kubernetes Engine(GKE) cluster with Config Connector installed:

Run kubectl config get-contexts to see all configured contexts. You should see at least two contexts: one associated with the target cluster, and one associated with the GKE cluster with Config Connector installed.

Run kubectl config use-context GKE_CONTEXT to switch back to the GKE context.

Custom Resource Definition Properties

Spec

Schema

annotations:
  string: string
authorization:
  adminUsers:
  - string
binaryAuthorization:
  evaluationMode: string
deletionPolicy: string
description: string
distribution: string
fleet:
  membership: string
  projectRef:
    external: string
    name: string
    namespace: string
location: string
loggingConfig:
  componentConfig:
    enableComponents:
    - string
monitoringConfig:
  managedPrometheusConfig:
    enabled: boolean
oidcConfig:
  issuerUrl: string
  jwks: string
platformVersion: string
projectRef:
  external: string
  kind: string
  name: string
  namespace: string
resourceID: string

Fields
annotations Optional	map (key: string, value: string) Optional. Annotations on the cluster. This field has the same restrictions as Kubernetes annotations. The total size of all keys and values combined is limited to 256k. Key can have 2 segments: prefix (optional) and name (required), separated by a slash (/). Prefix must be a DNS subdomain. Name must be 63 characters or less, begin and end with alphanumerics, with dashes (-), underscores (_), dots (.), and alphanumerics between.
authorization Optional	object Optional. Configuration related to the cluster RBAC settings.
authorization.adminUsers Optional	list (string) Optional. Users that can perform operations as a cluster admin. A managed ClusterRoleBinding will be created to grant the `cluster-admin` ClusterRole to the users. Up to ten admin users can be provided. For more info on RBAC, see https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles
authorization.adminUsers[] Optional	string
binaryAuthorization Optional	object Optional. Binary Authorization configuration for this cluster.
binaryAuthorization.evaluationMode Optional	string Mode of operation for binauthz policy evaluation. If unspecified, defaults to DISABLED. Possible values: ["DISABLED", "PROJECT_SINGLETON_POLICY_ENFORCE"].
deletionPolicy Optional	string Optional. Policy to determine what flags to send on delete.
description Optional	string Optional. A human readable description of this Attached cluster. Cannot be longer than 255 UTF-8 encoded bytes.
distribution Required	string Immutable. The Kubernetes distribution of the underlying attached cluster. Supported values: ["eks", "aks", "generic"].
fleet Required	object Required. Fleet configuration.
fleet.membership Optional	string Output only. The name of the managed Hub Membership resource associated to this cluster. Membership names are formatted as `projects//locations/global/membership/`.
fleet.projectRef Required	object The id of the Fleet host project where this cluster will be registered.
fleet.projectRef.external Optional	string The project of the fleet. Allowed value: The Google Cloud resource name of a `Project` resource (format: `projects/{{name}}`).
fleet.projectRef.name Optional	string Name of the project resource. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
fleet.projectRef.namespace Optional	string Namespace of the project resource. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
location Required	string Immutable. The location for the resource.
loggingConfig Optional	object Optional. Logging configuration for this cluster.
loggingConfig.componentConfig Optional	object The configuration of the logging components;
loggingConfig.componentConfig.enableComponents Optional	list (string) The components to be enabled. Possible values: ["SYSTEM_COMPONENTS", "WORKLOADS"].
loggingConfig.componentConfig.enableComponents[] Optional	string
monitoringConfig Optional	object Optional. Monitoring configuration for this cluster.
monitoringConfig.managedPrometheusConfig Optional	object Enable Google Cloud Managed Service for Prometheus in the cluster.
monitoringConfig.managedPrometheusConfig.enabled Optional	boolean Enable Managed Collection.
oidcConfig Required	object Required. OpenID Connect (OIDC) discovery information of the target cluster. Kubernetes Service Account (KSA) tokens are JWT tokens signed by the cluster API server. This field indicates how GCP services validate KSA tokens in order to allow system workloads (such as GKE Connect and telemetry agents) to authenticate back to GCP. Both clusters with public and private issuer URLs are supported. Clusters with public issuers only need to specify the 'issuerUrl' field while clusters with private issuers need to provide both 'issuerUrl' and 'jwks'.
oidcConfig.issuerUrl Required	string Immutable. A JSON Web Token (JWT) issuer URI. `issuer` must start with `https://`.
oidcConfig.jwks Optional	string Immutable, Optional. OIDC verification keys in JWKS format (RFC 7517). It contains a list of OIDC verification keys that can be used to verify OIDC JWTs. This field is required for cluster that doesn't have a publicly available discovery endpoint. When provided, it will be directly used to verify the OIDC JWT asserted by the IDP.
platformVersion Required	string Required. The platform version for the cluster (e.g. `1.30.0-gke.1`).
projectRef Required	object The ID of the project in which the resource belongs.
projectRef.external Optional	string The `projectID` field of a project, when not managed by Config Connector.
projectRef.kind Optional	string The kind of the Project resource; optional but must be `Project` if provided.
projectRef.name Optional	string The `name` field of a `Project` resource.
projectRef.namespace Optional	string The `namespace` field of a `Project` resource.
resourceID Optional	string Immutable, Optional. The ContainerAttachedCluster name. If not given, the metadata.name will be used.

Status

Schema

clusterRegion: string
conditions:
- lastTransitionTime: string
  message: string
  reason: string
  status: string
  type: string
createTime: string
errors:
- message: string
kubernetesVersion: string
observedGeneration: integer
observedState:
  fleetMembership: string
reconciling: boolean
state: string
uid: string
updateTime: string
workloadIdentityConfig:
- identityProvider: string
  issuerUri: string
  workloadPool: string

Fields
clusterRegion	string The region where this cluster runs. For EKS clusters, this is an AWS region. For AKS clusters, this is an Azure region.
conditions	list (object) Conditions represent the latest available observations of the object's current state.
conditions[]	object
conditions[].lastTransitionTime	string Last time the condition transitioned from one status to another.
conditions[].message	string Human-readable message indicating details about last transition.
conditions[].reason	string Unique, one-word, CamelCase reason for the condition's last transition.
conditions[].status	string Status is the status of the condition. Can be True, False, Unknown.
conditions[].type	string Type is the type of the condition.
createTime	string The time at which this cluster was registered.
errors	list (object) A set of errors found in the cluster.
errors[]	object
errors[].message	string Human-friendly description of the error.
kubernetesVersion	string The Kubernetes version of the cluster.
observedGeneration	integer ObservedGeneration is the generation of the resource that was most recently observed by the Config Connector controller. If this is equal to metadata.generation, then that means that the current reported status reflects the most recent desired state of the resource.
observedState	object ObservedState is the state of the resource as most recently observed in GCP.
observedState.fleetMembership	string Output only. The name of the managed Hub Membership resource associated to this cluster. Membership names are formatted as `projects//locations/global/membership/`. This field mirrors the Spec.Fleet.Membership field.
reconciling	boolean If set, there are currently changes in flight to the cluster.
state	string The current state of the cluster. Possible values: STATE_UNSPECIFIED, PROVISIONING, RUNNING, RECONCILING, STOPPING, ERROR, DEGRADED.
uid	string A globally unique identifier for the cluster.
updateTime	string The time at which this cluster was last updated.
workloadIdentityConfig	list (object) Workload Identity settings.
workloadIdentityConfig[]	object
workloadIdentityConfig[].identityProvider	string The ID of the OIDC Identity Provider (IdP) associated to the Workload Identity Pool.
workloadIdentityConfig[].issuerUri	string The OIDC issuer URL for this cluster.
workloadIdentityConfig[].workloadPool	string The Workload Identity Pool associated to the cluster.

Sample YAML(s)

Container Attached Cluster Basic

# Copyright 2023 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: containerattached.cnrm.cloud.google.com/v1beta1
kind: ContainerAttachedCluster
metadata:
  name: containerattachedcluster-sample-basic
spec:
  # Replace ${ATTACHED_CLUSTER_NAME?} with the name of the underlying attached cluster
  resourceID: ${ATTACHED_CLUSTER_NAME?}
  location: us-west1
  projectRef:
    # Replace ${PROJECT_ID?} with your Google Cloud project id
    external: ${PROJECT_ID?}
  description: "Test attached cluster basic sample"
  # Replace ${DISTRIBUTION?} with the Kubernetes distribution of the underlying attached cluster
  # Supported values: "eks", "aks".
  distribution: ${DISTRIBUTION?}
  oidcConfig:
    # Replace ${ISSUER_URL?} with the OIDC issuer URL of the underlying attached cluster
    issuerUrl: ${ISSUER_URL?}
  # Replace ${ATTACHED_CLUSTER_PLATFORM_VERSION?} with the platform version of the underlying attached cluster
  platformVersion: ${ATTACHED_CLUSTER_PLATFORM_VERSION?}
  fleet:
    projectRef:
      name: containerattachedcluster-dep-basic
---
apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: Project
metadata:
  name: containerattachedcluster-dep-basic
  annotations:
    cnrm.cloud.google.com/deletion-policy: abandon
spec:
  # Replace ${PROJECT_ID?} with your Google Cloud project id
  resourceID: ${PROJECT_ID?}
  organizationRef:
    # Replace ${ORG_ID?} with your Google Cloud ord id your project associates to
    external: "${ORG_ID?}"
  # Replace ${PROJECT_ID?} with your Google Cloud project id
  name: ${PROJECT_ID?}

Container Attached Cluster Full

# Copyright 2023 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: containerattached.cnrm.cloud.google.com/v1beta1
kind: ContainerAttachedCluster
metadata:
  name: containerattachedcluster-sample-full
spec:
  # Replace ${ATTACHED_CLUSTER_NAME?} with the name of the underlying attached cluster
  resourceID: ${ATTACHED_CLUSTER_NAME?}
  location: us-west1
  projectRef:
    # Replace ${PROJECT_ID?} with your Google Cloud project id
    external: ${PROJECT_ID?}
  description: "Test attached cluster full sample"
  # Replace ${DISTRIBUTION?} with the Kubernetes distribution of the underlying attached cluster
  # Supported values: "eks", "aks".
  distribution: ${DISTRIBUTION?}
  annotations:
    label-one: "value-one"
  authorization:
    admin_users: [ "[e-mail]", "[e-mail]"]
  oidcConfig:
    # Replace ${ISSUER_URL?} with the OIDC issuer URL of the underlying attached cluster
    issuerUrl: ${ISSUER_URL?}
  # Replace ${ATTACHED_CLUSTER_PLATFORM_VERSION?} with the platform version of the underlying attached cluster
  platformVersion: ${ATTACHED_CLUSTER_PLATFORM_VERSION?}
  fleet:
    projectRef:
      name: containerattachedcluster-dep-full
  loggingConfig:
    componentConfig:
      enableComponents: ["SYSTEM_COMPONENTS", "WORKLOADS"]
  monitoringConfig:
    managedPrometheusConfig:
      enabled: true
---
apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: Project
metadata:
  name: containerattachedcluster-dep-full
  annotations:
    cnrm.cloud.google.com/deletion-policy: abandon
spec:
  # Replace ${PROJECT_ID?} with your Google Cloud project id
  resourceID: ${PROJECT_ID?}
  organizationRef:
    # Replace ${ORG_ID?} with your Google Cloud ord id your project associates to
    external: "${ORG_ID?}"
  # Replace ${PROJECT_ID?} with your Google Cloud project id
  name: ${PROJECT_ID?}

Container Attached Cluster Ignore Errors

# Copyright 2023 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: containerattached.cnrm.cloud.google.com/v1beta1
kind: ContainerAttachedCluster
metadata:
  name: containerattachedcluster-sample-ignore-errors
spec:
  # Replace ${ATTACHED_CLUSTER_NAME?} with the name of the underlying attached cluster
  resourceID: ${ATTACHED_CLUSTER_NAME?}
  location: us-west1
  projectRef:
    # Replace ${PROJECT_ID?} with your Google Cloud project id
    external: ${PROJECT_ID?}
  description: "Test attached cluster ignore errors sample"
  # Replace ${DISTRIBUTION?} with the Kubernetes distribution of the underlying attached cluster
  # Supported values: "eks", "aks".
  distribution: ${DISTRIBUTION?}
  oidcConfig:
    # Replace ${ISSUER_URL?} with the OIDC issuer URL of the underlying attached cluster
    issuerUrl: ${ISSUER_URL?}
  # Replace ${ATTACHED_CLUSTER_PLATFORM_VERSION?} with the platform version of the underlying attached cluster
  platformVersion: ${ATTACHED_CLUSTER_PLATFORM_VERSION?}
  fleet:
    projectRef:
      name: containerattachedcluster-dep-ignore-errors
  deletionPolicy: "DELETE_IGNORE_ERRORS"
---
apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: Project
metadata:
  name: containerattachedcluster-dep-ignore-errors
  annotations:
    cnrm.cloud.google.com/deletion-policy: abandon
spec:
  # Replace ${PROJECT_ID?} with your Google Cloud project id
  resourceID: ${PROJECT_ID?}
  organizationRef:
    # Replace ${ORG_ID?} with your Google Cloud ord id your project associates to
    external: "${ORG_ID?}"
  # Replace ${PROJECT_ID?} with your Google Cloud project id
  name: ${PROJECT_ID?}

Note: If you have any trouble with instantiating the resource, refer to Troubleshoot Config Connector.