Cluster configuration samples

This document provides YAML samples of the most common Google Distributed Cloud cluster configurations. The sample cluster configuration files provide permutations of the following features and capabilities:

How to use the samples

This collection of YAML samples is intended primarily as an educational reference that illustrates how various features look when properly configured. If you want to use these samples to create your own clusters, you need to make changes. Many of the values used, such as the values for the storage section, are defaults and work for most clusters. However, other values, such as spec.authentication.oidc.clientID and spec.gkeConnect.projectID, are specific to your project and environment.

Familiarize yourself with the related feature documentation before attempting to use any of the YAML content provided in this document.

Features in each sample

The following table list basic configuration information for each sample:

Samples
Standalone clusters
Basic edge profile
  • Single node
  • Edge profile
  • No node pools
High-availability edge profile
  • High availability with three nodes
  • Edge profile
  • Bundled Layer 2 load balancing
  • No node pools
Hybrid clusters
Basic hybrid cluster
  • Non-high availability
  • Bundled Layer 2 load balancing
High-availability hybrid cluster
  • High availability
  • OIDC
  • Behind a proxy
  • Registry mirror
  • Private package repository
  • Bundled Layer 2 load balancing
High-availability hybrid cluster with load balancing outside the Control Plane
  • High availability
  • OIDC
  • Behind a proxy
  • Registry mirror
  • Private package repository
  • Bundled Layer 2 load balancing outside the Control Plane
Admin clusters
Basic admin cluster
  • Non-high availability
  • Bundled Layer 2 load balancing
Admin cluster with manual load balancing
  • Non-high availability
  • External, manually configured load balancing
High-availability admin cluster
  • High availability
  • OIDC
  • Behind a proxy
  • Registry mirror
  • Private package repository
  • Bundled Layer 2 load balancing
User clusters
Basic user cluster
  • Non-high availability
  • Bundled Layer 2 load balancing
High-availability user cluster with multiple node pools
  • SSH key overrides
  • High availability
  • Behind a proxy
  • Registry mirror
  • Private package repository
  • Bundled Layer 2 load balancing
  • Multiple node pools
High-availability user cluster with OIDC
  • High availability
  • OIDC
  • Behind a proxy
  • Registry mirror
  • Private package repository
  • Bundled Layer 2 load balancing
High-availability user cluster with LDAP and BGP load balancing
  • High availability
  • LDAP
  • Bundled load balancing with BGP

Standalone clusters

Note the following capabilities of a standalone cluster:

  • It can administer itself
  • It can run workloads
  • It can't create or manage other/user clusters

Standalone clusters are suited for cluster installations that require a small footprint or in situations where you want to run clusters in network-isolated partitions.

For more information about standalone clusters, see Standalone cluster deployment and Create standalone clusters.

Basic edge profile

Note the following features and options in this standalone cluster configuration:

  • Single node
  • Edge profile
  • No node pools
gcrKeyPath: baremetal/gcr.json
sshPrivateKeyPath: .ssh/id_rsa
gkeConnectAgentServiceAccountKeyPath: baremetal/connect-agent.json
gkeConnectRegisterServiceAccountKeyPath: baremetal/connect-register.json
cloudOperationsServiceAccountKeyPath: baremetal/cloud-ops.json
---
apiVersion: v1
kind: Namespace
metadata:
  name: cluster-edge-basic
---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: edge-basic
  namespace: cluster-edge-basic
spec:
  type: standalone
  profile: edge
  anthosBareMetalVersion: 1.31.0-gke.889
  gkeConnect:
    projectID: project-fleet
  controlPlane:
    nodePoolSpec:
      nodes:
      - address: 10.200.0.2
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/20
  loadBalancer:
    mode: bundled
    ports:
      controlPlaneLBPort: 443
    vips:
      controlPlaneVIP: 10.200.0.71
      ingressVIP: 10.200.0.72
    addressPools:
    - name: pool1
      addresses:
      - 10.200.0.72-10.200.0.90
  clusterOperations:
    projectID: project-fleet
    location: us-central1
  storage:
    lvpNodeMounts:
      path: /mnt/localpv-disk
      storageClassName: local-disks
    lvpShare:
      path: /mnt/localpv-share
      storageClassName: local-shared
      numPVUnderSharedPath: 5
  nodeConfig:
    podDensity:
      maxPodsPerNode: 110

High-availability edge profile

Note the following features and options in this standalone cluster configuration:

  • High availability with three nodes
  • Edge profile
  • Bundled Layer 2 load balancing
  • No node pools
gcrKeyPath: baremetal/gcr.json
sshPrivateKeyPath: .ssh/id_rsa
gkeConnectAgentServiceAccountKeyPath: baremetal/connect-agent.json
gkeConnectRegisterServiceAccountKeyPath: baremetal/connect-register.json
cloudOperationsServiceAccountKeyPath: baremetal/cloud-ops.json
---
apiVersion: v1
kind: Namespace
metadata:
  name: cluster-edge-ha
---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: edge-ha
  namespace: cluster-edge-ha
spec:
  type: standalone
  profile: edge
  anthosBareMetalVersion: 1.31.0-gke.889
  gkeConnect:
    projectID: project-fleet
  controlPlane:
    nodePoolSpec:
      nodes:
      - address: 10.200.0.2
      - address: 10.200.0.3
      - address: 10.200.0.4
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/20
  loadBalancer:
    mode: bundled
    ports:
      controlPlaneLBPort: 443
    vips:
      controlPlaneVIP: 10.200.0.71
      ingressVIP: 10.200.0.72
    addressPools:
    - name: pool1
      addresses:
      - 10.200.0.72-10.200.0.90
  clusterOperations:
    projectID: project-fleet
    location: us-central1
  storage:
    lvpNodeMounts:
      path: /mnt/localpv-disk
      storageClassName: local-disks
    lvpShare:
      path: /mnt/localpv-share
      storageClassName: local-shared
      numPVUnderSharedPath: 5
  nodeConfig:
    podDensity:
      maxPodsPerNode: 110

Hybrid clusters

Note the following capabilities of a hybrid cluster:

  • It can administer itself
  • It can run workloads
  • It can manage other user clusters

Hybrid clusters function like admin clusters that can run user workloads. Just like admin clusters, hybrid clusters can manage other user clusters. For more information about standalone clusters, see Hybrid cluster deployment and Create hybrid clusters.

Basic hybrid cluster

Note the following features and options in this hybrid cluster configuration:

  • Non-high availability
  • Bundled Layer 2 load balancing
gcrKeyPath: baremetal/gcr.json
sshPrivateKeyPath: .ssh/id_rsa
gkeConnectAgentServiceAccountKeyPath: baremetal/connect-agent.json
gkeConnectRegisterServiceAccountKeyPath: baremetal/connect-register.json
cloudOperationsServiceAccountKeyPath: baremetal/cloud-ops.json
---
apiVersion: v1
kind: Namespace
metadata:
  name: cluster-hybrid-basic
---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: hybrid-basic
  namespace: cluster-hybrid-basic
spec:
  type: hybrid
  profile: default
  anthosBareMetalVersion: 1.31.0-gke.889
  gkeConnect:
    projectID: project-fleet
  controlPlane:
    nodePoolSpec:
      nodes:
      - address: 10.200.0.2
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/20
  loadBalancer:
    mode: bundled
    ports:
      controlPlaneLBPort: 443
    vips:
      controlPlaneVIP: 10.200.0.71
      ingressVIP: 10.200.0.72
    addressPools:
    - name: pool1
      addresses:
      - 10.200.0.72-10.200.0.90
  clusterOperations:
    projectID: project-fleet
    location: us-central1
  storage:
    lvpNodeMounts:
      path: /mnt/localpv-disk
      storageClassName: local-disks
    lvpShare:
      path: /mnt/localpv-share
      storageClassName: local-shared
      numPVUnderSharedPath: 5
  nodeConfig:
    podDensity:
      maxPodsPerNode: 250
---
apiVersion: baremetal.cluster.gke.io/v1
kind: NodePool
metadata:
  name: np1
  namespace: cluster-hybrid-basic
spec:
  clusterName: hybrid-basic
  nodes:
  - address:  10.200.0.10
  - address:  10.200.0.11
  - address:  10.200.0.12

High-availability hybrid cluster

Note the following features and options in this hybrid cluster configuration:

  • High availability
  • OIDC
  • Behind a proxy
  • Registry mirror
  • Private package repository
  • Bundled Layer 2 load balancing
registryMirrors:
  - endpoint: https://10.194.2.13:5007/v2/test-namespace
    caCertPath: /root/cert.pem
    pullCredentialConfigPath: /root/dockerconfig.json
sshPrivateKeyPath: .ssh/id_rsa
gkeConnectAgentServiceAccountKeyPath: baremetal/connect-agent.json
gkeConnectRegisterServiceAccountKeyPath: baremetal/connect-register.json
cloudOperationsServiceAccountKeyPath: baremetal/cloud-ops.json
---
apiVersion: v1
kind: Namespace
metadata:
  name: cluster-hybrid-ha
---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: hybrid-ha
  namespace: cluster-hybrid-ha
spec:
  type: hybrid
  profile: default
  anthosBareMetalVersion: 1.31.0-gke.889
  gkeConnect:
    projectID: project-fleet
  controlPlane:
    nodePoolSpec:
      nodes:
      - address: 10.200.0.2
      - address: 10.200.0.3
      - address: 10.200.0.4
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/20
  proxy:
    url: http://10.194.2.140:3128
    noProxy:
    - 127.0.0.1
    - localhost
  osEnvironmentConfig:
    addPackageRepo: false
  loadBalancer:
    mode: bundled
    ports:
      controlPlaneLBPort: 443
    vips:
      controlPlaneVIP: 10.200.0.71
      ingressVIP: 10.200.0.72
    addressPools:
    - name: pool1
      addresses:
      - 10.200.0.72-10.200.0.90
  clusterOperations:
    projectID: project-fleet
    location: us-central1
  storage:
    lvpNodeMounts:
      path: /mnt/localpv-disk
      storageClassName: local-disks
    lvpShare:
      path: /mnt/localpv-share
      storageClassName: local-shared
      numPVUnderSharedPath: 5
  authentication:
    oidc:
      issuerURL: "https://infra.example.dev/adfs"
      clientID: "be654652-2c45-49ff-9d7c-3663cee9ba51"
      clientSecret: "clientSecret"
      kubectlRedirectURL: "http://localhost:44320/callback"
      username: "unique_name"
      usernamePrefix: "oidc:"
      group: "groups"
      groupPrefix: "oidc:"
      scopes: "allatclaims"
      extraParams: "resource=token-groups-claim"
      deployCloudConsoleProxy: true
      certificateAuthorityData: base64EncodedCACertificate
      proxy: http://10.194.2.140:3128
  nodeConfig:
    podDensity:
      maxPodsPerNode: 250
---
apiVersion: baremetal.cluster.gke.io/v1
kind: NodePool
metadata:
  name: np1
  namespace: cluster-hybrid-ha
spec:
  clusterName: hybrid-ha
  nodes:
  - address:  10.200.0.10
  - address:  10.200.0.11
  - address:  10.200.0.12

High-availability hybrid cluster with load balancing outside the Control Plane

Note the following features and options in this hybrid cluster configuration:

  • High availability
  • OIDC
  • Behind a proxy
  • Registry mirror
  • Private package repository
  • Bundled Layer 2 load balancing outside the Control Plane
registryMirrors:
  - endpoint: https://10.194.2.13:5007/v2/test-namespace
    caCertPath: /root/cert.pem
    pullCredentialConfigPath: /root/dockerconfig.json
sshPrivateKeyPath: .ssh/id_rsa
gkeConnectAgentServiceAccountKeyPath: baremetal/connect-agent.json
gkeConnectRegisterServiceAccountKeyPath: baremetal/connect-register.json
cloudOperationsServiceAccountKeyPath: baremetal/cloud-ops.json
---
apiVersion: v1
kind: Namespace
metadata:
  name: cluster-hybrid-ha-lb
---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: hybrid-ha-lb
  namespace: cluster-hybrid-ha-lb
spec:
  type: hybrid
  profile: default
  anthosBareMetalVersion: 1.31.0-gke.889
  gkeConnect:
    projectID: project-fleet
  controlPlane:
    nodePoolSpec:
      nodes:
      - address: 10.200.0.2
      - address: 10.200.0.3
      - address: 10.200.0.4
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/20
  proxy:
    url: http://10.194.2.140:3128
    noProxy:
    - 127.0.0.1
    - localhost
  osEnvironmentConfig:
    addPackageRepo: false
  loadBalancer:
    mode: bundled
    ports:
      controlPlaneLBPort: 443
    vips:
      controlPlaneVIP: 10.200.0.71
      ingressVIP: 10.200.0.72
    addressPools:
    - name: pool1
      addresses:
      - 10.200.0.72-10.200.0.90
    nodePoolSpec:
      nodes:
      - address: 10.200.0.5
      - address: 10.200.0.6
      - address: 10.200.0.7
  clusterOperations:
    projectID: project-fleet
    location: us-central1
  storage:
    lvpNodeMounts:
      path: /mnt/localpv-disk
      storageClassName: local-disks
    lvpShare:
      path: /mnt/localpv-share
      storageClassName: local-shared
      numPVUnderSharedPath: 5
  authentication:
    oidc:
      issuerURL: "https://infra.example.dev/adfs"
      clientID: "be654652-2c45-49ff-9d7c-3663cee9ba51"
      clientSecret: "clientSecret"
      kubectlRedirectURL: "http://localhost:44320/callback"
      username: "unique_name"
      usernamePrefix: "oidc:"
      group: "groups"
      groupPrefix: "oidc:"
      scopes: "allatclaims"
      extraParams: "resource=token-groups-claim"
      deployCloudConsoleProxy: true
      certificateAuthorityData: base64EncodedCACertificate
      proxy: http://10.194.2.140:3128
  nodeConfig:
    podDensity:
      maxPodsPerNode: 250
---
apiVersion: baremetal.cluster.gke.io/v1
kind: NodePool
metadata:
  name: np1
  namespace: cluster-hybrid-ha-lb
spec:
  clusterName: hybrid-ha-lb
  nodes:
  - address:  10.200.0.10
  - address:  10.200.0.11
  - address:  10.200.0.12

Admin clusters

An admin cluster is used to manage other user clusters. Use admin clusters if you have a fleet of clusters in the same data center that you want to manage from a centralized place, and for larger deployments that need isolation between different teams or between development and production workloads.

For more information about admin clusters, see Multi-cluster deployment and Create admin clusters.

Basic admin cluster

Note the following features and options in this admin cluster configuration:

  • Non-high availability
  • Bundled Layer 2 load balancing
gcrKeyPath: baremetal/gcr.json
sshPrivateKeyPath: .ssh/id_rsa
gkeConnectAgentServiceAccountKeyPath: baremetal/connect-agent.json
gkeConnectRegisterServiceAccountKeyPath: baremetal/connect-register.json
cloudOperationsServiceAccountKeyPath: baremetal/cloud-ops.json
---
apiVersion: v1
kind: Namespace
metadata:
  name: cluster-admin-basic
---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: admin-basic
  namespace: cluster-admin-basic
spec:
  type: admin
  profile: default
  anthosBareMetalVersion: 1.31.0-gke.889
  gkeConnect:
    projectID: project-fleet
  controlPlane:
    nodePoolSpec:
      nodes:
      - address: 10.200.0.2
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/20
  loadBalancer:
    mode: bundled
    ports:
      controlPlaneLBPort: 443
    vips:
      controlPlaneVIP: 10.200.0.71
  clusterOperations:
    projectID: project-fleet
    location: us-central1
  storage:
    lvpNodeMounts:
      path: /mnt/localpv-disk
      storageClassName: local-disks
    lvpShare:
      path: /mnt/localpv-share
      storageClassName: local-shared
      numPVUnderSharedPath: 5
  nodeConfig:
    podDensity:
      maxPodsPerNode: 250

Admin cluster with manual load balancing

Note the following features and options in this admin cluster configuration:

  • Non-high availability
  • External, manually configured load balancing
gcrKeyPath: baremetal/gcr.json
sshPrivateKeyPath: .ssh/id_rsa
gkeConnectAgentServiceAccountKeyPath: baremetal/connect-agent.json
gkeConnectRegisterServiceAccountKeyPath: baremetal/connect-register.json
cloudOperationsServiceAccountKeyPath: baremetal/cloud-ops.json
---
apiVersion: v1
kind: Namespace
metadata:
  name: cluster-admin-manlb
---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: admin-manlb
  namespace: cluster-admin-manlb
spec:
  type: admin
  profile: default
  anthosBareMetalVersion: 1.31.0-gke.889
  gkeConnect:
    projectID: project-fleet
  controlPlane:
    nodePoolSpec:
      nodes:
      - address: 10.200.0.2
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/20
  loadBalancer:
    mode: manual
    ports:
      controlPlaneLBPort: 443
    vips:
      controlPlaneVIP: 10.200.0.71
  clusterOperations:
    projectID: project-fleet
    location: us-central1
  storage:
    lvpNodeMounts:
      path: /mnt/localpv-disk
      storageClassName: local-disks
    lvpShare:
      path: /mnt/localpv-share
      storageClassName: local-shared
      numPVUnderSharedPath: 5
  nodeConfig:
    podDensity:
      maxPodsPerNode: 250

High-availability admin cluster

Note the following features and options in this admin cluster configuration:

  • High availability
  • OIDC
  • Behind a proxy
  • Registry mirror
  • Private package repository
  • Bundled Layer 2 load balancing
registryMirrors:
  - endpoint: https://10.194.2.13:5007/v2/test-namespace
    caCertPath: /root/cert.pem
    pullCredentialConfigPath: /root/dockerconfig.json
sshPrivateKeyPath: .ssh/id_rsa
gkeConnectAgentServiceAccountKeyPath: baremetal/connect-agent.json
gkeConnectRegisterServiceAccountKeyPath: baremetal/connect-register.json
cloudOperationsServiceAccountKeyPath: baremetal/cloud-ops.json
---
apiVersion: v1
kind: Namespace
metadata:
  name: cluster-admin-ha
---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: admin-ha
  namespace: cluster-admin-ha
spec:
  type: admin
  profile: default
  anthosBareMetalVersion: 1.31.0-gke.889
  gkeConnect:
    projectID: project-fleet
  controlPlane:
    nodePoolSpec:
      nodes:
      - address: 10.200.0.2
      - address: 10.200.0.3
      - address: 10.200.0.4
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/20
  proxy:
    url: http://10.194.2.140:3128
    noProxy:
    - 127.0.0.1
    - localhost
  osEnvironmentConfig:
    addPackageRepo: false
  loadBalancer:
    mode: bundled
    ports:
      controlPlaneLBPort: 443
    vips:
      controlPlaneVIP: 10.200.0.71
  clusterOperations:
    projectID: project-fleet
    location: us-central1
  storage:
    lvpNodeMounts:
      path: /mnt/localpv-disk
      storageClassName: local-disks
    lvpShare:
      path: /mnt/localpv-share
      storageClassName: local-shared
      numPVUnderSharedPath: 5
  authentication:
    oidc:
      issuerURL: "https://infra.example.dev/adfs"
      clientID: "be654652-2c45-49ff-9d7c-3663cee9ba51"
      clientSecret: "clientSecret"
      kubectlRedirectURL: "http://localhost:44320/callback"
      username: "unique_name"
      usernamePrefix: "oidc:"
      group: "groups"
      groupPrefix: "oidc:"
      scopes: "allatclaims"
      extraParams: "resource=token-groups-claim"
      deployCloudConsoleProxy: true
      certificateAuthorityData: base64EncodedCACertificate
      proxy: http://10.194.2.140:3128
  nodeConfig:
    podDensity:
      maxPodsPerNode: 250

User clusters

A user cluster runs your containerized workloads. User clusters must contain one or more worker nodes that run user workloads. Use user clusters if you have a fleet of clusters in the same data center that you want to manage from a centralized place. User clusters are also recommended for larger deployments that need isolation between different teams or between development and production workloads.

For more information about admin clusters, see Multi-cluster deployment and Create user clusters.

Basic user cluster

Note the following features and options in this user cluster configuration:

  • Non-high availability
  • Bundled Layer 2 load balancing
apiVersion: v1
kind: Namespace
metadata:
  name: cluster-user-basic
---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: user-basic
  namespace: cluster-user-basic
spec:
  type: user
  profile: default
  anthosBareMetalVersion: 1.31.0-gke.889
  gkeConnect:
    projectID: project-fleet
  controlPlane:
    nodePoolSpec:
      nodes:
      - address: 10.200.0.20
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/20
  loadBalancer:
    mode: bundled
    ports:
      controlPlaneLBPort: 443
    vips:
      controlPlaneVIP: 10.200.0.91
      ingressVIP: 10.200.0.92
    addressPools:
    - name: pool1
      addresses:
      - 10.200.0.92-10.200.0.100
  clusterOperations:
    projectID: project-fleet
    location: us-central1
  storage:
    lvpNodeMounts:
      path: /mnt/localpv-disk
      storageClassName: local-disks
    lvpShare:
      path: /mnt/localpv-share
      storageClassName: local-shared
      numPVUnderSharedPath: 5
  nodeConfig:
    podDensity:
      maxPodsPerNode: 250
---
apiVersion: baremetal.cluster.gke.io/v1
kind: NodePool
metadata:
  name: np1
  namespace: cluster-user-basic
spec:
  clusterName: user-basic
  nodes:
  - address:  10.200.0.30
  - address:  10.200.0.31
  - address:  10.200.0.32

High-availability user cluster with multiple node pools

Note the following features and options in this user cluster configuration:

  • SSH key overrides
  • High availability
  • Behind a proxy
  • Registry mirror
  • Private package repository
  • Bundled Layer 2 load balancing
  • Multiple node pools
registryMirrors:
  - endpoint: https://10.194.2.13:5007/v2/test-namespace
    caCertPath: /root/cert.pem
    pullCredentialConfigPath: /root/dockerconfig.json
---
apiVersion: v1
kind: Namespace
metadata:
  name: cluster-user-ha-np
---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: user-ha-np
  namespace: cluster-user-ha-np
spec:
  type: user
  profile: default
  anthosBareMetalVersion: 1.31.0-gke.889
  gkeConnect:
    projectID: project-fleet
  controlPlane:
    nodePoolSpec:
      nodes:
      - address: 10.200.0.20
      - address: 10.200.0.21
      - address: 10.200.0.22
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/20
  proxy:
    url: http://10.194.2.140:3128
    noProxy:
    - 127.0.0.1
    - localhost
  osEnvironmentConfig:
    addPackageRepo: false
  loadBalancer:
    mode: bundled
    ports:
      controlPlaneLBPort: 443
    vips:
      controlPlaneVIP: 10.200.0.91
      ingressVIP: 10.200.0.92
    addressPools:
    - name: pool1
      addresses:
      - 10.200.0.92-10.200.0.100
  clusterOperations:
    projectID: project-fleet
    location: us-central1
  storage:
    lvpNodeMounts:
      path: /mnt/localpv-disk
      storageClassName: local-disks
    lvpShare:
      path: /mnt/localpv-share
      storageClassName: local-shared
      numPVUnderSharedPath: 5
  nodeConfig:
    podDensity:
      maxPodsPerNode: 250
  credential:
    sshKeySecret:
      name: ssh-key
      namespace: cluster-user-ha-np
---
apiVersion: baremetal.cluster.gke.io/v1
kind: NodePool
metadata:
  name: np1
  namespace: cluster-user-ha-np
spec:
  clusterName: user-ha-np
  nodes:
  - address:  10.200.0.30
  - address:  10.200.0.31
  - address:  10.200.0.32
---
apiVersion: baremetal.cluster.gke.io/v1
kind: NodePool
metadata:
  name: np2
  namespace: cluster-user-ha-np
spec:
  clusterName: user-ha-np
  nodes:
  - address:  10.200.0.33
  - address:  10.200.0.34
  - address:  10.200.0.35

High-availability user cluster with OIDC

Note the following features and options in this user cluster configuration:

  • High availability
  • OIDC
  • Behind a proxy
  • Registry mirror
  • Private package repository
  • Bundled Layer 2 load balancing off the Control Plane
registryMirrors:
  - endpoint: https://10.194.2.13:5007/v2/test-namespace
    caCertPath: /root/cert.pem
    pullCredentialConfigPath: /root/dockerconfig.json
---
apiVersion: v1
kind: Namespace
metadata:
  name: cluster-user-ha-oidc
---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: user-ha-oidc
  namespace: cluster-user-ha-oidc
spec:
  type: user
  profile: default
  anthosBareMetalVersion: 1.31.0-gke.889
  gkeConnect:
    projectID: project-fleet
  controlPlane:
    nodePoolSpec:
      nodes:
      - address: 10.200.0.20
      - address: 10.200.0.21
      - address: 10.200.0.22
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/20
  proxy:
    url: http://10.194.2.140:3128
    noProxy:
    - 127.0.0.1
    - localhost
  osEnvironmentConfig:
    addPackageRepo: false
  loadBalancer:
    mode: bundled
    ports:
      controlPlaneLBPort: 443
    vips:
      controlPlaneVIP: 10.200.0.91
      ingressVIP: 10.200.0.92
    addressPools:
    - name: pool1
      addresses:
      - 10.200.0.92-10.200.0.100
    nodePoolSpec:
      nodes:
      - address: 10.200.0.25
      - address: 10.200.0.26
      - address: 10.200.0.27
  clusterOperations:
    projectID: project-fleet
    location: us-central1
  storage:
    lvpNodeMounts:
      path: /mnt/localpv-disk
      storageClassName: local-disks
    lvpShare:
      path: /mnt/localpv-share
      storageClassName: local-shared
      numPVUnderSharedPath: 5
  authentication:
    oidc:
      issuerURL: "https://infra.example.dev/adfs"
      clientID: "be654652-2c45-49ff-9d7c-3663cee9ba51"
      clientSecret: "clientSecret"
      kubectlRedirectURL: "http://localhost:44320/callback"
      username: "unique_name"
      usernamePrefix: "oidc:"
      group: "groups"
      groupPrefix: "oidc:"
      scopes: "allatclaims"
      extraParams: "resource=token-groups-claim"
      deployCloudConsoleProxy: true
      certificateAuthorityData: base64EncodedCACertificate
      proxy: http://10.194.2.140:3128
  nodeConfig:
    podDensity:
      maxPodsPerNode: 250
---
apiVersion: baremetal.cluster.gke.io/v1
kind: NodePool
metadata:
  name: np1
  namespace: cluster-user-ha-oidc
spec:
  clusterName: user-ha-oidc
  nodes:
  - address:  10.200.0.30
  - address:  10.200.0.31
  - address:  10.200.0.32

High-availability user cluster with LDAP and BGP load balancing

Note the following features and options in this user cluster configuration:

  • High availability
  • LDAP
  • Bundled load balancing with BGP
apiVersion: v1
kind: Namespace
metadata:
  name: cluster-user-ha-ldap
---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: user-ha-ldap
  namespace: cluster-user-ha-ldap
spec:
  type: user
  profile: default
  anthosBareMetalVersion: 1.31.0-gke.889
  gkeConnect:
    projectID: project-fleet
  controlPlane:
    nodePoolSpec:
      nodes:
      - address: 10.200.0.20
      - address: 10.200.0.21
      - address: 10.200.0.22
  clusterNetwork:
    advancedNetworking: true
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/20
  loadBalancer:
    mode: bundled
    type: bgp
    localASN: 65001
    bgpPeers:
    - ip: 10.8.0.10
      asn: 65002
    - ip: 10.8.0.11
      asn: 65002
    ports:
      controlPlaneLBPort: 443
    vips:
      controlPlaneVIP: 10.200.0.91
      ingressVIP: 10.200.0.92
    addressPools:
    - name: pool1
      addresses:
      - 10.200.0.92-10.200.0.100
  clusterOperations:
    projectID: project-fleet
    location: us-central1
  storage:
    lvpNodeMounts:
      path: /mnt/localpv-disk
      storageClassName: local-disks
    lvpShare:
      path: /mnt/localpv-share
      storageClassName: local-shared
      numPVUnderSharedPath: 5
  nodeConfig:
    podDensity:
      maxPodsPerNode: 250
  authentication:
  - name: ldap
    ldap:
      connectionType: ldaps
      group:
        baseDN: ou=Groups,dc=onpremidp,dc=example,dc=net
        filter: (objectClass=*)
        identifierAttribute: dn
      host: ldap.google.com:636
      user:
        baseDN: ou=Users,dc=onpremidp,dc=example,dc=net
        filter: (objectClass=*)
        identifierAttribute: uid
        loginAttribute: uid
      serviceAccountSecret:
        name: google-ldap-client-secret
        namespace: anthos-identity-service
        type: tls
---
apiVersion: baremetal.cluster.gke.io/v1
kind: NodePool
metadata:
  name: np1
  namespace: cluster-user-ha-ldap
spec:
  clusterName: user-ha-ldap
  nodes:
  - address:  10.200.0.30
  - address:  10.200.0.31
  - address:  10.200.0.32
---
apiVersion: networking.gke.io/v1
kind: NetworkGatewayGroup
metadata:
  name: default
  namespace: cluster-user-ha-ldap
spec:
  floatingIPs:
  - 10.0.1.100
  - 10.0.2.100