在裸机上的 Anthos 集群中，用户集群运行您的工作负载，而在多集群架构中，用户集群由管理员集群创建和管理。
创建管理员集群后，调用 bmctl create config 命令可创建一个 YAML 文件，您可以修改该文件以定义用户集群。如需应用配置并创建用户集群，请使用 bmctl create cluster 命令。预检检查适用于使用 bmctl create cluster 命令创建的用户集群。
从管理员集群卸载工作负载可以保护敏感的管理数据（例如存储在管理员集群中的 SSH 密钥），避免无需访问这些信息的人员访问它。此外，让用户集群彼此隔离有助于为您的工作负载提供良好的通用安全性。
前提条件
- 从 Cloud Storage 下载了最新的
bmctl(gs://anthos-baremetal-release/bmctl/1.9.8/linux-amd64/bmctl)。 - 通过对与集群 API 服务器 (
controlPlaneVIP) 的访问权限使用管理员集群。 - 管理员集群节点与目标用户集群上的所有节点之间有网络连接。
- 运行
bmctl的工作站与目标用户集群中的所有节点之间有网络连接。 - 用于创建用户集群的 SSH 密钥可供用户集群中的 root 或 SUDO 用户使用。
- 在管理员集群上配置了连接注册服务帐号，以便与 Connect 搭配使用。
启用 SELinux
如果要启用 SELinux 来保护容器，则必须确保在所有宿主机上以 Enforced 模式启用 SELinux。从 Anthos clusters on Bare Metal 1.9.0 版或更高版本开始，您可以在创建集群或升级集群之前或之后启用或停用 SELinux。Red Hat Enterprise Linux (RHEL) 和 CentOS 上默认启用 SELinux。如果宿主机上停用了 SELinux，或者不确定，请参阅使用 SELinux 保护容器，了解如何启用 SELinux。
Anthos clusters on Bare Metal 仅支持 RHEL 和 CentOS 系统中的 SELinux。
创建用户集群配置文件
用于创建用户集群的配置文件与用于创建管理员集群的配置文件几乎相同。唯一的区别是需移除本地凭据配置部分，使配置成为有效的 Kubernetes 资源集合。配置部分位于文件顶部的 bmctl configuration variables 部分下方。
默认情况下，用户集群会从管理它们的管理员集群继承其凭据。您可以有选择地替换这些凭据的部分或全部。如需了解详情，请参阅用户集群配置文件示例。
使用
bmctl create config命令创建用户集群配置文件：bmctl create config -c USER_CLUSTER_NAME
例如，发出以下命令来为
user1用户集群创建配置文件：bmctl create config -c user1
该文件将写入
bmctl-workspace/user1/user1.yaml。该文件的通用路径为bmctl-workspace/CLUSTER NAME/CLUSTER_NAME.yaml修改配置文件，并进行以下更改：
从配置中移除本地凭据文件路径：
....gcrKeyPath: (path to GCR service account key)sshPrivateKeyPath: (path to SSH private key, used for node access)gkeConnectAgentServiceAccountKeyPath: (path to Connect agent service account key)gkeConnectRegisterServiceAccountKeyPath: (path to Hub registration service account key)cloudOperationsServiceAccountKeyPath: (path to Cloud Operations service account key)....更改配置以指定
user集群类型而不是admin：.... spec: # Cluster type. This can be: # 1) admin: to create an admin cluster. This can later be used to create # user clusters. # 2) user: to create a user cluster. Requires an existing admin cluster. # 3) hybrid: to create a hybrid cluster that runs admin cluster # components and user workloads. # 4) standalone: to create a cluster that manages itself, runs user # workloads, but does not manage other clusters. type: user ....确保负载平衡器 VIP 和地址池的管理员和用户集群规范互为补充，并且不会与现有集群重叠。下面显示了管理员和用户集群配置对的示例，其中指定了负载平衡和地址池：
.... # Sample admin cluster config for load balancer and address pools loadBalancer: vips: controlPlaneVIP: 10.200.0.49 ingressVIP: 10.200.0.50 addressPools: - name: pool1 addresses: - 10.200.0.50-10.200.0.70 .... .... # Sample user cluster config for load balancer and address pools loadBalancer: vips: controlPlaneVIP: 10.200.0.71 ingressVIP: 10.200.0.72 addressPools: - name: pool1 addresses: - 10.200.0.72-10.200.0.90 ....用户集群配置文件的其余部分与管理员集群配置相同。
指定集群节点的 pod 密度和容器运行时：
.... # NodeConfig specifies the configuration that applies to all nodes in the cluster. nodeConfig: # podDensity specifies the pod density configuration. podDensity: # maxPodsPerNode specifies at most how many pods can be run on a single node. maxPodsPerNode: 110 # containerRuntime specifies which container runtime to use for scheduling containers on nodes. # containerd and docker are supported. containerRuntime: containerd ....对于用户集群，
maxPodsPerNode的允许值为32-250。如果未指定，则默认值为110。集群创建后，此值无法更新。默认容器运行时环境为 containerd。或者，您也可以使用 Docker。如需详细了解如何更改运行时，请参阅我们的更改容器运行时指南。
pod 密度也受集群的可用 IP 资源的限制。如需了解详情，请参阅 pod 网络。
创建用户集群
发出 bmctl 命令应用用户集群配置并创建集群：
bmctl create cluster -c USER_CLUSTER_NAME --kubeconfig ADMIN_KUBECONFIG
请替换以下内容：
USER_CLUSTER_NAME：在上一部分中创建的集群名称。ADMIN_KUBECONFIG：管理员集群 kubeconfig 文件的路径。
例如，对于名为 user1 的用户集群和路径为 kubeconfig bmctl-workspace/admin/admin-kubeconfig 的管理员集群 kubeconfig 文件，命令如下：
bmctl create cluster -c user1 --kubeconfig bmctl-workspace/admin/admin-kubeconfig
完整的用户集群配置示例
以下是使用 bmctl 命令创建的用户集群配置文件示例。请注意，此示例配置中使用了占位符集群名称、VIP 和地址。这些信息可能不适用于您的网络。 默认情况下，凭据将从管理员集群继承。如果您要替换凭据，则需要在 configuration variables 部分提供相应的密钥路径。
# Sample user cluster config:
# ---
# To override default credentials
# gcrKeyPath: #/bmctl/bmctl-workspace/.sa-keys/my-gcp-project-anthos-baremetal-gcr.json
# sshPrivateKeyPath: /bmctl/bmctl-workspace/.ssh/id_rsa
# gkeConnectAgentServiceAccountKeyPath: #/bmctl/bmctl-workspace/.sa-keys/my-gcp-project-anthos-baremetal-connect.json
# gkeConnectRegisterServiceAccountKeyPath: #/bmctl/bmctl-workspace/.sa-keys/my-gcp-project-anthos-baremetal-register.json
# cloudOperationsServiceAccountKeyPath: #/bmctl/bmctl-workspace/.sa-keys/my-gcp-project-anthos-baremetal-cloud-ops.json
---
apiVersion: v1
kind: Namespace
metadata:
name: cluster-user1
---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
name: user1
namespace: cluster-user1
spec:
# Cluster type. This can be:
# 1) admin: to create an admin cluster. This can later be used to create user clusters.
# 2) user: to create a user cluster. Requires an existing admin cluster.
# 3) hybrid: to create a hybrid cluster that runs admin cluster components and user workloads.
# 4) standalone: to create a cluster that manages itself, runs user workloads,
# but does not manage other clusters.
type: user
# Anthos cluster version.
anthosBareMetalVersion: 1.9.8
# GKE connect configuration
gkeConnect:
projectID: GOOGLE_PROJECT_ID
# Control plane configuration
controlPlane:
nodePoolSpec:
nodes:
# Control plane node pools. Typically, this is either a single machine
# or 3 machines if using a high availability deployment.
- address: 10.200.0.4
# Cluster networking configuration
clusterNetwork:
# Pods specify the IP ranges from which pod networks are allocated.
pods:
cidrBlocks:
- 192.168.0.0/16
# Services specify the network ranges from which service virtual IPs are allocated.
# This can be any RFC 1918 range that does not conflict with any other IP range
# in the cluster and node pool resources.
services:
cidrBlocks:
- 10.96.0.0/20
# Load balancer configuration
loadBalancer:
# Load balancer mode can be either 'bundled' or 'manual'.
# In 'bundled' mode a load balancer will be installed on load balancer nodes during cluster creation.
# In 'manual' mode the cluster relies on a manually-configured external load balancer.
mode: bundled
# Load balancer port configuration
ports:
# Specifies the port the load balancer serves the Kubernetes control plane on.
# In 'manual' mode the external load balancer must be listening on this port.
controlPlaneLBPort: 443
# There are two load balancer virtual IP (VIP) addresses: one for the control plane
# and one for the L7 Ingress service. The VIPs must be in the same subnet as the load balancer nodes.
# These IP addresses do not correspond to physical network interfaces.
vips:
# ControlPlaneVIP specifies the VIP to connect to the Kubernetes API server.
# This address must not be in the address pools below.
controlPlaneVIP: 10.200.0.71
# IngressVIP specifies the VIP shared by all services for ingress traffic.
# Allowed only in non-admin clusters.
# This address must be in the address pools below.
ingressVIP: 10.200.0.72
# AddressPools is a list of non-overlapping IP ranges for the data plane load balancer.
# All addresses must be in the same subnet as the load balancer nodes.
# Address pool configuration is only valid for 'bundled' LB mode in non-admin clusters.
addressPools:
- name: pool1
addresses:
# Each address must be either in the CIDR form (1.2.3.0/24)
# or range form (1.2.3.1-1.2.3.5).
- 10.200.0.72-10.200.0.90
# A load balancer node pool can be configured to specify nodes used for load balancing.
# These nodes are part of the Kubernetes cluster and run regular workloads as well as load balancers.
# If the node pool config is absent then the control plane nodes are used.
# Node pool configuration is only valid for 'bundled' LB mode.
# nodePoolSpec:
# nodes:
# - address: <Machine 1 IP>
# Proxy configuration
# proxy:
# url: http://[username:password@]domain
# # A list of IPs, hostnames or domains that should not be proxied.
# noProxy:
# - 127.0.0.1
# - localhost
# Logging and Monitoring
clusterOperations:
# Cloud project for logs and metrics.
projectID: $GOOGLE_PROJECT_ID
# Cloud location for logs and metrics.
location: us-central1
# Whether collection of application logs/metrics should be enabled (in addition to
# collection of system logs/metrics which correspond to system components such as
# Kubernetes control plane or cluster management agents).
# enableApplication: false
# Storage configuration
storage:
# lvpNodeMounts specifies the config for local PersistentVolumes backed by mounted disks.
# These disks need to be formatted and mounted by the user, which can be done before or after
# cluster creation.
lvpNodeMounts:
# path specifies the host machine path where mounted disks will be discovered and a local PV
# will be created for each mount.
path: /mnt/localpv-disk
# storageClassName specifies the StorageClass that PVs will be created with. The StorageClass
# is created during cluster creation.
storageClassName: local-disks
# lvpShare specifies the config for local PersistentVolumes backed by subdirectories in a shared filesystem.
# These subdirectories are automatically created during cluster creation.
lvpShare:
# path specifies the host machine path where subdirectories will be created on each host. A local PV
# will be created for each subdirectory.
path: /mnt/localpv-share
# storageClassName specifies the StorageClass that PVs will be created with. The StorageClass
# is created during cluster creation.
storageClassName: local-shared
# numPVUnderSharedPath specifies the number of subdirectories to create under path.
numPVUnderSharedPath: 5
# NodeConfig specifies the configuration that applies to all nodes in the cluster.
nodeConfig:
# podDensity specifies the pod density configuration.
podDensity:
# maxPodsPerNode specifies at most how many pods can be run on a single node.
maxPodsPerNode: 250
# containerRuntime specifies which container runtime to use for scheduling containers on nodes.
# containerd and docker are supported.
containerRuntime: containerd
# KubeVirt configuration, uncomment this section if you want to install kubevirt to the cluster
# kubevirt:
# # if useEmulation is enabled, hardware accelerator (i.e relies on cpu feature like vmx or svm)
# # will not be attempted. QEMU will be used for software emulation.
# # useEmulation must be specified for KubeVirt installation
# useEmulation: false
# Authentication; uncomment this section if you wish to enable authentication to the cluster with OpenID Connect.
# authentication:
# oidc:
# # issuerURL specifies the URL of your OpenID provider, such as "https://accounts.google.com". The Kubernetes API
# # server uses this URL to discover public keys for verifying tokens. Must use HTTPS.
# issuerURL: <URL for OIDC Provider; required>
# # clientID specifies the ID for the client application that makes authentication requests to the OpenID
# # provider.
# clientID: <ID for OIDC client application; required>
# # clientSecret specifies the secret for the client application.
# clientSecret: <Secret for OIDC client application; optional>
# # kubectlRedirectURL specifies the redirect URL (required) for the gcloud CLI, such as
# # "http://localhost:[PORT]/callback".
# kubectlRedirectURL: <Redirect URL for the gcloud CLI; optional, default is "http://kubectl.redirect.invalid">
# # username specifies the JWT claim to use as the username. The default is "sub", which is expected to be a
# # unique identifier of the end user.
# username: <JWT claim to use as the username; optional, default is "sub">
# # usernamePrefix specifies the prefix prepended to username claims to prevent clashes with existing names.
# usernamePrefix: <Prefix prepended to username claims; optional>
# # group specifies the JWT claim that the provider will use to return your security groups.
# group: <JWT claim to use as the group name; optional>
# # groupPrefix specifies the prefix prepended to group claims to prevent clashes with existing names.
# groupPrefix: <Prefix prepended to group claims; optional>
# # scopes specifies additional scopes to send to the OpenID provider as a comma-delimited list.
# scopes: <Additional scopes to send to OIDC provider as a comma-separated list; optional>
# # extraParams specifies additional key-value parameters to send to the OpenID provider as a comma-delimited
# # list.
# extraParams: <Additional key-value parameters to send to OIDC provider as a comma-separated list; optional>
# # proxy specifies the proxy server to use for the cluster to connect to your OIDC provider, if applicable.
# # Example: https://user:password@10.10.10.10:8888. If left blank, this defaults to no proxy.
# proxy: <Proxy server to use for the cluster to connect to your OIDC provider; optional, default is no proxy>
# # deployCloudConsoleProxy specifies whether to deploy a reverse proxy in the cluster to allow Google Cloud
# # Console access to the on-premises OIDC provider for authenticating users. If your identity provider is not
# # reachable over the public internet, and you wish to authenticate using Google Cloud console, then this field
# # must be set to true. If left blank, this field defaults to false.
# deployCloudConsoleProxy: <Whether to deploy a reverse proxy for Google Cloud console authentication; optional>
# # certificateAuthorityData specifies a Base64 PEM-encoded certificate authority certificate of your identity
# # provider. It's not needed if your identity provider's certificate was issued by a well-known public CA.
# # However, if deployCloudConsoleProxy is true, then this value must be provided, even for a well-known public
# # CA.
# certificateAuthorityData: <Base64 PEM-encoded certificate authority certificate of your OIDC provider; optional>
# Node access configuration; uncomment this section if you wish to use a non-root user
# with passwordless sudo capability for machine login.
# nodeAccess:
# loginUser: <login user name>
---
# Node pools for worker nodes
apiVersion: baremetal.cluster.gke.io/v1
kind: NodePool
metadata:
name: node-pool-1
namespace: cluster-user1
spec:
clusterName: user1
nodes:
- address: 10.200.0.5