This page shows you how to create an Anthos on bare metal user cluster on the Google Cloud console. A user cluster requires an admin cluster to manage it. The script from the first section creates an Anthos on bare metal admin cluster and helps you get set up to create a user cluster. The script creates Compute Engine Virtual Machines (VMs), configures a Virtual Extensible LAN (VXLAN) overlay network between the VMs, and installs the admin cluster on one of the VMs. Then you use the Google Cloud console to create the user cluster on the remaining VMs created by the script. You can try out Anthos clusters on bare metal quickly and without having to prepare any hardware. Completing the steps on this page provides you with a working Anthos clusters on bare metal test environment that runs on Compute Engine.
Before you begin
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
-
Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
-
Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.
- Make a note of the project ID because you need it to set an environment variable that is used in the script and commands on this page. If you selected an existing project, make sure that you are either a project owner or editor.
-
You can run the script on Cloud Shell or your
local machine running Linux or macOS. If you aren't using
Cloud Shell:
- Make sure you have installed the latest
Google Cloud CLI, the command line tool for
interacting with Google Cloud. If you already have gcloud CLI
installed, update its components by running the following command:
gcloud components update
Depending on how the gcloud CLI was installed, you might see the following message: "You cannot perform this action because the Google Cloud CLI component manager is disabled for this installation. You can run the following command to achieve the same result for this installation:" Follow the instructions to copy and paste the command to update the components.
- Make sure you have
kubectl
installed. If you need to installkubectl
, run the following command:gcloud components install kubectl
- Make sure you have installed the latest
Google Cloud CLI, the command line tool for
interacting with Google Cloud. If you already have gcloud CLI
installed, update its components by running the following command:
Create the VM infrastructure and admin cluster
Do the following steps to get set up and run the script. The script that you download and run is from the anthos-samples repository. Note that it takes about 15 to 20 minutes to create and configure the Compute Engine VMs and create the admin cluster. If you want to learn more about the script before you run it, see the next section, About the script.
Setup environment variables:
export PROJECT_ID=PROJECT_ID export ZONE=ZONE export ADMIN_CLUSTER_NAME=ADMIN_CLUSTER_NAME export BMCTL_VERSION=1.14.1
For the
ZONE
, you can useus-central1-a
or any of the other Compute Engine zones.Run the following commands to log in with your Google account and set the default project and zone.
gcloud auth login gcloud config set project $PROJECT_ID gcloud config set compute/zone $ZONE
Clone the
anthos-samples
repository and change to the directory where the script is located:git clone https://github.com/GoogleCloudPlatform/anthos-samples cd anthos-samples/anthos-bm-gcp-bash
Run the script:
bash install_admin_cluster.sh
The script outputs each command it runs and the status. When it finishes, the script outputs the following:
✅ Installation complete. Please check the logs for any errors!!! ✅ If you do not see any errors in the output log, then you now have the following setup: |---------------------------------------------------------------------------------------------------------| | VM Name | L2 Network IP (VxLAN) | INFO | |---------------------------------------------------------------------------------------------------------| | abm-admin-cluster-cp1 | 10.200.0.3 | Has control plane of admin cluster running inside | | abm-user-cluster-cp1 | 10.200.0.4 | 🌟 Ready for use as control plane for the user cluster | | abm-user-cluster-w1 | 10.200.0.5 | 🌟 Ready for use as worker for the user cluster | | abm-user-cluster-w2 | 10.200.0.6 | 🌟 Ready for use as worker for the user cluster | |---------------------------------------------------------------------------------------------------------|
When you create the user cluster in the Google Cloud console, you use the following IP addresses from the earlier output:
- abm-user-cluster-cp1: Use this IP address for the Control plane node.
- abm-user-cluster-w1: Use this IP address when you configure the default node pool.
- abm-user-cluster-w2: Use this IP address after the cluster is created, to add a node pool to the user cluster.
About the script
This section describes what the
install_admin_cluster.sh
script does and
provides some background information on the admin/user cluster deployment model.
About the script
The script automates the following manual steps:
-
Creates a service account called
baremetal-gcr
, and grants the service account additional permissions to avoid needing multiple service accounts for different APIs and services. -
Enables the following Google Cloud APIs:
anthos.googleapis.com anthosaudit.googleapis.com anthosgke.googleapis.com cloudresourcemanager.googleapis.com connectgateway.googleapis.com container.googleapis.com gkeconnect.googleapis.com gkehub.googleapis.com serviceusage.googleapis.com stackdriver.googleapis.com monitoring.googleapis.com logging.googleapis.com opsconfigmonitoring.googleapis.com
-
Creates the following VMs:
- One VM for the admin workstation. An admin workstation hosts command-line interface (CLI) tools and configuration files to provision clusters during installation, and CLI tools for interacting with provisioned clusters post-installation. The admin workstation will have access to all the other nodes in the cluster via SSH.
- One VM for the control plane node of the admin cluster. An admin cluster manages user clusters, helping with creation, updates, and deletion of user clusters.
- Two VMs for the worker nodes of the user cluster. A user cluster hosts the user workloads that you deploy.
- One VM for the control plane node of the user cluster. The user cluster requires a control plane node.
- Creates a Virtual Extensible LAN (VXLAN) overlay network for layer 2 connectivity between the VMs. The network is setup to be on the 10.200.0.0/24 subnet. The layer 2 connectivity is a requirement for the bundled load balancer.
-
Installs the following tools on the admin workstation:
bmctl
kubectl
- Docker
The script also downloads the service account key for the
baremetal-gcr
service account to the admin workstation. -
Ensures that
root@10.200.0.x
from the admin workstation works by doing the following tasks:- Generate a new SSH key on the admin workstation.
- Adds the public key to all the other VMs in the deployment.
-
Creates the admin cluster by running the following command:
The script uses SSH to log in to the admin workstation as the root user. Next, the script runs the
bmctl
command-line tool to create the admin cluster. This is the same tool that you use to create clusters. Although you can create user clusters withbmctl
, this page shows you how to create the user cluster using the Google Cloud console.When Anthos clusters on bare metal creates clusters, it deploys a Kubernetes in Docker (kind) cluster on the admin workstation. This bootstrap cluster hosts the Kubernetes controllers needed to create clusters and is used to create the admin cluster. Upon creation, relevant controllers are moved from the bootstrap cluster into the admin cluster. Finally, unless you specify otherwise, the bootstrap cluster is removed when cluster creation completes successfully. The bootstrap cluster requires Docker to pull container images.
The admin/user cluster deployment model separates the admin cluster, user cluster control plane, and user cluster worker nodes onto separate node machines. You use this model for data center environments at scale, as it provides greater fault tolerance by isolating the control plane from the worker nodes. The script creates and configures four nodes, which is suitable for an admin/user cluster testing environment. In a production environment, you would add additional node machines to configure the clusters for a high-availability (HA) deployment.
Verify the admin cluster
You can find your admin cluster's kubeconfig
file on the admin workstation in
the bmctl-workspace
directory of the root account. To verify your deployment,
complete the following steps.
SSH into the admin workstation as root:
gcloud compute ssh root@abm-ws --zone ${ZONE}
You can ignore any messages about updating the VM and complete this tutorial. If you plan to keep the VMs as a test environment, you might want to update the OS or upgrade to the next release as described in the Ubuntu documentation.
Set the
KUBECONFIG
environment variable with the path to the cluster's configuration file to runkubectl
commands on the cluster.export clusterid=ADMIN_CLUSTER_NAME export KUBECONFIG=$HOME/bmctl-workspace/$clusterid/$clusterid-kubeconfig kubectl get nodes
The output is similar to the following:
NAME STATUS ROLES AGE VERSION abm-admin-cluster-cp Ready control-plane,master 91m v1.24.2-gke.1900
Set the current context in an environment variable:
export CONTEXT="$(kubectl config current-context)"
Run the following
gcloud
command. This command does the following:- Grants your user account the Kubernetes
clusterrole/cluster-admin
role on the cluster. - Configures the cluster so that you can run
kubectl
commands on your local computer without having to SSH to the admin workstation.
Replace
GOOGLE_ACCOUNT_EMAIL
with the email address that is associated with your Google Cloud account. For example:--users=alex@example.com
.gcloud container fleet memberships generate-gateway-rbac \ --membership=ADMIN_CLUSTER_NAME \ --role=clusterrole/cluster-admin \ --users=GOOGLE_ACCOUNT_EMAIL \ --project=PROJECT_ID \ --kubeconfig=$KUBECONFIG \ --context=$CONTEXT\ --apply
The output of this command is similar to the following, which is truncated for readability:
Validating input arguments. Specified Cluster Role is: clusterrole/cluster-admin Generated RBAC policy is: -------------------------------------------- ... Applying the generate RBAC policy to cluster with kubeconfig: /root/bmctl-workspace/ADMIN_CLUSTER_NAME/ADMIN_CLUSTER_NAME-kubeconfig, context: ADMIN_CLUSTER_NAME-admin@ADMIN_CLUSTER_NAME Writing RBAC policy for user: GOOGLE_ACCOUNT_EMAIL to cluster. Successfully applied the RBAC policy to cluster.
- Grants your user account the Kubernetes
When you are finished exploring, enter exit to log out of the admin workstation.
Run the following command on your local computer to get the
kubeconfig
entry that can access the cluster through the Connect gateway:gcloud container fleet memberships get-credentials ADMIN_CLUSTER_NAME
The output is similar to the following:
Starting to build Gateway kubeconfig... Current project_id: PROJECT_ID A new kubeconfig entry "connectgateway_PROJECT_ID_global_ADMIN_CLUSTER_NAME" has been generated and set as the current context.
You can now run
kubectl
commands through the Connect gateway:kubectl get nodes kubectl get namespaces
The output is similar to the following:
NAME STATUS ROLES AGE VERSION abm-admin-cluster-cp Ready control-plane,master 94m v1.24.2-gke.1900 NAME STATUS AGE anthos-creds Active 91m anthos-identity-service Active 94m capi-kubeadm-bootstrap-system Active 91m capi-system Active 91m cert-manager Active 94m cluster-admin-cluster-1 Active 91m default Active 94m gke-connect Active 92m gke-managed-metrics-server Active 94m kube-node-lease Active 94m kube-public Active 94m kube-system Active 94m vm-system Active 94m
Create the user cluster in the console
Do the following steps to create a user cluster in the Google Cloud console:
In the Google Cloud console, go to the Anthos clusters page.
Make sure that the Cloud project in which you created the admin cluster is selected. You should see the admin cluster on the list.
The selected project is used as the fleet host project. This must be the same project that the admin cluster is registered to, which is set in the
gkeConnect.projectID
field in the admin cluster configuration file. After the user cluster is created, it is automatically registered to the selected project's fleet.Click Create Cluster.
In the dialog box, click On-premises.
Next to Bare metal, click Configure.
Cluster basics
On the Cluster basics page:
- Enter a name for the user cluster or use the default.
Make sure that the newly created admin cluster is selected.
You can use the defaults for the rest of the settings on this page.
Click Continue to go to the Networking page.
Networking
When the script created the L2 VXLAN for the VMs, it assigned the following IP addresses in the 10.200.0.0/24 network:
VM Name | Network IP | Node description |
---|---|---|
abm-admin-cluster-cp1 | 10.200.0.3 | Control plane node for the admin cluster |
abm-user-cluster-cp1 | 10.200.0.4 | Control plane node for the user cluster |
abm-user-cluster-w1 | 10.200.0.5 | Worker node for the user cluster |
abm-user-cluster-w2 | 10.200.0.6 | Another worker node for the user cluster |
In the Control plane section, enter the following in the Control plane node IP 1 field:
10.200.0.4
This is the IP address of the abm-user-cluster-cp1 VM in the VXLAN created by the script.
In the Load balancer section, use the default load balancer, Bundled with MetalLB.
In the New address pool section, enter the following IP address range in the IP address range 1 field:
10.200.0.51-10.200.0.70
Click Done.
In the Virtual IPs section, enter the following IP address in the Control Plane VIP field:
10.200.0.50
Enter the following IP address for the Ingress VIP:
10.200.0.51
Use the default IP addresses in the Service and Pod CIDRs section.
Default node pool
In the navigation bar on the left, click default pool. Enter the following IP address in the Nodes address 1 field:
10.200.0.5
This is the IP address of the abm-user-cluster-w1 VM in the VXLAN created by the script.
Verify and create
Click Verify and Create to create the user cluster.
It takes 15 minutes or more to create the user cluster. The console displays status messages as it verifies the settings and creates the cluster.
If there is a problem with the configuration, the console displays an error message that should be clear enough for you to fix the configuration issue and try again to create the cluster.
Click Show details to display a side panel that displays additional information.
Click
to close the details panel.When the cluster is created, Cluster status: running is displayed.
Click
Clusters to go back to the Clusters page.
Connect to the user cluster
When you create a user cluster in the console, the cluster is configured with
the same Kubernetes role-based access control (RBAC) policies that you
configured for the admin cluster when you ran
gcloud container fleet memberships generate-gateway-rbac
. These RBAC
policies let you connect to the cluster using your Google Cloud identity, which
is the email address associated with your Google Cloud account. These
RBAC policies let you log in to the console without any
additional configuration.
On the Clusters page, notice that the user cluster that you created in the console has Anthos (Bare metal: User) in the Type column. This indicates that the cluster is managed by the Anthos On-Prem API, which is the API the console used to create the user cluster.
The admin cluster has External in the Type column. This indicates that the cluster isn't managed by the Anthos On-Prem API.
Click the link on the user cluster name, and on the side panel, click Login.
Select Use your Google identity to log in.
Click Login.
Repeat the same steps to log into the admin cluster as well.
Connect to the cluster on the command line
The console configures the RBAC policies for you as the user
cluster creator. These policies let you run kubectl
commands on your local
desktop using the Connect gateway's kubeconfig
.
From your local computer:
Get the
kubeconfig
entry that can access the cluster through the Connect gateway.gcloud container fleet memberships get-credentials USER_CLUSTER_NAME
The output is similar to the following:
Starting to build Gateway kubeconfig... Current project_id: PROJECT_ID A new kubeconfig entry "connectgateway_PROJECT_ID_global_USER_CLUSTER_NAME" has been generated and set as the current context.
You can now run
kubectl
commands through the Connect gateway:kubectl get nodes
The output is similar to the following:
NAME STATUS ROLES AGE VERSION abm-user-cluster-cp Ready control-plane,master 14m v1.24.2-gke.1900 abm-user-cluster-w1 Ready worker 8m28s v1.24.2-gke.1900
Add a node pool to the user cluster
In the Google Cloud console, go to the Anthos clusters page.
In the cluster list, click the name of the cluster, and then click More details in the Details panel.
Click the Nodes tab.
Click
Add Node Pool.Enter a name for the node pool.
In the Nodes address 1 field, enter the following IP address:
10.200.0.6
This is the IP address of the abm-user-cluster-w2 VM that the script created.
Click Create
Click the Nodes tab again if needed.
The new node pool shows a status of Reconciling.
Click
in the top-right corner to view the status of the node pool creation. You might have to refresh the page to see the updated status in the node pools list.You can also verify the new node using
kubectl
. You first have to run thegcloud container fleet memberships get-credentials
command as shown earlier to fetch the cluster config:kubectl get nodes
The output is similar to the following:
NAME STATUS ROLES AGE VERSION abm-user-cluster-cp Ready control-plane,master 24m v1.24.2-gke.1900 abm-user-cluster-w1 Ready worker 18m v1.24.2-gke.1900 abm-user-cluster-w2 Ready worker 52s v1.24.2-gke.1900
Clean up
Delete the user cluster
In the console, go to the Anthos clusters page.
In the list of clusters, click the user cluster.
In the Details panel, click More details.
Near the top of the window, click
Delete.When prompted to confirm, enter the cluster name and click Confirm.
Click
in the top-right corner to view the status of the deletion. You might have to refresh the page to update the clusters list.
Delete the admin cluster and VMs
Connect to the admin workstation to reset the cluster VMs to their state prior to installation and unregister the cluster from your Google Cloud project:
gcloud compute ssh root@abm-ws --zone ${ZONE} << EOF set -x export clusterid=ADMIN_CLUSTER_NAME bmctl reset -c \$clusterid EOF
Wait for the cluster to be deleted.
List all VMs that have
abm
in their name:gcloud compute instances list | grep 'abm'
Verify that you're fine with deleting all VMs that contain
abm
in the name.After you've verified, you can delete
abm
VMs by running the following command:gcloud compute instances list --format="value(name)" | \ grep 'abm' | \ xargs gcloud --quiet compute instances delete
Delete the service account:
gcloud iam service-accounts delete baremetal-gcr@PROJECT_ID.iam.gserviceaccount.com
At the confirmation prompt, enter y.