Run Autopilot workloads in GKE Standard clusters

Autopilot Standard

---

Cluster administrators and application operators can get the benefits of Google Kubernetes Engine (GKE) Autopilot, like pricing and pre-configured settings, in Standard mode clusters. This page shows you how to use ComputeClasses to deploy an Autopilot workload in a Standard cluster. You should already be familiar with the following concepts:

• GKE Autopilot mode
• GKE ComputeClasses
• Kubernetes node selectors

About Autopilot ComputeClasses

GKE provides Kubernetes custom resources named ComputeClasses that can be deployed in your cluster like any other Kubernetes resources. A ComputeClass defines a list of node configurations, like machine types or Spot VMs. You can select ComputeClasses in your workloads, which indicates to GKE that any new nodes should use one of the configurations in that list.

If a workload selects a ComputeClass that has the autopilot field enabled, GKE runs the Pods in Autopilot mode. The nodes that GKE creates are managed by Google and include many of the default Autopilot feature and security settings. For more information about the implications of running an Autopilot workload in your Standard clusters, including differences that you might notice when you deploy those workloads, see About Autopilot mode workloads in GKE Standard.

Types of Autopilot ComputeClasses

GKE provides built-in Autopilot ComputeClasses that you can use for most general-purpose workloads. You can also configure a new or existing custom ComputeClass to use Autopilot mode. The type of Autopilot ComputeClass that you use depends on whether your workloads need specific hardware, as follows:

• General-purpose workloads: use one of the built-in Autopilot ComputeClasses, which place Pods on the container-optimized compute platform.
• Workloads that require specific hardware: enable Autopilot mode for any custom ComputeClass, deploy that ComputeClass to the cluster, and select that ComputeClass in your workloads.

For more information about these options, when to use them, and the pricing for each option, see Hardware selection in Autopilot ComputeClasses.

Pricing

GKE Autopilot pricing applies to the workloads and nodes that use an Autopilot ComputeClass. The pricing model that applies depends on whether you use a built-in Autopilot ComputeClass or a custom Autopilot ComputeClass. For more information, see Pricing in "About Autopilot mode workloads in GKE Standard".

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.
  Note: For existing gcloud CLI installations, make sure to set the compute/region property. If you use primarily zonal clusters, set the compute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

• Use a GKE Standard cluster that runs version 1.33.1-gke.1107000 or later and is enrolled in the Rapid release channel. To create a new cluster, see Creating a regional cluster.
• To avoid workload rejections, learn about the requirements and security constraints of Autopilot. For more information, see predefined settings for Autopilot nodes.

Requirements

• At least one node pool in the cluster must have no node taints.

  This node pool is required to run GKE Standard system Pods that can't run on Autopilot nodes in Standard clusters because of the taints that GKE adds to those nodes.

• Shielded GKE Nodes is required, and is enabled by default.

• You must use a VPC-native cluster.

Limitations

• Only the Rapid release channel is supported.
• To update existing ComputeClass resources in the cluster to use Autopilot mode, you must recreate that ComputeClass with an updated specification. For more information, see Enable Autopilot for an existing custom ComputeClass.
• You can't use the podFamily priority rule in your own ComputeClasses. This rule is available only in built-in Autopilot ComputeClasses.
• The built-in Autopilot ComputeClasses don't support enabling Confidential GKE Nodes for your entire cluster. If you enable Confidential GKE Nodes for the cluster, any new Pods that select the built-in Autopilot ComputeClasses remain in the Pending state indefinitely.

Required roles and permissions

To get the permissions that you need to deploy ComputeClasses, ask your administrator to grant you the Kubernetes Engine Developer (roles/container.developer) IAM role on your cluster or project . For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Select an Autopilot ComputeClass in a workload

To select an Autopilot ComputeClass in a workload, use a node selector for the cloud.google.com/compute-class label. This is the same label that you use to select any other ComputeClass in GKE. The following steps show you how to create an example Deployment that selects a ComputeClass and verify that the Pods run in Autopilot mode:

1. Save the following example Deployment as autopilot-cc-deployment.yaml:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: helloweb
     labels:
       app: hello
   spec:
     selector:
       matchLabels:
         app: hello
     template:
       metadata:
         labels:
           app: hello
       spec:
         nodeSelector:
           # Replace with the name of a compute class
           cloud.google.com/compute-class: COMPUTE_CLASS 
         containers:
         - name: hello-app
           image: us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0
           ports:
           - containerPort: 8080
           resources:
             requests:
               cpu: "250m"
               memory: "4Gi"

   Replace COMPUTE_CLASS with the name of the compute class to use. This value can be any of the following:

   • One of the following built-in Autopilot ComputeClasses, which place general-purpose workloads on the Autopilot container-optimized compute platform:

     • autopilot
     • autopilot-spot

   • A ComputeClass that you created, such as the autopilot-n2-class ComputeClass from the Configure a custom Autopilot ComputeClass section.

2. Deploy the workload:

   kubectl apply -f autopilot-cc-deployment.yaml
   

3. To verify that the Pods use Autopilot, check the names of the host nodes:

   kubectl get pods -l=app=hello -o wide
   

   The output is similar to the following:

   NAME                       READY   STATUS    RESTARTS   AGE     IP             NODE                                         NOMINATED NODE   READINESS GATES
   helloweb-79b9f6f75-5wwc9   1/1     Running   0          152m    10.102.1.135   gk3-cluster-1-nap-10abc8ya1-f66c6cef-wg5g   <none>           <none>
   helloweb-79b9f6f75-9skb9   1/1     Running   0          4d3h    10.102.0.140   gk3-cluster-1-nap-10abc8ya1-632bac02-hjl6   <none>           <none>
   helloweb-79b9f6f75-h7bdv   1/1     Running   0          152m    10.102.1.137   gk3-cluster-1-nap-10abc8ya1-f66c6cef-wg5g   <none>           <none>
   

   In this output, the gk3- prefix in the Node column indicates that the node is managed by Autopilot.

Configure a custom Autopilot ComputeClass

You can configure custom ComputeClasses to use Autopilot. Use a custom Autopilot ComputeClass if your workloads require specific hardware to run optimally, like GPUs or a certain Compute Engine machine series.

If your workloads don't require specific hardware, we recommend that you use one of the built-in Autopilot ComputeClasses instead. To select a built-in Autopilot ComputeClass, see the preceding Select an Autopilot ComputeClass in a workload section.

Create a new custom Autopilot ComputeClass

1. Save the following example ComputeClass manifest as autopilot-n2-class.yaml:

   apiVersion: cloud.google.com/v1
   kind: ComputeClass
   metadata:
     name: autopilot-n2-class
   spec:
     autopilot:
       enabled: true
     priorities:
     - machineFamily: n2
       spot: true
       minCores: 64
     - machineFamily: n2
       spot: true
     - machineFamily: n2
       spot: false
     activeMigration:
       optimizeRulePriority: true
     whenUnsatisfiable: DoNotScaleUp
   

   This manifest includes the following fields:

   • autopilot: enables Autopilot mode for the ComputeClass. If you specify this field in a ComputeClass that you deploy to an Autopilot cluster, GKE ignores the field.
   • priorities: defines an array of three different N2 machine family configurations.
   • activeMigration: lets GKE migrate Pods to configurations that are higher in the list of priorities when resources become available.
   • whenUnsatisfiable: the value of DoNotScaleUp in this field makes it mandatory for GKE to use one of the rules in the priorities field for new nodes. If none of these priority rules can be met, GKE leaves the Pod in a Pending state until resources are available. DoNotScaleUp is the default and recommended value for this field.

2. Deploy the ComputeClass:

   kubectl apply -f autopilot-n2-class.yaml
   

3. Verify that the ComputeClass exists:

   kubectl get computeclasses
   

   The output is similar to the following:

   NAME                  AGE
   autopilot-n2-class    3s
   

Enable Autopilot for an existing custom ComputeClass

Best Practice:

To migrate your workloads to Autopilot mode in stages, or to immediately start node creation, create a new Autopilot ComputeClass and recreate specific workloads to select the new ComputeClass.

You can enable Autopilot in existing custom ComputeClasses that are in a Standard cluster and have the nodePoolAutoCreation.enabled field in the specification set to true. Enabling Autopilot in a compute class that's in an Autopilot cluster has no effect, because the entire cluster uses Autopilot mode.

After you enable Autopilot for an existing ComputeClass, GKE uses Autopilot to run new Pods that select the ComputeClass. If you have existing Pods on Standard nodes that select the Autopilot ComputeClass, those Pods use Autopilot only when they're recreated.

To update an existing custom ComputeClass to use Autopilot mode, follow these steps:

1. In a text editor, update the manifest file for your existing ComputeClass to add the spec.autopilot field:

   spec:
     autopilot:
       enabled: true
   

2. Replace the existing ComputeClass resource in the Kubernetes API with the updated specification:

   kubectl replace --force -f PATH_TO_UPDATED_MANIFEST
   

   Replace PATH_TO_UPDATED_MANIFEST with the path to your updated manifest file.

3. To trigger new node creation, recreate any workloads that use the compute class.

After you apply the updated manifest, any new nodes that GKE creates for this ComputeClass use Autopilot. GKE doesn't modify any existing nodes that were created prior to the update.

Verify that your workload uses Autopilot

• Get a list of the Pods in your cluster by using the kubectl get command:

  kubectl get pods --output=custom-columns=Pod:.metadata.name,Node:.spec.nodeName
  

  The output is similar to the following:

  Pod                         Node
  helloweb-84c865764b-nzhxt   gk3-cluster-1-pool-3-68fc8dca-t54b
  # lines omitted for clarity
  

  The gk3- prefix in the node name indicates that the Pod is running in Autopilot mode.

Apply an Autopilot ComputeClass by default

GKE lets you set a ComputeClass as the default for a namespace. The namespace default class applies to all Pods in that namespace that don't explicitly select a different ComputeClass. Setting an Autopilot ComputeClass as the default means that you can run all Pods in a namespace in Autopilot mode by default unless the workload selects a different option.

For more information, see Configure a default ComputeClass for a namespace.

What's next

• For the parameters that you can specify in ComputeClasses, see the ComputeClass CustomResourceDefinition.