Manage fleet-level features

Fleets let you manage enterprise and other fleet-enabled features across multiple clusters at once. This lets you, for example, apply a common set of policies or create a single-service mesh across your fleet of clusters. This page provides an overview of how you can manage features for your fleet. For more information about configuring and using individual features, see their documentation.

If you've enabled Google Kubernetes Engine (GKE) Enterprise edition you can manage some features using the legacy Anthos pages in the console, and view some fleet feature status in the GKE console pages. All fleet users can manage features using the command line.

Some features let you create fleet-level default feature configuration for your fleet clusters. For example, you can ensure that every cluster you create in your fleet has Policy Controller installed and configured. Fleet-level default configuration is available only for users who have enabled GKE Enterprise, and is only available using the command line.

Some preview feature management commands are available only as part of the beta and alpha Google Cloud SDK components. You might need to install these components to use these commands

You can learn more about how fleet-level feature management on your clusters works "under the hood" in the Feature authorization section.

Fleet-level features

You can manage the following features at fleet level:

This list does not include all features that use or require fleets. For example, fleet Workload Identity relies on clusters being members of a fleet but does not require configuration at fleet level, and Anthos Service Mesh requires fleet membership for all control plane and setup options.

If you have enabled GKE Enterprise, you're entitled to use all fleet features as part of your GKE Enterprise entitlement. If you're a GKE on Google Cloud user who wants to pay for and use available features separately, see the relevant documentation and pricing page for any charges you might incur by using the feature. Some features are included in basic GKE pricing. You can find out more about which features are available in which environments in the Deployment options page.

Configure fleet-level settings

The following sections describe how you can enable and configure fleet-level features.

To use a feature at fleet level, you enable the feature for your fleet and then configure it for your fleet members. Some configuration (or other additional setup) is generally required to actually use the feature with your clusters and workloads.

If you have enabled GKE Enterprise, you can also create fleet-default cluster configurations for Anthos Service Mesh, security posture, and Policy Controller.

Enable and configure fleet features on individual clusters

To enable a feature for your fleet, complete the following steps:

Console

To enable a feature for your fleet:

  1. In your fleet host project, go to the Features page.
  2. Click Enable in the row for the feature you want to enable.
  3. Click the Enable... button in the details panel that displays.

gcloud

Each fleet-level feature has its own enable command. For example, to enable Anthos Service Mesh for your fleet, run the following command in your fleet host project:

gcloud container fleet mesh enable

See the Google Cloud SDK reference documentation (and its beta and alpha equivalents) for a complete list of commands, or the individual feature documentation sets for more details.

The configuration steps you follow next depend on the feature.

Configure fleet-level defaults

With GKE Enterprise, you can use the Google Cloud CLI to both enable features and create fleet-level default settings for your GKE clusters. You can create these defaults for Binary Authorization, Anthos Service Mesh, security posture, and Policy Controller.

For instructions on how to configure and edit these settings, see the following documentation:

After configuring fleet-level settings, any GKE cluster that you register during cluster creation is automatically configured with your fleet-level configurations.

View fleet feature status

To view the status of fleet-level features for your fleet, do the following in your fleet host project:

Console

Go to the Features page Enabled features are listed as Enabled in the Features list.

If you click the Details button for an enabled feature, a details panel displays. This panel shows the status of the feature on your fleet's clusters. For some features, this panel also provides links to configure or update the feature.

gcloud

Run the following command to list all currently enabled features:

gcloud container fleet features list

View the feature manager dashboard

View the feature manager dashboard to quickly the view the installation status of Anthos Service Mesh, security posture, Config Sync, and Policy Controller in your fleet clusters:

Go to feature manager

This dashboard displays how many of your fleet clusters have the following statuses:

  • Have the feature enabled
  • Have the feature enabled successfully
  • Have a warning
  • Have an error

You can also view whether fleet-level defaults are configured for the feature.

Disable a fleet-level feature

To disable a feature at fleet level, do the following in your fleet host project.

Console

  1. In your fleet host project, go to the Features page.
  2. Click Details in the row for the feature you want to disable.
  3. Click the Disable... button in the details panel that displays.

gcloud

Each fleet-level feature has its own disable command. For example, to disable Anthos Service Mesh for your fleet, run the following command in your fleet host project:

gcloud container fleet mesh disable

See the Google Cloud SDK reference documentation (and its beta and alpha equivalents) for a complete list of commands, or the individual feature documentation sets for more details.

For expected behavior after you disable a feature for your fleet, see the relevant feature documentation. In many cases, the relevant configuration still exists on your cluster but you are no longer able to centrally manage the feature using fleet commands or the Google Cloud console.

Feature authorization

In order to manage features at fleet-level, they must be authorized through role-based access control to perform their functions on clusters. Google Cloud uses a service called Feature Authorizer that automatically sets and updates permissions for fleet-enabled features, which saves you from having to set feature permissions manually on every cluster, especially when Google releases feature updates.

When you register a cluster, the manifest applied to the cluster contains a ClusterRoleBinding that gives the Feature Authorizer a cluster-admin role on the cluster, and the role is attached to a service account named service-project-number@gcp-sa-gkehub.iam.gserviceaccount.com.

When you disable a fleet-enabled feature in your project, Feature Authorizer deletes the corresponding ClusterRole and ClusterRoleBinding for the feature, which removes the feature's ability to operate on the cluster.

View Feature Authorizer in audit logs

To view Feature Authorizer activity in GKE audit logs:

  1. Open Logs Explorer in the Google Cloud console.

    Go to the Logs page

  2. Run the following advanced query:

    resource.type="k8s_cluster"
resource.labels.cluster_name="CLUSTER_NAME"
resource.labels.location="CLUSTER_LOCATION"
protoPayload.authenticationInfo.principalEmail="system:serviceaccount:gke-connect:connect-agent-sa"
protoPayload.authenticationInfo.authoritySelector="service-PROJECT_NUMBER@gcp-sa-gkehub.iam.gserviceaccount.com"

    Replace the following:

    • CLUSTER_NAME: the name of the cluster that you want to view the logs for.
    • CLUSTER_LOCATION: the Google Cloud location that the cluster was created in.
    • PROJECT_NUMBER: the Google Cloud project number for the project that owns the cluster.

For non-GKE clusters, find out where the Kubernetes audit logs are stored, and run a similar query.