This page shows you how to customize the configuration of the containerd container runtime on your Google Kubernetes Engine (GKE) nodes. Before reading this document, ensure that you're familiar with what a container runtime is and why you'd want to customize it.
About containerd configuration in GKE
You can manually configure a set of options in the containerd runtime on GKE nodes that run an operating system like Container-Optimized OS. Customizing the runtime lets you configure special requirements like access to private image registries. To set these options, you create a YAML file called a runtime configuration file and pass the file to GKE when you create or update a cluster.
This method of customizing containerd lets you avoid deploying privileged
DaemonSets, which are a security risk. When you provide GKE with
a runtime configuration file, GKE recreates your nodes and
updates the containerd config.toml
file on every node with your configuration.
The configuration persists through node termination, upgrades, and recreations.
The runtime configuration file only lets you configure options in containerd. GKE also supports configuring specific kubelet options and low-level Linux kernel options by using a separate file called a node system configuration file. For more details, see Customizing node system configuration.
Limitations
You can't use a runtime configuration file to change containerd settings in Ubuntu node images. Only Container-Optimized OS with containerd is supported. This is the default node image for all GKE clusters.
Available containerd configuration options
The following table describes the options that you can configure using a runtime configuration file:
Runtime configuration file options | |
---|---|
|
Access private image registries with private credentials that you store in Secret Manager. For instructions, see Access private registries with private CA certificates. privateRegistryAccessConfig: enabled: true certificateAuthorityDomainConfig: - gcpSecretManagerCertificateConfig: secretURI: " This configuration has the following fields:
|
Apply containerd configuration to new clusters
This section shows you how to apply a containerd configuration file when you create a new GKE cluster.
Run the following command:
gcloud container clusters create-autoCLUSTER_NAME
\ --location=LOCATION
\ --scopes="cloud-platform" \ --containerd-config-from-file="PATH_TO_CONFIG_FILE
"
Replace the following:
CLUSTER_NAME
: the name of your new cluster.LOCATION
: the Compute Engine location of your new cluster.PATH_TO_CONFIG_FILE
: the path to the configuration file that you created, like~/containerd-configuration.yaml
.
You can enable private registry configuration on new Standard clusters by running the
gcloud container clusters create
command with the same options.
Apply containerd configuration to existing clusters
This section shows you how to apply a containerd configuration to existing clusters and nodes.
Check access scopes
Existing clusters must have the cloud-platform
access scope to use
this feature. This section shows you how to check your access scopes and update an existing
cluster with a new or modified private registry configuration file.
For details about the default access scopes in new clusters, see Access scopes in GKE.
Check Autopilot access scopes
Run the following command:
gcloud container clusters describeCLUSTER_NAME
\ --location=LOCATION
\ --flatten=nodeConfig \ --format='csv[delimiter="\\n",no-heading](oauthScopes)'
If your cluster doesn't have the https://www.googleapis.com/auth/cloud-platform
access scope, create a new cluster with this access scope.
Check Standard access scopes
To check your Standard cluster access scopes, check a node pool:
gcloud container node-pools describeNODE_POOL_NAME
\ --cluster=CLUSTER_NAME
\ --location=LOCATION
\ --flatten=nodeConfig \ --format='csv[delimiter="\\n",no-heading](oauthScopes)'
Replace NODE_POOL_NAME
with the name of the node pool.
If your cluster doesn't have the
https://www.googleapis.com/auth/cloud-platform
access scope, create a new node pool
with the cloud-platform
access scope and delete your existing node pool.
Update the cluster to use your configuration file
Run the following command:
gcloud container clusters updateCLUSTER_NAME
\ --location=LOCATION
\ --containerd-config-from-file="PATH_TO_CONFIG_FILE
"
Recreate nodes in Standard clusters
If your Standard cluster doesn't use automatic upgrades, you must manually recreate your node pools to apply the new configuration. To trigger a manual node recreation, upgrade your cluster to the same GKE version that it already uses.
gcloud container clusters upgradeCLUSTER_NAME
\ --location=LOCATION
\ --cluster-version=VERSION
Replace VERSION
with the same GKE patch version that the
cluster already uses.
Disable containerd configuration options
To remove your custom configuration, do the following:
-
Update your configuration file to specify
enabled: false
in the configuration item that you want to disable and delete any other fields in the item, like in the following example:privateRegistryAccessConfig: enabled: false
- Apply the updated configuration file to your cluster. For instructions, see Apply containerd configuration to existing clusters.