You can configure your Google Distributed Cloud cluster so that its worker nodes
can use private registries. Node-level private registries are intended for use
with your workloads to give you more control over image pulls and their related
security. When you configure a cluster with the private registries as described
in this document, Google Distributed Cloud updates the containerd configuration
accordingly. Once your cluster is configured, all Pods on qualified nodes can
use the registries without having to specify imagePullSecrets
in the Pod spec.
This feature can be enabled or disabled at any time in the cluster lifecycle.
The following list shows the launch stage for this feature per version:
- 1.30: GA
- 1.29: Preview
- 1.28: Not available
Prerequisites
1.30
To use this GA feature, your cluster must meet the following requirements:
- The cluster version must be at 1.30.
- The node pool version must be at 1.29 or later (a 1.30 cluster can have node pools at version 1.28, but the feature only works for node pools at version 1.29 or later).
This feature is for user clusters and self-managing (hybrid and standalone) clusters with worker node pools, as shown in the following table:
Deployment model Supported cluster types Admin and user cluster deployment Admin cluster
User cluster 1
User cluster 2
Hybrid cluster deployment Hybrid cluster
User cluster 1
User cluster 2
Standalone cluster deployment Standalone cluster
1.29
To use this Preview feature, your cluster must meet the following requirements:
- The cluster version must be at 1.29.
- The Node pool version must be at 1.29 (not all node pools need to be at version 1.29, but the feature only works for node pools at version 1.29).
- The cluster must have the
preview.baremetal.cluster.gke.io/private-registry: "enable"
preview feature annotation. This feature is for user clusters and self-managing (hybrid and standalone) clusters with worker node pools, as shown in the following table:
Deployment model Supported cluster types Admin and user cluster deployment Admin cluster
User cluster 1
User cluster 2
Hybrid cluster deployment Hybrid cluster
User cluster 1
User cluster 2
Standalone cluster deployment Standalone cluster
Configure a self-managing cluster for private registries
To configure a standalone or hybrid cluster to use node-level private registries:
Edit the cluster configuration file to add the
privateRegistries
block in the credentials section:--- gcrKeyPath: baremetal/gcr.json sshPrivateKeyPath: .ssh/id_rsa ... privateRegistries: - host: REGISTRY_HOST caCertPath: CA_CERT_PATH pullCredentialConfigPath: CREDENTIALS_FILE_PATH ... --- apiVersion: v1 kind: Namespace metadata: name: cluster-hybrid-basic --- apiVersion: baremetal.cluster.gke.io/v1 kind: Cluster metadata: name: hybrid-basic namespace: cluster-hybrid-basic annotations: preview.baremetal.cluster.gke.io/private-registry: "enable" # Version 1.29 clusters only ... spec: type: hybrid ...
Replace the following:
REGISTRY_HOST
: the domain name or IP address of the private registry and the port. For example:10.200.0.2:5007
.CA_CERT_PATH
: the path of the CA cert file (server root CA). For example:/root/cert.pem
. If your private registry doesn't require a private TLS certificate, then you can omit this field.CREDENTIALS_FILE_PATH
: the path of the file that contains the credentials to access your private registry. For example:/root/.docker/config.json
. If your private registry server doesn't require credentials for authentication, then you can omit this field.
Apply the changes to your cluster:
bmctl update cluster -c CLUSTER_NAME --kubeconfig=CLUSTER_KUBECONFIG
Replace the following:
CLUSTER_NAME
: the name of the cluster you want to update.CLUSTER_KUBECONFIG
: path of the self-managed (hybrid or standalone) cluster kubeconfig file.
Configure a user cluster for private registries
With user clusters, the private registry configuration is specified in the Cluster resource spec. Additionally, for user clusters, you need to store the private registry credentials in a Secret:
Create a
kubernetes.io/dockerconfigjson
type Kubernetes Secret for the registry credentials:If you want to scope the Secret to a specific namespace, add the
--namespace
flag to the following command to specify the name of the namespace.kubectl create secret docker-registry CREDS_SECRET_NAME \ --from-file=.dockerconfigjson=CREDENTIALS_FILE_PATH \ --kubeconfig=ADMIN_KUBECONFIG
Replace the following:
CREDS_SECRET_NAME
: the name for your Secret.CREDENTIALS_FILE_PATH
: the path of the Docker config file. For example,/root/.docker/config.json
.
Your Secret should look similar to the following example:
apiVersion: v1 data: .dockerconfigjson: ewoJImF1dGhzIjogewoJ...clpYSXdNak14IgoJCX0KCX0KfQ== kind: Secret metadata: creationTimestamp: "2024-04-28T22:06:06Z" name: private-registry-secret namespace: default resourceVersion: "5055821" ... annotations: ... baremetal.cluster.gke.io/mark-source: "true" type: kubernetes.io/dockerconfigjson
If the Secret isn't in the same namespace as the cluster, add the annotation
baremetal.cluster.gke.io/mark-source: "true"
, as shown in the preceding example.If applicable, store the CA cert for the registry in a Secret.
The Secret looks similar to the following:
apiVersion: v1 kind: Secret metadata: annotations: baremetal.cluster.gke.io/mark-source: "true" name: ca-9dd74fd308bac6df562c7a7b220590b5 namespace: some-namespace type: Opaque data: ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2RENDQXFTZ0F3SUJBZ0lVQi 3UGxjUzVFVk8vS0xuYjZiMHRhRFVleXJvd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2ZqRUxNQWtHQ ... QnpPTkxTRFZJVk5LMm9YV1JvNEpJY0ZoNFZ4MWRMRHpqMldEaHhrUEljWEhLdGR3dk5iS2tocU LUVORCBDRVJUSUZJQ0FURS0tLS0tCg== ```
Edit the user cluster configuration file to enable and configure the private registry:
For version 1.29 clusters only, add the Preview feature annotation
preview.baremetal.cluster.gke.io/private-registry: "enable"
to enable the feature. For version 1.30 and later clusters, the private registry feature is enabled by default.apiVersion: baremetal.cluster.gke.io/v1 kind: Cluster metadata: name: user-basic namespace: cluster-user-basic resourceVersion: "766027" annotations: ... preview.baremetal.cluster.gke.io/private-registry: "enable" ...
In the
nodeConfig
section of the user cluster configuration file, add theprivateRegistries
block:apiVersion: baremetal.cluster.gke.io/v1 kind: Cluster metadata: name: user-basic ... spec: bypassPreflightCheck: false ... nodeConfig: containerRuntime: containerd podDensity: maxPodsPerNode: 250 privateRegistries: - caCertSecretRef: name: CA_CERT_SECRET_NAME namespace: CA_CERT_SECRET_NAMESPACE host: REGISTRY_HOST pullCredentialSecretRef: name: CREDS_SECRET_NAME namespace: CREDS_SECRET_NAMESPACE
Replace the following:
CA_CERT_SECRET_NAME
: the name of the Secret that you created to store the CA cert. If you didn't create this secret, remove thecaCertSecretRef
block.CA_CERT_SECRET_NAMESPACE
: the name of the namespace for the CA cert Secret, if you created it.REGISTRY_HOST
: the domain name or IP address of the private registry and the port. For example:10.200.0.2:5007
.CREDS_SECRET_NAME
: the name of thekubernetes.io/dockerconfigjson
type Secret for the registry credentials.CREDS_SECRET_NAMESPACE
: the namespace name for the Secret for the registry credentials.
Apply the changes to your cluster:
bmctl update cluster -c USER_CLUSTER_NAME --kubeconfig=ADMIN_KUBECONFIG
Replace the following:
USER_CLUSTER_NAME
: the name of the cluster you are updating.ADMIN_KUBECONFIG
: path of the admin cluster kubeconfig file.