Configure nodes to authenticate to a private registry

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

You can configure your Google Distributed Cloud cluster so that its worker nodes can use private registries. Node-level private registries are intended for use with your workloads to give you more control over image pulls and their related security. When you configure a cluster with the private registries as described in this document, Google Distributed Cloud updates the containerd configuration accordingly. Once your cluster is configured, all Pods on qualified nodes can use the registries without having to specify imagePullSecrets in the Pod spec.

This feature can be enabled or disabled at any time in the cluster lifecycle.

Prerequisites

To use this Preview feature, your cluster must meet the following requirements:

• This feature is for user clusters and self-managing (hybrid and standalone) clusters with worker node pools.
• The cluster version must be at 1.29.
• The Node pool version must be at 1.29 (not all node pools need to be at version 1.29, but the feature only works for node pools at version 1.29).
• The cluster must have the preview.baremetal.cluster.gke.io/private-registry: "enable" preview feature annotation.

Configure a self-managing cluster for private registries

To configure a standalone or hybrid cluster to use node-level private registries:

1. Edit the cluster configuration file to add the privateRegistries block in the credentials section:

   ---
   gcrKeyPath: baremetal/gcr.json
   sshPrivateKeyPath: .ssh/id_rsa
   ...
   privateRegistries:
     - host: REGISTRY_HOST
       caCertPath: CA_CERT_PATH
       pullCredentialConfigPath: CREDENTIALS_FILE_PATH
   ...
   ---
   apiVersion: v1
   kind: Namespace
   metadata:
     name: cluster-hybrid-basic
   ---
   apiVersion: baremetal.cluster.gke.io/v1
   kind: Cluster
   metadata:
     name: hybrid-basic
     namespace: cluster-hybrid-basic
     annotations:
       preview.baremetal.cluster.gke.io/private-registry: "enable"
       ...
   spec:
     type: hybrid
     ...
   

   Replace the following:

   • REGISTRY_HOST: the domain name or IP address of the private registry and the port. For example: 10.200.0.2:5007.

   • CA_CERT_PATH: the path of the CA cert file (server root CA). For example: /root/cert.pem. If your private registry doesn't require a private TLS certificate, then you can omit this field.

   • CREDENTIALS_FILE_PATH: the path of the file that contains the credentials to access your private registry. For example: /root/.docker/config.json. If your private registry server doesn't require credentials for authentication, then you can omit this field.

2. Apply the changes to your cluster:

   bmctl update cluster -c CLUSTER_NAME --kubeconfig=CLUSTER_KUBECONFIG
   

   Replace the following:

   • CLUSTER_NAME: the name of the cluster you want to update.

   • CLUSTER_KUBECONFIG: path of the self-managed (hybrid or standalone) cluster kubeconfig file.

Configure a user cluster for private registries

With user clusters, the private registry configuration is specified in the Cluster resource spec. Additionally, for user clusters, you need to store the private registry credentials in a Secret:

1. Create a kubernetes.io/dockerconfigjson type Kubernetes Secret for the registry credentials:

   If you want to scope the Secret to a specific namespace, add the --namespace flag to the following command to specify the name of the namespace.

   kubectl create secret docker-registry CREDS_SECRET_NAME \
       --from-file=.dockerconfigjson=CREDENTIALS_FILE_PATH \
       --kubeconfig=ADMIN_KUBECONFIG
   

   Replace the following:

   • CREDS_SECRET_NAME: the name for your Secret.

   • CREDENTIALS_FILE_PATH: the path of the Docker config file. For example, /root/.docker/config.json.

   Your Secret should look similar to the following example:

   apiVersion: v1
   data:
     .dockerconfigjson: ewoJImF1dGhzIjogewoJ...clpYSXdNak14IgoJCX0KCX0KfQ==
   kind: Secret
   metadata:
     creationTimestamp: "2024-04-28T22:06:06Z"
     name: private-registry-secret
     namespace: default
     resourceVersion: "5055821"
     ...
     annotations:
       ...
       baremetal.cluster.gke.io/mark-source: "true"
   type: kubernetes.io/dockerconfigjson
   

   If the Secret isn't in the same namespace as the cluster, add the annotation baremetal.cluster.gke.io/mark-source: "true", as shown in the preceding example.

2. If applicable, store the CA cert for the registry in a Secret.

   The Secret looks similar to the following:

   apiVersion: v1
   kind: Secret
   metadata:
     annotations:
       baremetal.cluster.gke.io/mark-source: "true"
     name: ca-9dd74fd308bac6df562c7a7b220590b5
     namespace: some-namespace
   type: Opaque
   data:
     ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2RENDQXFTZ0F3SUJBZ0lVQi
     3UGxjUzVFVk8vS0xuYjZiMHRhRFVleXJvd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2ZqRUxNQWtHQ
     ...
     QnpPTkxTRFZJVk5LMm9YV1JvNEpJY0ZoNFZ4MWRMRHpqMldEaHhrUEljWEhLdGR3dk5iS2tocU
     LUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
     ```
   

3. Edit the user cluster configuration file to enable and configure the private registry:

   1. Add the annotation preview.baremetal.cluster.gke.io/private-registry: "enable" to enable the private registry while it's in Preview.

      apiVersion: baremetal.cluster.gke.io/v1
      kind: Cluster
      metadata:
        name: user-basic
        namespace: cluster-user-basic
        resourceVersion: "766027"
        annotations:
          ...
          preview.baremetal.cluster.gke.io/private-registry: "enable"
      ...
      

   2. In the nodeConfig section of the user cluster configuration file, add the privateRegistries block:

      apiVersion: baremetal.cluster.gke.io/v1
      kind: Cluster
      metadata:
        name: user-basic
      ...
      spec:
        bypassPreflightCheck: false
      ...
        nodeConfig:
          containerRuntime: containerd
          podDensity:
            maxPodsPerNode: 250
          privateRegistries:
          - caCertSecretRef:
              name: CA_CERT_SECRET_NAME
              namespace: CA_CERT_SECRET_NAMESPACE
            host: REGISTRY_HOST
            pullCredentialSecretRef:
              name: CREDS_SECRET_NAME
              namespace: CREDS_SECRET_NAMESPACE
      

   Replace the following:

   • CA_CERT_SECRET_NAME: the name of the Secret that you created to store the CA cert. If you didn't create this secret, remove the caCertSecretRef block.

   • CA_CERT_SECRET_NAMESPACE: the name of the namespace for the CA cert Secret, if you created it.

   • REGISTRY_HOST: the domain name or IP address of the private registry and the port. For example: 10.200.0.2:5007.

   • CREDS_SECRET_NAME: the name of the kubernetes.io/dockerconfigjson type Secret for the registry credentials.

   • CREDS_SECRET_NAMESPACE: the namespace name for the Secret for the registry credentials.

4. Apply the changes to your cluster:

   bmctl update cluster -c USER_CLUSTER_NAME --kubeconfig=ADMIN_KUBECONFIG
   

   Replace the following:

   • USER_CLUSTER_NAME: the name of the cluster you are updating.

   • ADMIN_KUBECONFIG: path of the admin cluster kubeconfig file.