Use a registry mirror for container images

This page explains how to create and upgrade clusters using images pulled from a registry mirror rather than from a public registry, such as gcr.io. This feature can be enabled or disabled at any time in the cluster lifecycle.

This page is for Operators and Storage specialists who configure and manage storage performance, usage, and expense. To learn more about common roles and example tasks that we reference in Google Cloud content, see Common GKE Enterprise user roles and tasks.

A registry mirror is a local copy of a public registry that copies or mirrors the file structure of the public registry. For example, suppose that the path to your local registry mirror is 172.18.0.20:5000. When containerd encounters a request to pull an image such as gcr.io/kubernetes-e2e-test-images/nautilus:1.0, containerd attempts to pull that image, not from gcr.io, but from your local registry at the following path: 172.18.0.20:5000/kubernetes-e2e-test-images/nautilus:1.0. If the image isn't in this local registry path, the image is pulled automatically from the public registry gcr.io.

Using a registry mirror provides the following benefits:

• Protects you from public registry outages.
• Speeds up pod creation.
• Lets you conduct your own vulnerability scanning.
• Avoids limits imposed by public registries on how frequently you can issue commands to them.

Note: Image pulls for gcr.io/anthos-baremetal-release/kindest/node don't fall back to gcr.io. These images are pulled from the registry mirror only. If the registry mirror doesn't work, the pull fails.

Before you begin

• You must have a container registry server set up in your network.
• If your registry server runs a private TLS certificate, you must have the certificate authority (CA) file.
• If your registry server needs authentication, you must have the proper login credentials or Docker configuration file.
• To use a registry mirror, you must set the container runtime to containerd.

Download all required images for Google Distributed Cloud

Download the latest version of the bmctl tool and images package from the Downloads page.

Upload container images to your registry server

Upload the images from the images package to your registry server by running:

[HTTPS_PROXY=http://PROXY_IP:PORT] ./bmctl push images \
    --source=./bmpackages_VERSION.tar.xz \
    --private-registry=REGISTRY_IP:PORT \
    [--cacert=CERT_PATH] \
    [--need-credential=false]

Replace the following:

• PROXY_IP:PORT with the IP address and port of the proxy if you need a proxy to upload the images from your workstation to the registry server.
• VERSION with the version of the images package you downloaded from the Downloads page.
• REGISTRY_IP:PORT with the IP address and port of the private registry server.
• CERT_PATH with the path of the CA cert file if your registry server uses a private TLS certificate.

Enter your username and password when prompted or select a Docker configuration file. If your registry server doesn't require credentials, then specify --need-credential=false.

For more information on the bmctl push images command, run:

bmctl push images --help

Use your own namespace

If you want to use your own namespace in your registry server instead of the root namespace, containerd can pull from this namespace if you provide the API endpoint for your private registry in registryMirrors.endpoint. The endpoint is usually in the format of <REGISTRY_IP:PORT>/v2/<NAMESPACE>. Check your private registry user guide for specific details.

For example, if you only have access to 172.18.0.20:5000/test-namespace/, you can use the following command to upload all the images under namespace test-namespace:

./bmctl push images \
    --source=./bmpackages_VERSION.tar.xz \
    --private-registry=172.18.0.20:5000/test-namespace \
    --username=<USERNAME> \
    --password=<PASSWORD> \
    --cacert <path/to/cert.crt>

Then in the cluster configuration file, you can add the following to make containerd pull from the namespace:

registryMirrors:
  - endpoint: https://172.18.0.20:5000/v2/test-namespace

Configure clusters to use a registry mirror

You can configure a registry mirror for a cluster at cluster creation or whenever you update an existing cluster. The configuration method that you use depends on the cluster type and, for user clusters, whether you are creating a cluster or updating a cluster. The following two sections describe the two methods available for configuring registry mirrors.

Use the header section in the cluster configuration file

Google Distributed Cloud lets you specify registry mirrors outside of the Cluster spec, in the header section of the cluster configuration file:

• For admin, hybrid, and standalone clusters, the only way to configure a registry mirror is to use the header section in the cluster configuration file. This method applies for cluster creation or cluster updates.

• For user clusters, you can use this method for configuring a registry mirror during cluster creation only. Alternatively, to configure a registry mirror for an existing user cluster, you use the nodeConfig.registryMirrors section in the Cluster spec.

The following sample cluster configuration file specifies that images are to be pulled from a local registry mirror whose endpoint is https://172.18.0.20:5000. Some of the fields appearing at the beginning of this configuration file are described in the following sections.

# Sample cluster config with registry mirror:
---
gcrKeyPath: /bmctl/bmctl-workspace/.sa-keys/my-gcp-project-anthos-baremetal-gcr.json
sshPrivateKeyPath: /root/ssh-key/id_rsa
registryMirrors:
  - endpoint: https://172.18.0.20:5000
    caCertPath: /root/ca.crt
    pullCredentialConfigPath: /root/.docker/config.json
    hosts:
      - somehost.io
      - otherhost.io
---
apiVersion: v1
kind: Namespace
metadata:
  name: cluster-admin1
---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: admin1
  namespace: cluster-admin1
spec:
  nodeConfig:
    containerRuntime: containerd
...

Use the nodeConfig.registryMirrors section in the Cluster spec

This method of creating or updating a registry mirror applies to user clusters only. Because you can share the secrets created for the managing cluster with your user cluster, you can use the nodeConfig.registryMirrors from the managing admin or hybrid cluster to specify the registry mirror in the Cluster spec for the user cluster.

To configure a user cluster to use the same registry mirror as the admin cluster, follow these steps:

1. Get the nodeConfig.registryMirror section, including secret references, from nodeConfig.registryMirrors of the admin cluster resource:

   kubectl get CLUSTER_NAME -n CLUSTER_NAMESPACE \
       --kubeconfig ADMIN_KUBECONFIG \
       -o yaml
   

   Replace the following:

   • CLUSTER_NAME: the name of the admin or hybrid cluster that manages the user cluster.

   • CLUSTER_NAMESPACE: the namespace name for the managing cluster.

   • ADMIN_KUBECONFIG: the path of the kubeconfig file of the managing cluster.

2. Add the nodeConfig.registryMirrors configuration from the admin cluster to the cluster configuration file of the user cluster.

   The registryMirrors section in the user cluster configuration file should look similar to the following example:

   ---
   gcrKeyPath: /bmctl/bmctl-workspace/.sa-keys/my-gcp-project-anthos-baremetal-gcr.json
   sshPrivateKeyPath: /root/ssh-key/id_rsa
   ---
   apiVersion: v1
   kind: Namespace
   metadata:
     name: cluster-user1
   ---
   apiVersion: baremetal.cluster.gke.io/v1
   kind: Cluster
   metadata:
     name: user1
     namespace: cluster-user1
   spec:
     nodeConfig:
       containerRuntime: containerd
       registryMirrors:
       -   caCertSecretRef:
           name: the-secret-created-for-the-admin-cluster
           namespace: anthos-creds
         endpoint: https://172.18.0.20:5000
         hosts:
           -   somehost.io
           -   otherhost.io
         pullCredentialSecretRef:
           name: the-image-pull-creds-created-for-the-admin-cluster
           namespace: anthos-creds
   ...
   

To make subsequent changes to the registry mirror configuration for your user cluster, edit the nodeConfig.registryMirrors in the user cluster configuration file and apply your changes with bmctl update.

You can't use the header section of the cluster configuration file to update the registry mirrors configuration for a user cluster.

hosts field

containerd checks the hosts section of the cluster configuration file to discover which hosts are mirrored locally. In the example configuration file, the public registries listed in the hosts section are somehost.io and otherhost.io. Because these public registries appear in the hosts section, containerd checks the private registry mirror first when it encounters pull requests for images from somehost.io or otherhost.io.

For example, suppose containerd receives a pull command to somehost.io/kubernetes-e2e-test-images/nautilus:1.0. Because somehost.io is listed as one of the hosts in the hosts section of the cluster configuration file, containerd looks in the local registry mirror for the kubernetes-e2e-test-images/nautilus:1.0 image. If somehost.io isn't listed in the hosts section, containerd doesn't know that a local mirror of somehost.io exists. In that case, containerd doesn't check the mirror for the image, and it pulls the image from the somehost.io public registry.

Note that by default, Google Distributed Cloud automatically mirrors images from gcr.io so you don't need to list gcr.io as a host in the hosts section.

gcrKeyPath field

If you want Google Distributed Cloud to automatically use Container Registry (gcr.io) to pull images that don't appear in your local registry, you must specify the path to the Container Registry service account key. Google Distributed Cloud doesn't have a mechanism for providing keys for other public registries.

If you don't plan on using the feature in which images are pulled from gcr.io when they don't appear in your local registry, then you don't need to add a gcrKeyPath to the cluster configuration file.

caCertPath field

If your registry requires a private transport layer security (TLS) certificate, this field takes the path to the server root CA certificate file. This certificate file should be on the admin workstation, the machine running bmctl commands. If your registry doesn't require a private TLS certificate, then you can leave the caCertPath field blank.

pullCredentialConfigPath field

If your registry server doesn't require an authentication Docker configuration file, then you can leave the pullCredentialConfigPath field blank. Note that this is the path to the configuration file on the machine running bmctl commands.

Use a registry mirror with user clusters

User clusters don't automatically pull images from a registry mirror if their admin cluster has been configured to do so. To have user clusters pull from a registry mirror, you need to configure them individually as described in Configure clusters to use a registry mirror.

Update registry mirror endpoints, certificates, and pull credentials

To update registry mirror endpoints, certificates, or pull credentials:

1. In the cluster configuration file, update the endpoint, CA certificate file, and path of the pull credential configuration file.

2. Apply the changes by running:

   bmctl update cluster -c CLUSTER_NAME --kubeconfig=ADMIN_KUBECONFIG
   

   Replace the following:

   • CLUSTER_NAME with the name of the cluster you want to update.

   • ADMIN_KUBECONFIG with the path of its admin cluster configuration file.

Verify images are pulled from registry mirror

You can determine whether containerd is pulling images from your local registry by examining the contents of a config.toml file as shown in the following steps:

1. Log into a node and examine the contents of the following file: /etc/containerd/config.toml

2. Check the plugins."io.containerd.grpc.v1.cri".registry.mirrors section of the config.toml file to see whether your registry server is listed in the endpoint field. Here's an excerpt from an example config.toml file in which the endpoint field appears in bold:

   version = 2
   root = "/var/lib/containerd"
   state = "/run/containerd"
   ...
   [plugins."io.containerd.grpc.v1.cri".registry]
     [plugins."io.containerd.grpc.v1.cri".registry.configs]
       [plugins."io.containerd.grpc.v1.cri".registry.configs."gcr.io"]
         [plugins."io.containerd.grpc.v1.cri".registry.configs."privateregistry2.io".tls]
           ca_file = '/etc/containerd/certs.d/privateregistry2.io/ca.crt'
     [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
       [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]
         endpoint = ["http://privateregistry.io", "http://privateregistry2.io"]
   ...
   

   If your registry mirror appears in the endpoint field, then the node is pulling images from your registry mirror rather than from a public registry.