Use a registry mirror to install Google Distributed Cloud

This page shows you how to install Google Distributed Cloud using a registry mirror instead of using gcr.io. To use a registry mirror, you must set the container runtime to containerd.

Registry mirrors are designated to mirror the entirety of gcr.io; not just gcr.io/anthos-baremetal-release/, which is where Google Distributed Cloud images are typically stored.

For example, if you try to pull an image gcr.io/kubernetes-e2e-test-images/nautilus:1.0, this only works if your registry service has this image in the exact same path, like 172.18.0.20:5000/kubernetes-e2e-test-images/nautilus:1.0. All non-gcr.io images still work as normal, for example, you can still pull k8s.gcr.io/pause:3.1.

Using a registry mirror helps you save on traffic and offers an alternative to using gcr.io in case you need to insulate your clusters from gcr.io outages. It also allows you to conduct your own vulnerability scanning.

Before you begin

  • You must have a container registry server set up in your network.
  • If your registry server runs a private TLS certificate, you must have the certificate authority (CA) file.
  • If your registry server needs authentication, you must have the proper login credentials or Docker configuration file.

Download all required images for Google Distributed Cloud

Download the latest version of the bmctl tool and images package from the Download page.

Upload container images to your registry server

Upload the images from the images package to your registry server by running:

[HTTPS_PROXY=http://PROXY_IP:PORT] ./bmctl push images \
    --source=./bmpackages_1.8.0.tar.xz \
    --private-registry=REGISTRY_IP:PORT \
    [--cacert=CERT_PATH] \
    [--need-credential=false]

Replace the following:

  • PROXY_IP:PORT with the IP address and port of the proxy if you need a proxy to upload the images from your workstation to the registry server.
  • REGISTRY_IP:PORT with the IP address and port of the private registry server.
  • CERT_PATH with the path of the CA cert file if your registry server uses a private TLS certificate.

Enter your username and password when prompted or select a Docker configuration file. If your registry server doesn't require credentials, then specify --need-credential=false.

For more information on the bmctl push images command, run:

bmctl push images --help

Using your own namespace

If you want to use your own namespace in your registry server instead of the root namespace, containerd can pull from this sub namespace if you provide the API endpoint for your private registry in registryMirrors.endpoint. The endpoint is usually in the format of <REGISTRY_IP:PORT>/v2/<NAMESPACE>. Check your private registry's user guide for specific details.

For example, if you only have access to 172.18.0.20:5000/test-namespace/, you can use the following command to upload all the images under namespace test-namespace:

./bmctl push images \
    --source=./bmpackages_1.8.0.tar.xz \
    --private-registry=172.18.0.20:5000/test-namespace
    --username=<USERNAME>
    --password=<PASSWORD>
    --cacert <path/to/cert.crt>

Then in the cluster YAML file, you can input the following to make containerd pull from the sub namespace:

registryMirrors:
  - endpoint: https://172.18.0.20:5000/v2/test-namespace

Create clusters from the registry mirror

Below is a sample cluster configuration file that uses your own registry mirror server instead of gcr.io.

If your registry doesn't require a private TLS certificate, then you can leave the caCertPath field blank.

If your registry server doesn't require an authentication Docker configuration file, then you can leave the pullCredentialConfigPath field blank.

For detailed information on creating clusters, see Creating clusters.

# Sample cluster config with registry mirror:
---
gcrKeyPath: /bmctl/bmctl-workspace/.sa-keys/my-gcp-project-anthos-baremetal-gcr.json
sshPrivateKeyPath: /root/ssh-key/id_rsa
registryMirrors:
  - endpoint: https://172.18.0.20:5000
    caCertPath: /root/ca.crt
    pullCredentialConfigPath: /root/.docker/config.json
---
apiVersion: v1
kind: Namespace
metadata:
  name: cluster-admin1
---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: admin1
  namespace: cluster-admin1
spec:
  nodeConfig:
    containerRuntime: containerd
...

All nodes in this cluster will use this registry mirror 172.18.0.20:5000 instead of gcr.io.

Fail over to gcr.io

If your cluster fails to pull from your registry mirror, it will automatically fail over to gcr.io. This is why we recommend providing a value for gcrKeyPath in the cluster configuration file. If a value is not provided, your cluster is not able to pull from gcr.io in the event that your registry mirror fails.

If you don't need the pull failover feature, then you don't need add a gcrKeyPath or add gcr.io to your proxy allow list.

Update registry mirror endpoints, certificates, and pull credentials

To update registry mirror endpoints, certificates, or pull credentials:

  1. In the cluster configuration file, update the endpoint, CA certificate file, pull credential configuration file path.

  2. Apply the changes by running:

    bmctl update cluster -c CLUSTER_NAME --kubeconfig=ADMIN_KUBECONFIG

    Replace the following:

    • CLUSTER_NAME with the name of the cluster you want to update.
    • ADMIN_KUBECONFIG with the path of its admin cluster's configuration file.