Certificate signed by unknown authority

Problem

When trying to pull Docker Containerd images from a private registry on Google Kubernetes Engine 1.19 and above, you may see the below error:

x509: certificate signed by unknown authority.

Environment

  • Images being pulled from a private registry.
  • Private registry does not use an SSL certificate signed by a well-known CA.
  • The IP of the registry is in the subnet 10.0.0.0/8.

Solution

Add the Private Registry's CA PEM file to the Containerd configuration to allow communication with the registry. You can do this by either:

  • Manually (or by bootstrapping with a DaemonSet) updating Containerd configuration with a CA certificate store in PEM format. More details are available in Configure Registry TLS Communication .
  • Alternatively, instead of uploading the CA PEM file you can skip the insure verification by appending the following lines to /etc/containerd/config.toml.
[plugins.cri.registry.configs."<Registry URL>".tls]

     insecure_skip_verify = true

Since nodes are ephemeral (nodes are added and deleted regularly), the DaemonSet approach would be ideal as a pod is created on every node of the cluster. 

Cause

Starting with Google Kubernetes Engine node version 1.19, Containerd became the default node image. Due to this during Google Kubernetes Engine upgrade ability to pull images maybe effected due to changes in configuration between docker and Containerd.

All Google Kubernetes Engine nodes add the flag --insecure-registry 10.0.0.0/8 while starting Docker daemon. This is due to upstream PR2620. This would mean if you are using a private registry with an insecure SSL certs in the subnet 10.0.0.0/8, docker is allowed to pull images.

But in Containerd, all image pulls verify TLS and an explicit exemption must be granted for your private registry. 

If PDB is not configured, this can lead to application outages, as pods would not start as image pull fails.