Troubleshooting

This section explains how to troubleshoot common Container Registry and Docker issues.

Domain-scoped projects

If you receive the error invalid reference format, one problem may be that you have a domain-scoped project.

If your project is scoped to your domain, then the project ID includes the domain and a colon, for example: example.com:my-project. See Domain-scoped projects for how to work with projects IDs that include a domain.

Error: Status 405: v1 Registry API is disabled.

If you continuously encounter an error such as "v1 Registry API is disabled" when pulling or pushing images, ensure that your hostname, project ID, image name, and tag or digest are spelled correctly.

To see your image's tags and digest, run the following command:

gcloud container images list-tags [HOSTNAME]/[PROJECT-ID]/[IMAGE]

For example:

gcloud container images list-tags gcr.io/my-project/my-image

For more information about listing image tags and digests, see Managing Images.

Permission issues

The following sections describe solutions for specific permissions issues. Always make sure that you have the required permissions to push or pull. See Permissions and roles.

If you have enabled uniform bucket-level access on a storage bucket that is used by Container Registry, then you might have access issues pushing and pulling to Container Registry.

To fix access issues, ensure that you have the required permissions to push or pull. These permissions are listed in Permissions and roles.

If you encounter a permission denied error, such as the following example:

FATA[0000] Post http://var/run/docker.sock/v1.17/images/gcr.io/container-engine-docs/example/push?tag=: dial unix /var/run/docker.sock: permission denied
ERROR: (gcloud.docker) A Docker command did not run successfully.
Tried to run: 'docker push gcr.io/container-engine-docs/example'
Exit code: 1

You may need to add yourself to the docker user group.

Run the following command in your shell or terminal window:

  sudo usermod -a -G docker ${USER}

Restart your system after adding yourself to the docker user group.

Permission issues when communicating with Container Registry

If you encounter a permission error like the following:

Permission denied: Unable to create the repository, please check that you have access to do so

  1. Verify that billing is enabled for your project.

  2. Verify your access:

    1. Ensure that you are authenticated for gcloud by running the following command:

      gcloud init

    2. Ensure that Docker is configured to use gcloud as a Container Registry credential helper by running the following command:

      gcloud auth configure-docker

    3. Verify that docker-credential-gcloud can be executed:

      docker-credential-gcloud list

      You should see a JSON object with your target registry as one of its keys. For example:

      {
  "https://asia.gcr.io": "oauth2accesstoken",
  "https://eu.gcr.io": "oauth2accesstoken",
  "https://gcr.io": "oauth2accesstoken",
  "https://us.gcr.io": "oauth2accesstoken"
}

    4. Next, check that you have permission to write to Cloud Storage in the project you are pushing to. If not, ask an admin to grant access to your user and try again.

    5. If the problem persists after you have the correct permission, it may be that your access token is obtained without one of these scopes:

      • https://www.googleapis.com/auth/devstorage.read_write
      • https://www.googleapis.com/auth/devstorage.full_control

      To verify this, first obtain the access token yourself. This varies from application to application, for example if you are using an access token from a Compute Engine default service account, you can follow instruction here to get it.

      Once you have obtained the access token yourself, you can use this command to look at the scopes used in obtaining the access token:

      curl -H "Authorization: Bearer <your access token>" https://www.googleapis.com/oauth2/v1/tokeninfo

      In case the mentioned scopes aren't included, fix the problem to make sure to include them when obtaining the access token in your code. For an example, for tokens generated specifically for the default service account on your Compute Engine virtual machine, you will need to fix the scope setting for that virtual machine and recreate it.

Permission issues pushing and pulling images

The VM instance that pushes or pulls images must be correctly configured with the required IAM permissions and access scopes to access the Container Registry storage bucket. For information about the required settings, see Using Container Registry with Google Cloud.

ImagePullBackoff error from Google Kubernetes Engine

GKE returns an ImagePullBackoff error when it is unable to pull an image from a registry. The error might occur because the image cannot be found or because your nodes do not have permissions to pull from the registry. By default GKE nodes have permissions to pull images from Container Registry when the registry is in the same Google Cloud project as your nodes.

The GKE documentation includes steps to identify the root cause and resolve the issue.

Organization policy enforcement

Organization policy constraints can affect usage of Container Registry when they apply to services that Container Registry uses, including constraints that require use of customer-managed encryption keys (CMEK).

Bad Request error when pushing an image

When the Cloud Storage API is in the Deny policy list for the constraint constraints/gcp.restrictNonCmekServices, you cannot push images to Container Registry if the underlying storage buckets aren't encrypted with CMEK. The following message is returned:

unknown: Bad Request

When constraints/gcp.restrictCmekCryptoKeyProjects is configured, storage buckets must be encrypted with a CryptoKey from an allowed project, folder, or organization. Existing buckets that are not compliant must be configured to use the required key by default.

To encrypt your storage buckets, refer to the Cloud Storage instructions. The bucket name for a registry host has one of the following formats:

  • artifacts.PROJECT-ID.appspot.com for images stored on the host gcr.io
  • STORAGE-REGION.artifacts.PROJECT-ID.appspot.com for images stored on asia.gcr.io, eu.gcr.io, or us.gcr.io.

The gcr Pub/Sub topic was not automatically created

When you activate the Container Registry API in a Google Cloud project, Container Registry tries to automatically create a Pub/Sub topic with the topic ID gcr using Google-managed encryption keys.

When the Pub/Sub API is in the Deny policy list for the constraint constraints/gcp.restrictNonCmekServices, topics must be encrypted with CMEK. Requests to create a topic without CMEK encryption will fail.

To create the gcr topic with CMEK encryption, see the Pub/Sub instructions for encrypting topics.

Quota Limits

If you exceed Container Registry's quota limit, you may see an error messages like this:

Error: Status 429 trying to pull repository [...] "Quota Exceeded."

To avoid exceeding the fixed quota limit, you can:

  • Increase the number of IP addresses talking to Container Registry. Quotas are per IP address.
  • Add retries that introduce a delay. For example, you could use exponential backoff.

Invalid registry endpoint with boot2docker

If you're having trouble reaching the Container Registry from a boot2docker environment:

docker push gcr.io/example/sample

Error response from daemon: invalid registry endpoint https://gcr.io/v0/:
  unable to ping registry endpoint https://gcr.io/v0/
v2 ping attempt failed with error: Get https://gcr.io/v2/:
  x509: certificate has expired or is not yet valid
v1 ping attempt failed with error: Get https://gcr.io/v1/_ping:
  x509: certificate has expired or is not yet valid.
If this private registry supports only HTTP or HTTPS with an unknown CA
certificate, please add `--insecure-registry gcr.io` to the daemon's
arguments. In the case of HTTPS, if you have access to the registry's CA
certificate, no need for the flag; simply place the CA certificate at
/etc/docker/certs.d/gcr.io/ca.crt

You may need to restart boot2docker:

boot2docker stop
boot2docker start

Error about pushing a root-level image

When you try to push a container image, the push fails with a message that includes:

Pushing to root-level images is disabled

This message indicates that you tagged the image with the hostname and image, but did not include the project ID.

Tag the image using the correct image path format:

HOSTNAME/PROJECT-ID/IMAGE:TAG

For example: gcr.io/web-project/web-app:1.0.

Docker on Mac

If you encounter any issues with Docker on Mac, you might need to try a workaround. Errors can include unresponsive Docker push/pull operations, or a network error similar to the following:

Post https://us.gcr.io/v2/[repo name]/blobs/uploads/: dial tcp xx.xxx.xx.xx:xxx: i/o timeout

If you encounter these errors, try these steps:

  • Run the docker-machine restart default command in the Mac terminal to restart the Docker daemon.

  • Ensure that "Securely store docker logins in macOS keychain" is not enabled in Docker's Preferences menu.

  • Ensure that you're running the most recent Docker version.