Troubleshooting

This section explains how to troubleshoot common Container Registry and Docker issues.

Domain-scoped projects

If you receive the error invalid reference format, one problem may be that you have a domain-scoped project.

If your project is scoped to your domain, then the project ID includes the domain and a colon, for example: example.com:my-project. See Domain-scoped projects for how to work with projects IDs that include a domain.

Error: Status 405: "v1 Registry API is disabled."

If you continuously encounter an error such as "v1 Registry API is disabled" when pulling or pushing images, ensure that your hostname, project ID, image name, and tag or digest are spelled correctly.

To see your image's tags and digest, run the following command:

gcloud container images list-tags [HOSTNAME]/[PROJECT-ID]/[IMAGE]

For example:

gcloud container images list-tags gcr.io/my-project/my-image

For more information about listing image tags and digests, see Managing Images.

Permission issues

The following sections describe solutions for specific permissions issues. Always make sure that you have the required permissions to push or pull. See Permissions and roles.

Permission issues related to uniform bucket-level access

If you have enabled uniform bucket-level access on a storage bucket that is used by Container Registry, then you might have access issues pushing and pulling to Container Registry.

To fix access issues, ensure that you have the required permissions to push or pull. These permissions are listed in Permissions and roles.

Permission issues related to Docker setup on your local machine

If you encounter a permission denied error, such as the following example:

FATA[0000] Post http://var/run/docker.sock/v1.17/images/gcr.io/container-engine-docs/example/push?tag=: dial unix /var/run/docker.sock: permission denied
ERROR: (gcloud.docker) A Docker command did not run successfully.
Tried to run: 'docker push gcr.io/container-engine-docs/example'
Exit code: 1

You may need to add yourself to the docker user group.

Run the following command in your shell or terminal window:

  sudo usermod -a -G docker ${USER}

Restart your system after adding yourself to the docker user group.

Permission issues when communicating with Container Registry

If you encounter a permission error like the following:

Permission denied: Unable to create the repository, please check that you have access to do so

Verify that billing is enabled for your project.
Verify your access:
1. Ensure that you are authenticated for gcloud by running the following command:
```
gcloud init
```
2. Ensure that Docker is configured to use gcloud as a Container Registry credential helper by running the following command:
```
gcloud auth configure-docker
```
3. Verify that docker-credential-gcloud can be executed:
```
docker-credential-gcloud list
```
  You should see a JSON object with your target registry as one of its keys. For example:
```
{
  "https://asia.gcr.io": "oauth2accesstoken",
  "https://eu.gcr.io": "oauth2accesstoken",
  "https://gcr.io": "oauth2accesstoken",
  "https://staging-k8s.gcr.io": "oauth2accesstoken",
  "https://us.gcr.io": "oauth2accesstoken"
}
```
4. Next, check that you have permission to write to Cloud Storage in the project you are pushing to. If not, ask an admin to grant access to your user and try again.
5. If the problem persists after you have the correct permission, it may be that your access token is obtained without one of these scopes:
  - https://www.googleapis.com/auth/devstorage.read_write
  - https://www.googleapis.com/auth/devstorage.full_control
  To verify this, first obtain the access token yourself. This varies from application to application, for example if you are using an access token from a Compute Engine default service account, you can follow instruction here to get it.
  
  Once you have obtained the access token yourself, you can use this command to look at the scopes used in obtaining the access token:
```
curl -H "Authorization: Bearer <your access token>" https://www.googleapis.com/oauth2/v1/tokeninfo
```
  In case the mentioned scopes aren't included, fix the problem to make sure to include them when obtaining the access token in your code. For an example, for tokens generated specifically for the default service account on your Compute Engine virtual machine, you will need to fix the scope setting for that virtual machine and recreate it.

Permission issues pushing and pulling images

The VM instance that pushes or pulls images must be correctly configured with the required IAM permissions and access scopes to access the Container Registry storage bucket. For information about the required settings, refer to the requirements for using Container Registry with Google Cloud.

ImagePullBackoff error from Google Kubernetes Engine

GKE returns an ImagePullBackoff error when it is unable to pull an image from a registry. The error might occur because the image cannot be found or because your nodes do not have permissions to pull from the registry. By default GKE nodes have permissions to pull images from Container Registry when the registry is in the same Google Cloud project as your nodes.

The GKE documentation includes steps to identify the root cause and resolve the issue.

Quota Limits

If you exceed Container Registry's quota limit, you may see an error messages like this:

Error: Status 429 trying to pull repository [...] "Quota Exceeded."

To avoid exceeding the fixed quota limit, you can:

Increase the number of IP addresses talking to Container Registry. Quotas are per IP address.
Add retries that introduce a delay. For example, you could use exponential backoff.

Invalid registry endpoint with boot2docker

If you're having trouble reaching the Container Registry from a boot2docker environment:

docker push gcr.io/example/sample

Error response from daemon: invalid registry endpoint https://gcr.io/v0/:
  unable to ping registry endpoint https://gcr.io/v0/
v2 ping attempt failed with error: Get https://gcr.io/v2/:
  x509: certificate has expired or is not yet valid
v1 ping attempt failed with error: Get https://gcr.io/v1/_ping:
  x509: certificate has expired or is not yet valid.
If this private registry supports only HTTP or HTTPS with an unknown CA
certificate, please add `--insecure-registry gcr.io` to the daemon's
arguments. In the case of HTTPS, if you have access to the registry's CA
certificate, no need for the flag; simply place the CA certificate at
/etc/docker/certs.d/gcr.io/ca.crt

You may need to restart boot2docker:

boot2docker stop
boot2docker start

Docker on Mac

If you encounter any issues with Docker on Mac, you might need to try a workaround. Errors can include unresponsive Docker push/pull operations, or a network error similar to the following:

Post https://us.gcr.io/v2/[repo name]/blobs/uploads/: dial tcp xx.xxx.xx.xx:xxx: i/o timeout

If you encounter these errors, try these steps:

Run the docker-machine restart default command in the Mac terminal to restart the Docker daemon.
Ensure that "Securely store docker logins in macOS keychain" is not enabled in Docker's Preferences menu.
Ensure that you're running the most recent Docker version.