Known issues

Linux VMs Windows VMs

This page describes known issues that you might run into while using Compute Engine.

General issues

Increased latency for some persistent disks

In some cases, snapshots of large persistent disks (~3 TB or larger) trigger operations that might be disruptive to the I/O performance of the disk. If you are impacted by this issue, your persistent disks might experience increased latency during the snapshot operation. This issue can impact persistent disks of any type and can be triggered by regular or scheduled snapshots.

Compute operations permission error

The following issue was resolved on December 21, 2020.

If your project uses gcloud tool version 286.0.0 or later, you might receive one of the following errors when you run commands that update data, such as gcloud compute instances start or gcloud compute disks snapshot:

ERROR: (gcloud.compute.METHOD) Could not fetch resource:
 - Required 'compute.globalOperations.get' permission

ERROR: (gcloud.compute.METHOD) Could not fetch resource:
 - Required 'compute.regionOperations.get' permission

ERROR: (gcloud.compute.METHOD) Could not fetch resource:
 - Required 'compute.zoneOperations.get' permission

ERROR: (gcloud.compute.METHOD) HTTPError 403:
Required 'compute.globalOperations.get' permission for RESOURCE

ERROR: (gcloud.compute.METHOD) HTTPError 403:
Required 'compute.regionOperations.get' permission for RESOURCE

ERROR: (gcloud.compute.METHOD) HTTPError 403:
Required 'compute.zoneOperations.get' permission for RESOURCE

You may also encounter similar errors when you call the compute.globalOperations.wait, compute.regionOperations.wait, or compute.zoneOperations.wait API methods.

If you have IAM permissions set at the instance level and not the project level, permission errors might be due to Compute Engine Issue 171947836. To mitigate this issue, try one of the following workarounds:

Create a custom IAM role at the project level that includes the missing permission: compute.regionOperations.get, compute.regionOperations.get, or compute.zoneOperations.get. Assign this role to users who received the permission error.
Downgrade the gcloud tool to version 285 or earlier.
Assign the roles/compute.viewer role to users who received the permission error.

Note: roles/compute.viewer has broad access that might not be suitable for all users in your organization.

Resolved: Some projects have the wrong default service account

The following issue was resolved on December 1, 2020.

When you enable the Compute Engine API in your project, a default service account is added to your project. Normally, the default service account has the following email:

PROJECT_NUMBER-compute@developer.gserviceaccount.com

For a small number of projects, the email for the default service account is incorrectly set to:

service-PROJECT_NUMBER@compute-system.iam.gserviceaccount.com

If you are impacted by this issue, you might encounter 403 Forbidden errors and you might not be able to create new Compute Engine resources. If this is the case, please contact Google Cloud Support to resolve the issue and answer any questions you have. There is an automated remediation rolling out to correct the issue while a long-term solution is developed.

To follow the status of this issue, see Compute Engine Issue 171305221.

Known issues for Linux VM instances

GPG error: EXPKEYSIG 3746C208A7317B0F when updating packages

On Debian and Ubuntu based systems, including your local workstations, you might encounter an error similar to the following example:

W: An error occurred during the signature verification.
The repository is not updated and the previous index files will be used.
GPG error: http://packages.cloud.google.com/apt cloud-sdk-stretch InRelease:
The following signatures were invalid: EXPKEYSIG 3746C208A7317B0F
Google Cloud Packages Automatic Signing Key <gc-team@google.com>

This error prevents you from obtaining the latest updates for several Google Cloud tools, including the following items:

Compute Engine guest environment
Cloud SDK with the gcloud command-line tool
Cloud Logging agent

To resolve this error, get the latest valid apt-key.gpg key file from https://packages.cloud.google.com by running the following command:

curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

Alternatively, on Compute Engine VM instances running Debian or Ubuntu images, you can get the latest keys if you recreate your instances using the following image versions:

Image project debian-cloud:
- debian-9-stretch-v20180401 or image family debian-9
- debian-8-jessie-v20180401 or image family debian-8
Image project ubuntu-os-cloud:
- ubuntu-1710-artful-v20180315 or image family ubuntu-1710
- ubuntu-1604-xenial-v20180323 or image family ubuntu-1604-lts
- ubuntu-1404-trusty-v20180308 or image family ubuntu-1404-lts

Red Hat Enterprise Linux 7 and CentOS 7 read-only root filesystem issue

VM instances running public images rhel-7-v20170719 or earlier or centos-7-v20170719 or earlier can boot with the root filesystem mounted in read-only mode. Apps, daemons, or scripts that require write access to the root filesystem will fail.

Do not reboot or restart running instances that use the affected public images or they will become stuck in read-only mode. If your instances are already stuck in read-only mode, you can remotely restore read/write mode to the root filesystem and then fix the issue.

Identify affected instances:

Identify potentially affected instances with the following gcloud compute disks list commands:

RHEL 7:

gcloud compute disks list --filter="sourceImage ~ rhel-7-v201[4-6].* OR sourceImage ~ rhel-7-v20170[1-7].*" --uri

CentOS 7:

gcloud compute disks list --filter="sourceImage ~ centos-7-v201[4-6].* OR sourceImage ~ centos-7-v20170[1-7].*" --uri

If these disks are attached to your instances, you can correct the issue on those instances. The process to correct affected instances depends on the image version that you used to create the instance.

Instances created using RHEL 7 images between rhel-7-v20160418 and rhel-7-v20170719, or CentOS 7 images between centos-7-v20160418 and centos-7-v20170719:

If the instance uses automatic updates, yum-cron automatically installs the fixed package and removes the broken mount option from the /etc/fstab file. If the instance does not have automatic updates enabled, you can fix the instance with the following process:

Connect to the instance using SSH. If the connection fails, the instance might be stuck in read-only mode. However, you can attempt to recover it. If you have connected to the affected instance over SSH before, your public SSH keys are already on the root file system and will continue to function. Run a remote command over ssh to remount the filesystem in rw mode. For example, you can use the following gcloud command to remount the root file system:
```
gcloud compute ssh [INSTANCE_NAME] --command "sudo mount -o remount,rw /dev/sda1 /"
```
After the filesystem is mounted in read/write mode again, connect to the instance over SSH.
Run sudo yum -y update to update all installed packages including the gce-disk-expand package, which contains the fix.

The instance can now reboot without mounting the root filesystem in read-only mode.

Instances created using RHEL 7 images created before rhel-7-v20160418 or CentOS 7 images created before centos-7-v20160418:

Connect to the instance using SSH. If the connection fails, the instance might be stuck in read-only mode. However, you can attempt to recover it. If you have connected to the affected instance over SSH before, your public SSH keys are already on the root file system and will continue to function. Run a remote command over ssh to remount the filesystem in rw mode. For example, you can use the following gcloud command to remount the root file system:
```
gcloud compute ssh [INSTANCE_NAME] --command "sudo mount -o remount,rw /dev/sda1 /"
```
After the filesystem is mounted in read/write mode again, connect to the instance over SSH.
Edit the /etc/fstab file and remove any barrier=1 mount options in that file. The default mount option must be the only mount option set for the root file system entry. You can correct this broken mount option with the following command:
```
sudo sed -i 's/defaults,barrier[^ ,]*/defaults/' /etc/fstab
```
After you remove the barrier=1 mount option, the root file system entry in the /etc/fstab file should look similar to the following example but with a different value for the UUID:
```
UUID=b5e54172-67e3-4d52-95f4-4314e71b25fd / xfs defaults 0 0
```

The instance can now reboot without mounting the root filesystem in read-only mode.

CentOS image `v20131120` introduced a breaking change where iptables are turned on by default

The v20131120 release of CentOS 6 image, centos-6-v20131120, has a breaking change where iptables are turned on by default. This prevents external traffic from reaching CentOS instances that are running centos-6-v20131120, even if there is a relevant firewall rule resource that permits the connection.

As a workaround, users can disable iptables or update iptables to permit the desired connection (in addition to permitting the traffic using firewall rules). To disable iptables, run the following commands:

# Save your iptable settings
user@centos-instance:~$ sudo service iptables save

# Stop the iptables service
user@centos-instance:~$ sudo service iptables stop

# Disable iptables on start up
user@centos-instance:~$ sudo chkconfig iptables off

Google-provided images have a known bug with the ext4/scsi driver in the stable Debian and CentOS kernels

For centos-6-v20131120 and debian-7-wheezy-v20131120 images, a known ext4 bug might cause a memory leak and eventual crash of a virtual machine instance under heavy persistent disk load. For details, see the Question about ext4 excessive stall time thread on the Linux kernel mailing list.

Instance names longer than 32 characters can cause problems with various UNIX tools

Date reported: June 2012

Although instance names can be up to 63 characters, names that are longer than 32 characters can cause some tools to be unreliable, including tools that may run during boot. As a workaround, choose instance names that are shorter than 32 characters.

Instances using OS Login return a login message after connection

On some instances that use OS Login, you might receive the following error message after the connection is established:

/usr/bin/id: cannot find name for group ID 123456789

Ignore the error message.

Known issues for Windows VM instances

Although Windows instances can use the NVMe interface with local SSDs, support for NVMe on Windows is in Beta, and we do not guarantee the same performance as Linux instances.
On Windows Server 2008 R2, installing Python 2.7.9 or later requires Visual C++. Python 2.7.8 avoids this dependency, but we recommend installing the latest version.
Compute Engine does not yet support IPv6. Even if you enable IPv6 by selecting the option in a Windows instance, the setting is ignored.
After you create an instance, you cannot connect to it instantly. All new Windows instances use the System preparation (sysprep) tool to set up your instance, which can take 5–10 mins to complete.
Windows Server images cannot activate without a network connection to kms.windows.googlecloud.com and stop functioning if they do not initially authenticate within 30 days. Software activated by the KMS must reactivate every 180 days, but the KMS attempts to reactivate every 7 days. Make sure to configure your Windows instances so that they remain activated.
Kernel software that accesses non-emulated model specific registers will generate general protection faults, which can cause a system crash depending on the guest operating system.