This page describes known issues that you might run into while using Compute Engine.
General issues
Increased latency for some persistent disks
In some cases, snapshots of large persistent disks (~3 TB or larger) trigger operations that might be disruptive to the I/O performance of the disk. If you are impacted by this issue, your persistent disks might experience increased latency during the snapshot operation. This issue can impact persistent disks of any type and can be triggered by regular or scheduled snapshots.
Compute operations permission error
The following issue was resolved on December 21, 2020.
If your project uses gcloud
tool version 286.0.0 or later, you might
receive one of the following errors when you run commands that update data, such
as gcloud compute instances start
or gcloud compute disks snapshot
:
ERROR: (gcloud.compute.METHOD) Could not fetch resource: - Required 'compute.globalOperations.get' permission
ERROR: (gcloud.compute.METHOD) Could not fetch resource: - Required 'compute.regionOperations.get' permission
ERROR: (gcloud.compute.METHOD) Could not fetch resource: - Required 'compute.zoneOperations.get' permission
ERROR: (gcloud.compute.METHOD) HTTPError 403: Required 'compute.globalOperations.get' permission for RESOURCE
ERROR: (gcloud.compute.METHOD) HTTPError 403: Required 'compute.regionOperations.get' permission for RESOURCE
ERROR: (gcloud.compute.METHOD) HTTPError 403: Required 'compute.zoneOperations.get' permission for RESOURCE
You may also encounter similar errors when you call the
compute.globalOperations.wait
, compute.regionOperations.wait
,
or compute.zoneOperations.wait
API methods.
If you have IAM permissions set at the instance level and not the project level, permission errors might be due to Compute Engine Issue 171947836. To mitigate this issue, try one of the following workarounds:
- Create a
custom IAM role at
the project level that includes the missing permission:
compute.regionOperations.get
,compute.regionOperations.get
, orcompute.zoneOperations.get
. Assign this role to users who received the permission error. - Downgrade the
gcloud
tool to version 285 or earlier. Assign the
roles/compute.viewer
role to users who received the permission error.
Resolved: Some projects have the wrong default service account
The following issue was resolved on December 1, 2020.
When you enable the Compute Engine API in your project, a default service account is added to your project. Normally, the default service account has the following email:
PROJECT_NUMBER-compute@developer.gserviceaccount.com
For a small number of projects, the email for the default service account is incorrectly set to:
service-PROJECT_NUMBER@compute-system.iam.gserviceaccount.com
If you are impacted by this issue, you might encounter 403 Forbidden
errors
and you might not be able to create new Compute Engine resources.
If this is the case, please contact Google Cloud Support to
resolve the issue and answer any questions you have. There is an automated
remediation rolling out to correct the issue while a long-term solution is
developed.
To follow the status of this issue, see Compute Engine Issue 171305221.
Known issues for Linux VM instances
GPG error: EXPKEYSIG 3746C208A7317B0F when updating packages
On Debian and Ubuntu based systems, including your local workstations, you might encounter an error similar to the following example:
W: An error occurred during the signature verification.
The repository is not updated and the previous index files will be used.
GPG error: http://packages.cloud.google.com/apt cloud-sdk-stretch InRelease:
The following signatures were invalid: EXPKEYSIG 3746C208A7317B0F
Google Cloud Packages Automatic Signing Key <gc-team@google.com>
This error prevents you from obtaining the latest updates for several Google Cloud tools, including the following items:
- Compute Engine guest environment
- Cloud SDK with the
gcloud
command-line tool - Cloud Logging agent
To resolve this error, get the latest valid apt-key.gpg
key
file from https://packages.cloud.google.com
by running the following command:
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
Alternatively, on Compute Engine VM instances running Debian or Ubuntu images, you can get the latest keys if you recreate your instances using the following image versions:
- Image project
debian-cloud
:debian-9-stretch-v20180401
or image familydebian-9
debian-8-jessie-v20180401
or image familydebian-8
- Image project
ubuntu-os-cloud
:ubuntu-1710-artful-v20180315
or image familyubuntu-1710
ubuntu-1604-xenial-v20180323
or image familyubuntu-1604-lts
ubuntu-1404-trusty-v20180308
or image familyubuntu-1404-lts
Red Hat Enterprise Linux 7 and CentOS 7 read-only root filesystem issue
VM instances running public images rhel-7-v20170719
or earlier or
centos-7-v20170719
or earlier can boot with the root filesystem mounted
in read-only mode. Apps, daemons, or scripts that require write
access to the root filesystem will fail.
Do not reboot or restart running instances that use the affected public images or they will become stuck in read-only mode. If your instances are already stuck in read-only mode, you can remotely restore read/write mode to the root filesystem and then fix the issue.
Identify affected instances:
Identify potentially affected instances with the following
gcloud compute disks list
commands:
RHEL 7:
gcloud compute disks list --filter="sourceImage ~ rhel-7-v201[4-6].* OR sourceImage ~ rhel-7-v20170[1-7].*" --uri
CentOS 7:
gcloud compute disks list --filter="sourceImage ~ centos-7-v201[4-6].* OR sourceImage ~ centos-7-v20170[1-7].*" --uri
If these disks are attached to your instances, you can correct the issue on those instances. The process to correct affected instances depends on the image version that you used to create the instance.
Instances created using RHEL 7 images between rhel-7-v20160418
and
rhel-7-v20170719
, or CentOS 7 images between centos-7-v20160418
and
centos-7-v20170719
:
If the instance uses automatic updates, yum-cron automatically installs
the fixed package and removes the broken mount option from the /etc/fstab
file. If the instance does not have automatic updates enabled, you can fix the
instance with the following process:
Connect to the instance using SSH. If the connection fails, the instance might be stuck in read-only mode. However, you can attempt to recover it. If you have connected to the affected instance over SSH before, your public SSH keys are already on the root file system and will continue to function. Run a remote command over
ssh
to remount the filesystem inrw
mode. For example, you can use the followinggcloud
command to remount the root file system:gcloud compute ssh [INSTANCE_NAME] --command "sudo mount -o remount,rw /dev/sda1 /"
After the filesystem is mounted in read/write mode again, connect to the instance over SSH.
Run
sudo yum -y update
to update all installed packages including thegce-disk-expand
package, which contains the fix.
The instance can now reboot without mounting the root filesystem in read-only mode.
Instances created using RHEL 7 images created before rhel-7-v20160418
or
CentOS 7 images created before centos-7-v20160418
:
Connect to the instance using SSH. If the connection fails, the instance might be stuck in read-only mode. However, you can attempt to recover it. If you have connected to the affected instance over SSH before, your public SSH keys are already on the root file system and will continue to function. Run a remote command over
ssh
to remount the filesystem inrw
mode. For example, you can use the followinggcloud
command to remount the root file system:gcloud compute ssh [INSTANCE_NAME] --command "sudo mount -o remount,rw /dev/sda1 /"
After the filesystem is mounted in read/write mode again, connect to the instance over SSH.
Edit the
/etc/fstab
file and remove anybarrier=1
mount options in that file. Thedefault
mount option must be the only mount option set for the root file system entry. You can correct this broken mount option with the following command:sudo sed -i 's/defaults,barrier[^ ,]*/defaults/' /etc/fstab
After you remove the
barrier=1
mount option, the root file system entry in the/etc/fstab
file should look similar to the following example but with a different value for the UUID:UUID=b5e54172-67e3-4d52-95f4-4314e71b25fd / xfs defaults 0 0
The instance can now reboot without mounting the root filesystem in read-only mode.
CentOS image v20131120
introduced a breaking change where iptables are turned on by default
The v20131120
release of CentOS 6 image, centos-6-v20131120
,
has a breaking change where
iptables
are turned on by default. This prevents external traffic from reaching CentOS
instances that are running centos-6-v20131120
, even if there is
a relevant firewall rule resource that permits the
connection.
As a workaround, users can disable iptables or update iptables to permit the desired connection (in addition to permitting the traffic using firewall rules). To disable iptables, run the following commands:
# Save your iptable settings
user@centos-instance:~$ sudo service iptables save
# Stop the iptables service
user@centos-instance:~$ sudo service iptables stop
# Disable iptables on start up
user@centos-instance:~$ sudo chkconfig iptables off
Google-provided images have a known bug with the ext4/scsi driver in the stable Debian and CentOS kernels
For centos-6-v20131120 and debian-7-wheezy-v20131120 images, a known ext4 bug might cause a memory leak and eventual crash of a virtual machine instance under heavy persistent disk load. For details, see the Question about ext4 excessive stall time thread on the Linux kernel mailing list.
Instance names longer than 32 characters can cause problems with various UNIX tools
Date reported: June 2012
Although instance names can be up to 63 characters, names that are longer than 32 characters can cause some tools to be unreliable, including tools that may run during boot. As a workaround, choose instance names that are shorter than 32 characters.
Instances using OS Login return a login message after connection
On some instances that use OS Login, you might receive the following error message after the connection is established:
/usr/bin/id: cannot find name for group ID 123456789
Ignore the error message.
Known issues for Windows VM instances
- Although Windows instances can use the NVMe interface with local SSDs, support for NVMe on Windows is in Beta, and we do not guarantee the same performance as Linux instances.
- On Windows Server 2008 R2, installing Python 2.7.9 or later requires Visual C++. Python 2.7.8 avoids this dependency, but we recommend installing the latest version.
- Compute Engine does not yet support IPv6. Even if you enable IPv6 by selecting the option in a Windows instance, the setting is ignored.
- After you create an instance, you cannot connect to it instantly. All new Windows instances use the System preparation (sysprep) tool to set up your instance, which can take 5–10 mins to complete.
- Windows Server images cannot activate without a network connection to
kms.windows.googlecloud.com
and stop functioning if they do not initially authenticate within 30 days. Software activated by the KMS must reactivate every 180 days, but the KMS attempts to reactivate every 7 days. Make sure to configure your Windows instances so that they remain activated. - Kernel software that accesses non-emulated model specific registers will generate general protection faults, which can cause a system crash depending on the guest operating system.