Troubleshooting Metadata Server access issues

This document shows you how to resolve issues with the Compute Engine metadata server.

Compute Engine VMs store metadata on a metadata server. VMs automatically have access to the metadata server API without any additional authorization. However, sometimes VMs might lose access to the metadata server due to one of the following causes:

  • Failure to resolve the metadata server domain name
  • Connection to the metadata server is blocked by one of the following:
    • OS level firewall configuration
    • Proxy setup
    • Custom routing

When VMs can't access the metadata server, some processes might fail.

For information about troubleshooting the gke-metadata-server, see Troubleshoot GKE authentication issues.

Before you begin

  • If you haven't already, set up authentication. Authentication is the process by which your identity is verified for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine as follows.

    Select the tab for how you plan to use the samples on this page:

    Console

    When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

    gcloud

    1. Install the Google Cloud CLI, then initialize it by running the following command: 

      gcloud init
    2. Set a default region and zone.

    REST

    To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

      Install the Google Cloud CLI, then initialize it by running the following command: 

      gcloud init

Common errors

The following are examples of that you might encounter if your VM fails to reach the metadata server:

curl: (6) Could not resolve host: metadata.google.internal

postAttribute error: Put "http://metadata.google.internal/computeMetadata/v1/instance/guest-attributes/guestInventory/ShortName": dial tcp: lookup metadata.google.internal on [::1]:53: read udp [::1]:58319->[::1]:53: read: connection refused

Troubleshooting failed requests to the metadata server

If your VM has lost access to the metadata server, do the following:

Linux

  1. Connect to your Linux VM.

  2. From your Linux VM, run the following commands to test connectivity to the metadata server:

    1. Query the Domain Name Server:

      nslookup metadata.google.internal

      The output should look similar to the following:

      Server:         169.254.169.254
Address:        169.254.169.254#53

Non-authoritative answer:
Name:   metadata.google.internal
Address: 169.254.169.254

    2. Check that the metadata server is reachable. To verify, run the following commands:

      ping -c 3 metadata.google.internal

      The output should look similar to the following:

      PING metadata.google.internal (169.254.169.254) 56(84) bytes of data.
64 bytes from metadata.google.internal (169.254.169.254): icmp_seq=1 ttl=255 time=0.812 ms

      ping -c 3 169.254.169.254

      The output should look similar to the following:

      PING 169.254.169.254 (169.254.169.254) 56(84) bytes of data.
64 bytes from 169.254.169.254: icmp_seq=1 ttl=255 time=1.11 ms

    3. If the output of the preceding commands matches the suggested output, your VM is connected to the metadata server and no further action is required. If the commands fail, do the following:

      1. Check that the name server is configured to the Metadata server:

        cat /etc/resolv.conf

        The output should look similar to the following:

        domain ZONE.c.PROJECT_ID.internal
search ZONE.c.PROJECT_ID.internal. c.PROJECT_ID.internal. google.internal.
nameserver 169.254.169.254

        If the output doesn't have the preceding lines, refer to your operating system documentation for information about how to edit the DHCP Policy to persist the name server configuration to 169.254.169.254. This is because changes to /etc/resolv.conf will be overwritten in one hour if zonal DNS settings are applied to the VMs within your project. In case your project is still using a global DNS, the resolv.conf file will be reverted to the default DHCP in 24 hours.

      2. Check that the mapping between the metadata server domain name and its IP address exist:

        cat /etc/hosts

        The following line should be in the output:

        169.254.169.254 metadata.google.internal  # Added by Google

        If the output doesn't have the preceding line, run the following command:

        echo "169.254.169.254 metadata.google.internal" >> /etc/hosts

Windows

  1. Connect to your Windows VM.

  2. From your Windows VM, run the following commands:

    1. Query the Domain Name Server:

      nslookup metadata.google.internal

      The output should look similar to the following:

      Server:  UnKnown
Address:  10.128.0.1

Non-authoritative answer:
Name:    metadata.google.internal
Address:  169.254.169.254

    2. Check that the metadata server is reachable. To verify, run the following commands:

      ping -n 3 metadata.google.internal

      The output should look similar to the following: 

      Pinging metadata.google.internal [169.254.169.254] with 32 bytes of data:
Reply from 169.254.169.254: bytes=32 time=1ms TTL=255


      ping -n 3 169.254.169.254

      The output should look similar to the following:

      Pinging metadata.google.internal [169.254.169.254] with 32 bytes of data:
Reply from 169.254.169.254: bytes=32 time=1ms TTL=255

    3. If the output of the preceding commands matches the suggested output, your VM is connected to the metadata server and no further action is required. If the commands fail, do the following:

      1. Check that there's a persistent route to the metadata server by running the command:

        route print

        The output should contain the following:

        Persistent Routes:
Network Address          Netmask  Gateway Address  Metric
169.254.169.254  255.255.255.255         On-link        1

        If the output doesn't have the preceding line, add the route using the following commands:

        $Adapters = Get-NetKVMAdapterRegistry
$FirstAdapter = $Adapters | Select-Object -First 1
route /p add 169.254.169.254 mask 255.255.255.255 0.0.0.0 'if' $FirstAdapter.InterfaceIndex metric 1

      2. Check that the mapping between the metadata server domain name and its IP address exists:

        type %WINDIR%\System32\Drivers\Etc\Hosts

        The following line should be in the output:

        169.254.169.254 metadata.google.internal  # Added by Google

        If the output doesn't have the preceding line, run the following command:

        echo 169.254.169.254 metadata.google.internal >> %WINDIR%\System32\Drivers\Etc\Hosts

Troubleshooting failed requests while using a network proxy

A network proxy server prevents VM's direct access to the Internet. All queries sent from within a VM are handled by the proxy server instead.

When using an application that gets credentials from the metadata server, like an authentication token, the VM requires direct access to the metadata server. If the VM is behind a proxy, you must set the NO_PROXY configuration for both the IP address and Hostname.

If you don't set the NO_PROXY configuration, you might see errors when running Google Cloud CLI commands or querying the metadata server directly since the calls to metadata.google.internal will be sent to the proxy without having them resolved locally on the instance itself.

The following is an example of an error that you might see:

ERROR 403 (Forbidden): Your client does not have permission to get URL

To resolve this proxy issue, add the metadata server hostname and IP address to the environment variable NO_PROXY by doing the following:

Linux

  1. Connect to your Linux VM.

  2. From your Linux VM, run the following commands:

    export no_proxy=169.254.169.254,metadata,metadata.google.internal

    To save the changes, run the following command:

    echo no_proxy=169.254.169.254,metadata,metadata.google.internal >> /etc/environment

Windows

  1. Connect to your Windows VM.

  2. From your Windows VM, run the following commands:

    set NO_PROXY=169.254.169.254,metadata,metadata.google.internal

    To save the changes, run the following command:

    setx NO_PROXY 169.254.169.254,metadata,metadata.google.internal /m

Incorrect header format

The metadata server performs formatting checks to ensure that headers comply with the header formatting guideline RFC 7230 Section 3.2. If the header format fails these checks the following occurs:

  • The request is accepted. However, you'll receive recommendations that you have VMs making requests to the metadata server with incorrectly formatted headers. Recommendations are sent once per VM. You can access these recommendations by using the Google Cloud CLI, or the Recommender REST API.

    After you have applied the recommendation, set the recommendation state to succeeded.

  • Starting January 20, 2024, the metadata server denies any request with an incorrectly formatted header.

The following shows examples of valid and invalid header request formats.

Not valid: contains whitespace between the header name and colon

Metadata-Flavor : Google

Valid: no whitespace between header name and colon, whitespace after the colon is ignored by the checker

Metadata-Flavor: Google

Valid: no whitespace in header

Metadata-Flavor:Google

For more information about making queries to metadata server, see Access VM metadata.