Migrating to v1 metadata server endpoint

Migration overview

To migrate to v1.0, complete the following steps:

  1. Identify the VM instances that are using the deprecated metadata server endpoints.

    If you have Google Kubernetes Engine clusters, you need to identify the GKE nodes that are using the deprecated endpoint.

  2. On the identified VM instances or nodes, find the processes that are using the deprecated metadata server endpoints.

    • If the process belongs to an application that you did not develop, update the application to use a v1 metadata server endpoint. For a list of known applications that require an update, see applications that require an update.
    • If the process belongs to an application that you developed and your code is calling any of the legacy endpoints, complete the following steps:
      1. Review the differences between v1 and the deprecated metadata server endpoints.
      2. Update queries to use the v1 metadata server endpoint.
    • If the process belongs to an application that you developed but your code is not making requests to any of the legacy endpoints, then the requests might be made by SDKs or other dependencies. To fix this, update all SDKs and dependencies used by the application.To view the versions of the Google libraries that require an update, see supported library versions.
  3. (Optional) Disable the v0.1 and v1beta1 metadata server endpoints.

    Identifying the VM instances

    To identify which VM instances are using v0.1 and v1beta1 endpoints, you can make requests to two new endpoints.

    The output from querying these new endpoints tells you how many times a given VM instance accessed the deprecated endpoints. Note: Each time a VM instance is stopped, the counter resets.

    To see if a VM instance has accessed the v0.1 and v1beta1 endpoints, run the following commands:

    curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/legacy-endpoint-access/0.1
    curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/legacy-endpoint-access/v1beta1
    

    The following python sample shows how to programmatically watch these new endpoints:

    def wait_for_legacy_usage(callback):
        url = '{}/instance/legacy-endpoint-access'.format(METADATA_URL)
        last_etag = '0'
        counts = None
        while True:
            r = requests.get(
                url,
                params={
                    'last_etag': last_etag,
                    'recursive': True,
                    'wait_for_change': True
                },
                headers=METADATA_HEADERS)
            if r.status_code == 503:  # Metadata server unavailable
                print('Metadata server unavailable. Sleeping for 1 second.')
                time.sleep(1)
                continue
            if r.status_code == 404:  # Feature not yet supported
                print('Legacy endpoint access not supported. Sleeping for 1 hour.')
                time.sleep(3600)
                continue
            r.raise_for_status()
    
            last_etag = r.headers['etag']
            access_info = json.loads(r.text)
            if not counts:
                counts = access_info
            if access_info != counts:
                diff = {
                    ver: access_info[ver] - counts[ver] for ver in counts
                }
                counts = access_info
                callback(diff)

    Identifying the GKE nodes

    To identify the nodes in your Google Kubernetes Engine clusters that are using the v0.1 and v1beta1 endpoints, see Identifying workloads using the legacy metadata server endpoints.

    Identifying the processes

    After you have identified the VM instances that are making requests to the deprecated metadata server endpoints, you can then find the processes on these VM instances that are making these requests.

    To identify the processes, use logging or monitoring tools such as ngrep or auditd.

    ngrep

    You can use ngrep (network grep) to collect packets and filter the contents of these packets. To collect packet information for processes that make requests to the deprecated endpoints, on your VM instance, run the following command:

    sudo ngrep -l -q "v1beta1|0\.1/meta-data" tcp \
    and dst host 169.254.169.254 or metadata.google.internal \
    and dst port 80
    

    If any requests are made to the deprecated endpoints, the output resembles the following:

    T 10.128.0.4:41312 -> 169.254.169.254:80 [AP]
    GET /computeMetadata/v1beta1/instance/id HTTP/1.1..Host: metadata.goog..User-Agent: curl/7.52.1..Accept: */*....
    

    The output contains the following information:

    • The root url: /computeMetadata/v1beta1/instance/id.
    • The user agent that is used to make the request. In this example, it is curl/7.52.1.
    • The IP address (source IP) and port for the VM instance that the request originated from. In this example, the IP address is 10.128.0.4 and the port is 41312 .
    • If the process is still alive, you can review the port information to find the process ID.

    Reviewing port information

    The following procedure might identify processes with outstanding hanging HTTP GET requests. This procedure finds the processes that are talking to both the current and deprecated metadata endpoints.

    To list the sockets with open connections to the metadata server, on your VM instance, run the following command:

    sudo lsof -n -P +c 0 -i @169.254.169.254

    The output resembles the following:

    COMMAND         PID USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
    google_network_ 798 root    7u  IPv4 1674967626      0t0  TCP 10.128.0.4:44876->169.254.169.254:80 (ESTABLISHED)
    google_accounts 805 root    5u  IPv4 1674980506      0t0  TCP 10.128.0.4:44878->169.254.169.254:80 (ESTABLISHED)
    google_clock_sk 809 root    5u  IPv4 1674982496      0t0  TCP 10.128.0.4:44880->169.254.169.254:80 (ESTABLISHED)
    google_clock_sk 809 root    6u  IPv4 1674914460      0t0  TCP 10.128.0.4:44874->169.254.169.254:80 (CLOSE_WAIT)
    

    The NAME column shows the source IP, destination IP, and ports. If this isn't enough to identify a process, you can use the ps command to view more information.

    ps 798 805 809

    The output resembles the following:

    PID TTY      STAT   TIME COMMAND
    798 ?        Ss     9:07 /usr/bin/python /usr/bin/google_network_daemon
    805 ?        Ss    14:19 /usr/bin/python /usr/bin/google_accounts_daemon
    809 ?        Ss     4:33 /usr/bin/python /usr/bin/google_clock_skew_daemon
    

    auditd (Linux only)

    Auditd is a daemon that can write audit logs to disk based on a set of rules. By setting up a rule for the connect system call, you can identify which process creates each connection to the metadata server deprecated endpoints.

    1. To set up a rule to audit connect system calls, on your VM instance, run the following command:

      sudo auditctl -a exit,always -F arch=b64 -S connect -k CONNECT
    2. To query for connections to the deprecated endpoints, on your VM instance, run the following command:

      sudo ausearch -m SOCKADDR -k CONNECT --interpret | grep -C 3 'computeMetadata/v1beta1\|0\.1/meta-data'

      Your output might resemble the following:

      type=PROCTITLE msg=audit(09/26/2019 22:20:21.498:847) : proctitle=curl metadata.goog/computeMetadata/v1beta1/instance/id
      type=SOCKADDR msg=audit(09/26/2019 22:20:21.498:847) : saddr={ fam=inet laddr=169.254.169.254 lport=80 }
      type=SYSCALL msg=audit(09/26/2019 22:20:21.498:847) : arch=x86_64 syscall=connect success=no exit=EINPROGRESS(Operation now in progress) a0=0x3 a1=0x7fffd965bd50 a2=0x10 a3=0x14856d3402026e items=0 ppid=27641 pid=28977 auid=alice uid=alice gid=alice euid=alice suid=alice fsuid=alice egid=alice sgid=alice fsgid=alice tty=pts9 ses=2828 comm=curl exe=/usr/bin/curl key=CONNECT
      

      The output contains the following information:

      • The process title: proctitle
      • The request url: curl metadata.goog/computeMetadata/v1beta1/instance/id
      • The location of the executable: exe=/usr/bin/curl
      • The user: uid=alice

      After you have identified the processes, you can remove the rule. To remove the rule, run the following command:

      sudo auditctl -d exit,always -F arch=b64 -S connect -k CONNECT

    Differences between the v1beta1 and v1.0 metadata server endpoints

    The v1 metadata server functions slightly differently than the previous v1beta1 server. The new metadata server requires that all requests provide the Metadata-Flavor: Google header, which indicates that the request was made with the intention of retrieving metadata values.

    Update your requests to include this new header. For example, a request to the disks/ attribute now looks like the following:

    user@myinst:~$ curl "http://metadata.google.internal/computeMetadata/v1/instance/disks/" -H "Metadata-Flavor: Google"
    

    Differences between the v0.1 and v1.0 metadata server endpoints

    In general, the v0.1 metadata server endpoints differ from v1 in the following ways:

    • The root used for querying metadata server endpoint has changed from http://metadata.google.internal/0.1/meta-data/ to http://metadata.google.internal/computeMetadata/v1/.
    • v1 metadata server requests must include the Metadata-Flavor: Google header. For example, a request to the disks/ attribute now looks like the following:

      curl "http://metadata.google.internal/computeMetadata/v1/instance/disks/" -H "Metadata-Flavor: Google"
    • Custom metadata can be queried at both the project and instance level.

      In the v0.1 metadata server, for the attributes/ property, both instance and project metadata values are in the same directory. If an attribute has both instance and project metadata values, the instance metadata value for a given key is returned.

      For v1 queries, the URL must specify whether instance or project metadata is requested. For example, to query the sshKeys attribute, run one or both of the following:

      For sshKeys that are set on the project, run the following command:

      curl "http://metadata.google.internal/computeMetadata/v1/project/attributes/sshKeys" -H "Metadata-Flavor: Google"

      For sshKeys that are set on this instance, run the following command:

      curl "http://metadata.google.internal/computeMetadata/v1/instance/attributes/sshKeys" -H "Metadata-Flavor: Google"
    • Some metadata entries in v0.1 return JSON objects. In the v1 metadata server, these metadata entries are organized by directories. This organization of entries lets you query specific index entries and values for these previously nested fields. For more information about querying a directory listing or accessing JSON objects in v1, see Attributes as directory listings.

    For a more detailed mapping of v0.1 to v1 metadata server endpoints, review the following tables.

    Instance attributes

    The following metadata entries are moved into the instance/ directory:

    v0.1 root - http://metadata.google.internal/0.1/meta-data/
    v1 root - http://metadata.google.internal/computeMetadata/v1
    Metadata entry Version URI (relative to the root) Example output
    Description v0.1 description My instance description
    v1 instance/description
    Hostname v0.1 hostname my-instance.c.my-project.internal
    v1 instance/hostname
    Image v0.1 image projects/eip-images/global/images/debian-9-drawfork-v20180109
    v1 instance/image
    Machine type v0.1 machine-type projects/12345/machineTypes/n1-standard-1
    v1 instance/machine-type
    Instance tags v0.1 tags ["cheese", "lettuce"]
    v1 instance/tags
    Zone v0.1 zone projects/12345/zones/us-central1-c
    v1 instance/zone

    Project attributes

    The following metadata entries are moved into the project/ directory:

    v0.1 root - http://metadata.google.internal/0.1/meta-data/
    v1 root - http://metadata.google.internal/computeMetadata/v1
    Metadata entry Version URI (relative to root) Example output
    Project ID v0.1 project-id my-project
    v1 project/project-id
    Numeric Project ID v0.1 numeric-project-id 12345
    v1 project/numeric-project-id

    Attributes that are moved

    The following metadata entries are moved to a new endpoint:

    v0.1 root - http://metadata.google.internal/0.1/meta-data/
    v1 root - http://metadata.google.internal/computeMetadata/v1
    Metadata entry Version URI (relative to root) Example output
    Domain v0.1 domain c.my-project.internal
    v1 instance/hostname my-instance.c.my-project.internal

    Attributes that are renamed

    The following metadata entries are renamed:

    v0.1 root - http://metadata.google.internal/0.1/meta-data/
    v1 root - http://metadata.google.internal/computeMetadata/v1
    Metadata entry Version URI (relative to root) Example output
    Instance ID v0.1 instance-id 123456789
    v1 instance/id 123456789

    Attributes as directory listings

    The following metadata entries returned JSON objects in v0.1 and are organized into directories in v1.

    For these metadata entries, you can use either of the following methods to access the endpoints:

    v0.1 root - http://metadata.google.internal/0.1/meta-data/
    v1 root - http://metadata.google.internal/computeMetadata/v1
    Metadata entry Version URI (relative to root) Example output
    Disks v0.1 disks
    {"disks":[{
    "deviceName":"persistent-disk-0",
    "index":0,
    "mode":"READ_WRITE",
    "type":"PERSISTENT"}]}
    v1 instance/attached-disks/?recursive=true
    [{"deviceName": "persistent-disk-0",
    "index":0,
    "mode":"READ_WRITE",
    "type":"PERSISTENT"}]
    Network interfaces v0.1 network
    {"networkInterface":
    [{"accessConfiguration":
    [{"externalIp":"35.194.6.47",
    "type":"ONE_TO_ONE_NAT"}],
    "ip":"10.128.0.4",
    "mac":"42:01:0a:80:00:04","mtu":1460,
    "network":"projects/12345/networks/default"}]}
    v1 instance/network-interfaces/?recursive=true
    [{"accessConfigs":
    [{"externalIp":"35.194.6.47",
    "type":"ONE_TO_ONE_NAT"}],
    "dnsServers":["169.254.169.254"],
    "forwardedIps":[],
    "gateway":"10.128.0.1",
    "ip":"10.128.0.4",
    "ipAliases [],
    "mac":"42:01:0a:80:00:04","mtu":1460,
    "network":"projects/12345/networks/default",
    "subnetmask":"255.255.240.0","targetInstanceIps":[]}]
    Service account information (all service accounts) v0.1 service-accounts
    {"serviceAccounts":
    [{"scopes":["https://www.googleapis.com/auth/devstorage.read_only"],
    "serviceAccount":"12345-compute@developer.gserviceaccount.com"}]}
    v1 instance/service-accounts/?recursive=true
    {"123451-compute@developer.gserviceaccount.com":
    {"aliases":["default"],
    "email":"123451-compute@developer.gserviceaccount.com",
    "scopes":["https://www.googleapis.com/auth/devstorage.read_only"]},
    "default":{"aliases":["default"],
    "email":"123451-compute@developer.gserviceaccount.com",
    "scopes":["https://www.googleapis.com/auth/devstorage.read_only"]}}
    Service account information (single service account) v0.1 service-accounts/<email|default>
    {"scopes":["https://www.googleapis.com/auth/devstorage.read_only"],
    "serviceAccount":"123451-compute@developer.gserviceaccount.com"}
    
    v1 instance/service-accounts/<email|default>/?recursive=true
    {"aliases":["default"],
    "email":"123451-compute@developer.gserviceaccount.com",
    "scopes":["https://www.googleapis.com/auth/devstorage.read_only"]}
    

    Querying OAuth 2.0 access tokens

    The following metadata entries return OAuth2 access tokens.

    v0.1 root - http://metadata.google.internal/0.1/meta-data/
    v1 root - http://metadata.google.internal/computeMetadata/v1
    Metadata entry Version URI (relative to root) Example output
    OAuth2 Token v0.1 (Method 1)

    auth-token?service_account=<email|default>&scope=scopeA,scopeB

    or

    auth_token?service_account=<email|default>&scope=scopeA,scopeB

    {"expires":1568854217,
    "oauth2_access_token":"ya29.c.KmyIB0i4tH1xLzKGrqeeK6TGWEW3b18Lcq...."}
    
    v0.1 (Method 2) service-accounts/<email|default>/acquire
    {"accessToken":"ya29.c.KmyIB0i4tH1xLNzKGrqeeK6TGWEW3b18Lcq.....",
    "expiresAt":1568854217,"expiresIn":2022}
    v1 instance/service-accounts/<email|default>/token
    {"access_token":"ya29.c.KmyIB0i4tH1xLNzKGrqeeK6TGWEW3b18Lcq....",
    "expires_in":2022,"token_type":"Bearer"}

    Applications that require an update

    To use the v1 metadata server endpoint, update the following applications to the minimum supported version or a higher version.

    Application Minimum supported version
    Facter (developed by Puppet) 3.14.5
    Stackdriver Monitoring agent 5.5.2-375

    Supported library versions

    Some Google libraries are unaffected by the migration to the v1 metadata server endpoint. However, if you are using any of the libraries listed in the following table, make sure that you upgrade to the minimum supported version or a higher version.

    Language Google library Minimum supported version
    Java com.google.api-client 1.18.0-rc
    Node.js google-cloud/debug-agent 3.0.0
    Node.js google-cloud/profiler 0.2.0
    Node.js google-cloud/trace-agent 2.11.0
    Node.js gcp-metadata 0.5.0
    Node.js gcs-resumable-upload 0.13.0
    Node.js googleapis 27.0.0
    Node.js google-auth-library 1.3.0
    Node.js google-cloud/bigquery 2.0.0
    Node.js google-cloud/bigquery-data-transfer 0.4.0
    Node.js google-cloud/bigtable 0.15.0
    Node.js google-cloud/cloud-container 0.3.0
    Node.js google-cloud/compute 0.11.0
    Node.js google-cloud/datastore 2.0.0
    Node.js dialogflow 0.6.0
    Node.js google-cloud/dlp 0.8.0
    Node.js google-cloud/dns 0.8.0
    Node.js google-cloud/error-reporting 0.5.0
    Node.js google-cloud/firestore 0.16.1
    Node.js google-cloud/language 2.0.0
    Node.js google-cloud/logging-bunyan 0.9.0
    Node.js google-cloud/logging 2.0.0
    Node.js google-cloud/logging-winston 0.10.0
    Node.js google-cloud/monitoring 0.6.0
    Node.js google-cloud/os-login 0.3.0
    Node.js google-cloud/pubsub 0.20.0
    Node.js google-cloud/redis 0.2.0
    Node.js google-cloud/resource 0.9.0
    Node.js google-cloud/spanner 2.0.0
    Node.js google-cloud/speech 2.0.0
    Node.js google-cloud/storage 2.0.0
    Node.js google-cloud/tasks 0.2.0
    Node.js google-cloud/text-to-speech 0.3.0
    Node.js google-cloud/translate 2.0.0
    Node.js google-cloud/vision 0.21.0
    Node.js google-cloud/profiler 0.2.0
    Node.js google-cloud/trace-agent 2.11.0
    Node.js google-gax 0.17.0
    Node.js gce-images 1.0.0
    Node.js gcp-metadata 0.5.0
    Node.js gcs-resumable-upload 0.13.0
    Node.js googleapis 27.0.0
    Node.js google-auth-library 1.3.0
    Node.js google-cloud/bigquery 2.0.0
    Node.js google-cloud/bigquery-data-transfer 0.4.0
    Node.js google-cloud/bigtable 0.15.0
    Node.js google-cloud/cloud-container 0.3.0
    Node.js google-cloud/common 0.18.0
    Node.js google-cloud/common-grpc 0.7.0
    Node.js google-cloud/compute 0.11.0
    Node.js google-cloud/datastore 2.0.0
    Node.js dialogflow 0.6.0
    Node.js google-cloud/dlp 0.8.0
    Node.js google-cloud/dns 0.8.0
    Node.js google-cloud/error-reporting 0.5.0
    Node.js google-cloud/firestore 0.16.1
    Node.js google-cloud/language 2.0.0
    Node.js google-cloud/logging 2.0.0
    Node.js google-cloud/logging-bunyan 0.9.0
    Node.js google-cloud/logging-winston 0.10.0
    Node.js google-cloud/monitoring 0.6.0
    Node.js google-cloud/os-login 0.3.0
    Node.js googleapis/nodejs-pubsub 0.20.0
    Node.js google-cloud/redis 0.2.0
    Node.js google-cloud/resource 0.9.0
    Node.js google-cloud/spanner 2.0.0
    Node.js google-cloud/speech 2.0.0
    Node.js google-cloud/storage 2.0.0
    Node.js google-cloud/tasks 0.2.0
    Node.js google-cloud/text-to-speech 0.3.0
    Node.js google-cloud/translate 2.0.0
    Node.js google-cloud/video-intelligence 1.3.0
    Node.js google-cloud/vision 0.21.0
    PHP google-cloud/video-intelligence 1.3.3
    Python oauth2client 2.0.0
    Python google-api-python-client 1.6.0
    Python googleapis/google-cloud-python 0.10.0
    Python google-cloud-happybase 0.20.0
    Ruby gcloud 0.11.1
    Ruby google-api-client 0.8.6
Var denne side nyttig? Giv os en anmeldelse af den:

Send feedback om...

Compute Engine Documentation