Migrating to v1 metadata server endpoint

Warning: The v0.1 and v1beta1 metadata server endpoints were deprecated on September 30, 2020 and are in the process of being shutdown.

Migration overview

To migrate to v1.0, complete the following steps:

1. Identify the VM instances that are using the deprecated metadata server endpoints.

   If you have Google Kubernetes Engine clusters, you need to identify the GKE nodes that are using the deprecated endpoint.

2. On the identified VM instances or nodes, find the processes, applications, or images that are using the deprecated metadata server endpoints.

   For processes and applications, review the following information:

   • If the process belongs to an application that you did not develop, update the application to use a v1 metadata server endpoint. For a list of known applications that require an update, see applications that require an update.
   • If the process belongs to an application that you developed and your code is calling any of the legacy endpoints, complete the following steps:
     1. Review the differences between v1 and the deprecated metadata server endpoints.
        • For v1beta1, see Differences between v1beta1 and v1.0.
        • For v0.1, see Differences between v0.1 and v1.0.
     2. Update queries to use the v1 metadata server endpoint.
   • If the process belongs to an application that you developed but your code is not making requests to any of the legacy endpoints, then the requests might be made by SDKs or other dependencies. To fix this, update all SDKs and dependencies used by the application.To view the versions of the Google libraries that require an update, see supported library versions.

3. (Optional) Disable the v0.1 and v1beta1 metadata server endpoints.

   Identifying the VM instances

   To identify which VM instances are using v0.1 and v1beta1 endpoints, you can make requests to two new endpoints.

   The output from querying these new endpoints tells you how many times a given VM instance accessed the deprecated endpoints. Note: Each time a VM instance is stopped, the counter resets.

   To see if a VM instance has accessed the v0.1 and v1beta1 endpoints, run the following commands:

   curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/legacy-endpoint-access/0.1
   curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/legacy-endpoint-access/v1beta1
   

   The following python sample shows how to programmatically watch these new endpoints:

   compute/metadata/detect_legacy_usage.py

   View on GitHub

   def wait_for_legacy_usage(callback):
       url = '{}/instance/legacy-endpoint-access'.format(METADATA_URL)
       last_etag = '0'
       counts = None
       while True:
           r = requests.get(
               url,
               params={
                   'last_etag': last_etag,
                   'recursive': True,
                   'wait_for_change': True
               },
               headers=METADATA_HEADERS)
           if r.status_code == 503:  # Metadata server unavailable
               print('Metadata server unavailable. Sleeping for 1 second.')
               time.sleep(1)
               continue
           if r.status_code == 404:  # Feature not yet supported
               print('Legacy endpoint access not supported. Sleeping for 1 hour.')
               time.sleep(3600)
               continue
           r.raise_for_status()
   
           last_etag = r.headers['etag']
           access_info = json.loads(r.text)
           if not counts:
               counts = access_info
           if access_info != counts:
               diff = {
                   ver: access_info[ver] - counts[ver] for ver in counts
               }
               counts = access_info
               callback(diff)

   Identifying the GKE nodes

   To identify the nodes in your Google Kubernetes Engine clusters that are using the v0.1 and v1beta1 endpoints, see Identifying workloads using the legacy metadata server endpoints.

   Identifying the processes

   After you have identified the VM instances that are making requests to the deprecated metadata server endpoints, you can then find the processes on these VM instances that are making these requests.

   To identify the processes, use logging or monitoring tools such as ngrep or auditd.

   ngrep

   You can use ngrep (network grep) to collect packets and filter the contents of these packets. To collect packet information for processes that make requests to the deprecated endpoints, on your VM instance, run the following command:

   sudo ngrep -l -q "v1beta1|0\.1/meta-data" tcp \
   and dst host 169.254.169.254 or metadata.google.internal \
   and dst port 80
   

   If any requests are made to the deprecated endpoints, the output resembles the following:

   T 10.128.0.4:41312 -> 169.254.169.254:80 [AP]
   GET /computeMetadata/v1beta1/instance/id HTTP/1.1..Host: metadata.goog..User-Agent: curl/7.52.1..Accept: */*....
   

   The output contains the following information:

   • The root url: /computeMetadata/v1beta1/instance/id.
   • The user agent that is used to make the request. In this example, it is curl/7.52.1.
   • The IP address (source IP) and port for the VM instance that the request originated from. In this example, the IP address is 10.128.0.4 and the port is 41312 .
   • If the process is still alive, you can review the port information to find the process ID.

   Reviewing port information

   The following procedure might identify processes with outstanding hanging HTTP GET requests. This procedure finds the processes that are talking to both the current and deprecated metadata endpoints.

   To list the sockets with open connections to the metadata server, on your VM instance, run the following command:

   sudo lsof -n -P +c 0 -i @169.254.169.254

   The output resembles the following:

   COMMAND         PID USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
   google_network_ 798 root    7u  IPv4 1674967626      0t0  TCP 10.128.0.4:44876->169.254.169.254:80 (ESTABLISHED)
   google_accounts 805 root    5u  IPv4 1674980506      0t0  TCP 10.128.0.4:44878->169.254.169.254:80 (ESTABLISHED)
   google_clock_sk 809 root    5u  IPv4 1674982496      0t0  TCP 10.128.0.4:44880->169.254.169.254:80 (ESTABLISHED)
   google_clock_sk 809 root    6u  IPv4 1674914460      0t0  TCP 10.128.0.4:44874->169.254.169.254:80 (CLOSE_WAIT)
   

   The NAME column shows the source IP, destination IP, and ports. If this isn't enough to identify a process, you can use the ps command to view more information.

   ps 798 805 809

   The output resembles the following:

   PID TTY      STAT   TIME COMMAND
   798 ?        Ss     9:07 /usr/bin/python /usr/bin/google_network_daemon
   805 ?        Ss    14:19 /usr/bin/python /usr/bin/google_accounts_daemon
   809 ?        Ss     4:33 /usr/bin/python /usr/bin/google_clock_skew_daemon
   

   auditd (Linux only)

   Auditd is a daemon that can write audit logs to disk based on a set of rules. By setting up a rule for the connect system call, you can identify which process creates each connection to the metadata server deprecated endpoints.

   Note: Each system call is checked against these rules, so you might experience degraded performance if you perform monitoring with this tool. Also, capturing information for every connect system call might cause a large amount of data to be written to disk.

   1. To set up a rule to audit connect system calls, on your VM instance, run the following command:

      sudo auditctl -a exit,always -F arch=b64 -S connect -k CONNECT

   2. To query for connections to the deprecated endpoints, on your VM instance, run the following command:

      sudo ausearch -m SOCKADDR -k CONNECT --interpret | grep -C 3 'computeMetadata/v1beta1\|0\.1/meta-data'

      Your output might resemble the following:

      type=PROCTITLE msg=audit(09/26/2019 22:20:21.498:847) : proctitle=curl metadata.goog/computeMetadata/v1beta1/instance/id
      type=SOCKADDR msg=audit(09/26/2019 22:20:21.498:847) : saddr={ fam=inet laddr=169.254.169.254 lport=80 }
      type=SYSCALL msg=audit(09/26/2019 22:20:21.498:847) : arch=x86_64 syscall=connect success=no exit=EINPROGRESS(Operation now in progress) a0=0x3 a1=0x7fffd965bd50 a2=0x10 a3=0x14856d3402026e items=0 ppid=27641 pid=28977 auid=alice uid=alice gid=alice euid=alice suid=alice fsuid=alice egid=alice sgid=alice fsgid=alice tty=pts9 ses=2828 comm=curl exe=/usr/bin/curl key=CONNECT
      

      The output contains the following information:

      • The process title: proctitle
      • The request url: curl metadata.goog/computeMetadata/v1beta1/instance/id
      • The location of the executable: exe=/usr/bin/curl
      • The user: uid=alice

      After you have identified the processes, you can remove the rule. To remove the rule, run the following command:

      sudo auditctl -d exit,always -F arch=b64 -S connect -k CONNECT

   Differences between the v1beta1 and v1.0 metadata server endpoints

   The v1 metadata server functions slightly differently than the previous v1beta1 server. The new metadata server requires that all requests provide the Metadata-Flavor: Google header, which indicates that the request was made with the intention of retrieving metadata values.

   Update your requests to include this new header. For example, a request to the disks/ attribute now looks like the following:

   user@myinst:~$ curl "http://metadata.google.internal/computeMetadata/v1/instance/disks/" -H "Metadata-Flavor: Google"
   

   Differences between the v0.1 and v1.0 metadata server endpoints

   In general, the v0.1 metadata server endpoints differ from v1 in the following ways:

   • The root used for querying metadata server endpoint has changed from http://metadata.google.internal/0.1/meta-data/ to http://metadata.google.internal/computeMetadata/v1/.

   • v1 metadata server requests must include the Metadata-Flavor: Google header. For example, a request to the disks/ attribute now looks like the following:

     curl "http://metadata.google.internal/computeMetadata/v1/instance/disks/" -H "Metadata-Flavor: Google"

   • Custom metadata can be queried at both the project and instance level.

     In the v0.1 metadata server, for the attributes/ property, both instance and project metadata values are in the same directory. If an attribute has both instance and project metadata values, the instance metadata value for a given key is returned.

     For v1 queries, the URL must specify whether instance or project metadata is requested. For example, to query the sshKeys attribute, run one or both of the following:

     For sshKeys that are set on the project, run the following command:

     curl "http://metadata.google.internal/computeMetadata/v1/project/attributes/sshKeys" -H "Metadata-Flavor: Google"

     For sshKeys that are set on this instance, run the following command:

     curl "http://metadata.google.internal/computeMetadata/v1/instance/attributes/sshKeys" -H "Metadata-Flavor: Google"

   • Some metadata entries in v0.1 return JSON objects. In the v1 metadata server, these metadata entries are organized by directories. This organization of entries lets you query specific index entries and values for these previously nested fields. For more information about querying a directory listing or accessing JSON objects in v1, see Attributes as directory listings.

   For a more detailed mapping of v0.1 to v1 metadata server endpoints, review the following tables.

   Instance attributes

   The following metadata entries are moved into the instance/ directory:

   v0.1 root - http://metadata.google.internal/0.1/meta-data/
   v1 root - http://metadata.google.internal/computeMetadata/v1
   Metadata entry	Version	URI (relative to the root)	Example output
   Description	v0.1	description	My instance description
   	v1	instance/description
   Hostname	v0.1	hostname	my-instance.c.my-project.internal
   	v1	instance/hostname
   Image	v0.1	image	projects/eip-images/global/images/debian-9-drawfork-v20180109
   	v1	instance/image
   Machine type	v0.1	machine-type	projects/12345/machineTypes/e2-standard-2
   	v1	instance/machine-type
   Instance tags	v0.1	tags	["cheese", "lettuce"]
   	v1	instance/tags
   Zone	v0.1	zone	projects/12345/zones/us-central1-c
   	v1	instance/zone

   Project attributes

   The following metadata entries are moved into the project/ directory:

   v0.1 root - http://metadata.google.internal/0.1/meta-data/
   v1 root - http://metadata.google.internal/computeMetadata/v1
   Metadata entry	Version	URI (relative to root)	Example output
   Project ID	v0.1	project-id	my-project
   	v1	project/project-id
   Numeric Project ID	v0.1	numeric-project-id	12345
   	v1	project/numeric-project-id

   Attributes that are moved

   The following metadata entries are moved to a new endpoint:

   v0.1 root - http://metadata.google.internal/0.1/meta-data/
   v1 root - http://metadata.google.internal/computeMetadata/v1
   Metadata entry	Version	URI (relative to root)	Example output
   Domain	v0.1	domain	c.my-project.internal
   	v1	instance/hostname	my-instance.c.my-project.internal

   Attributes that are renamed

   The following metadata entries are renamed:

   v0.1 root - http://metadata.google.internal/0.1/meta-data/
   v1 root - http://metadata.google.internal/computeMetadata/v1
   Metadata entry	Version	URI (relative to root)	Example output
   Instance ID	v0.1	instance-id	123456789
   	v1	instance/id	123456789

   Attributes as directory listings

   The following metadata entries returned JSON objects in v0.1 and are organized into directories in v1.

   For these metadata entries, you can use either of the following methods to access the endpoints:

   • Query the directory listings.
   • Recursively query the v1 endpoints to get JSON objects similar to those in v0.1.
   Note: Some fields in the JSON object have changed in the v1 metadata server endpoint. To see the changed fields, review the example outputs in the table below.
   Note: VPC networks have a default maximum transmission unit (MTU) of 1460 bytes. However, the network MTU can be set to 1500 bytes. See Maximum transmission unit for background on network MTUs.

   v0.1 root - http://metadata.google.internal/0.1/meta-data/
   v1 root - http://metadata.google.internal/computeMetadata/v1
   Metadata entry	Version	URI (relative to root)	Example output
   Disks	v0.1	disks	{"disks":[{ "deviceName":"persistent-disk-0", "index":0, "mode":"READ_WRITE", "type":"PERSISTENT"}]}
   	v1	instance/attached-disks/?recursive=true	[{"deviceName": "persistent-disk-0", "index":0, "mode":"READ_WRITE", "type":"PERSISTENT"}]
   Network interfaces	v0.1	network	{"networkInterface": [{"accessConfiguration": [{"externalIp":"35.194.6.47", "type":"ONE_TO_ONE_NAT"}], "ip":"10.128.0.4", "mac":"42:01:0a:80:00:04","mtu":1460, "network":"projects/12345/networks/default"}]}
   	v1	instance/network-interfaces/?recursive=true	[{"accessConfigs": [{"externalIp":"35.194.6.47", "type":"ONE_TO_ONE_NAT"}], "dnsServers":["169.254.169.254"], "forwardedIps":[], "gateway":"10.128.0.1", "ip":"10.128.0.4", "ipAliases [], "mac":"42:01:0a:80:00:04","mtu":1460, "network":"projects/12345/networks/default", "subnetmask":"255.255.240.0","targetInstanceIps":[]}]
   Service account information (all service accounts)	v0.1	service-accounts	{"serviceAccounts": [{"scopes":["https://www.googleapis.com/auth/devstorage.read_only"], "serviceAccount":"[e-mail]"}]}
   	v1	instance/service-accounts/?recursive=true	{"[e-mail]": {"aliases":["default"], "email":"[e-mail]", "scopes":["https://www.googleapis.com/auth/devstorage.read_only"]}, "default":{"aliases":["default"], "email":"[e-mail]", "scopes":["https://www.googleapis.com/auth/devstorage.read_only"]}}
   Service account information (single service account)	v0.1	service-accounts/<email|default>	{"scopes":["https://www.googleapis.com/auth/devstorage.read_only"], "serviceAccount":"[e-mail]"}
   	v1	instance/service-accounts/<email|default>/?recursive=true	{"aliases":["default"], "email":"[e-mail]", "scopes":["https://www.googleapis.com/auth/devstorage.read_only"]}

   Querying OAuth 2.0 access tokens

   The following metadata entries return OAuth2 access tokens.

   Note: Some OAuth 2.0 token fields have changed in the v1 metadata server endpoint. To see the changed fields, review the example outputs in the following table.

   v0.1 root - http://metadata.google.internal/0.1/meta-data/
   v1 root - http://metadata.google.internal/computeMetadata/v1
   Metadata entry	Version	URI (relative to root)	Example output
   OAuth2 Token	v0.1 (Method 1)	auth-token?service_account=<email|default>&scope=scopeA,scopeB or auth_token?service_account=<email|default>&scope=scopeA,scopeB	{"expires":1568854217, "oauth2_access_token":"ya29.c.KmyIB0i4tH1xLzKGrqeeK6TGWEW3b18Lcq...."}
   	v0.1 (Method 2)	service-accounts/<email|default>/acquire	{"accessToken":"ya29.c.KmyIB0i4tH1xLNzKGrqeeK6TGWEW3b18Lcq.....", "expiresAt":1568854217,"expiresIn":2022}
   	v1	instance/service-accounts/<email|default>/token	{"access_token":"ya29.c.KmyIB0i4tH1xLNzKGrqeeK6TGWEW3b18Lcq....", "expires_in":2022,"token_type":"Bearer"}

   Applications that require an update

   To use the v1 metadata server endpoint, update the following applications to the minimum supported version or a higher version.

   Note: This list is not exhaustive and will be updated as more applications are identified.

   Application	Minimum supported version
   Facter (developed by Puppet)	3.14.5(Puppet 6.10.0) 3.11.10(Puppet 5.5.17)
   Cloud Monitoring agent	5.5.2-375
   RightScale's server agent (RightLink)	RightLink 10

   Images to update

   If you have guest attributes enabled, the following images make calls to the v1beta1 metadata server to store SSH host keys. We recommend that you upgrade to later versions of the images that are listed.

   Operating system	Image versions
   CentOS	centos-6-v20190619 centos-7-v20190619
   Debian	debian-9-stretch-v20190618
   Red Hat Enterprise Linux (RHEL)	rhel-6-v20190618 rhel-7-v20190618 rhel-8-v20190521 rhel-8-v20190522 rhel-8-v20190618 rhel-7-4-sap-v20190618 rhel-7-6-sap-v20190618
   SUSE Linux Enterprise Server (SLES)	sles-15-sp1-v20190625 sles-15-sp1-sap-v20190625

   Supported library versions

   Some Google libraries are unaffected by the migration to the v1 metadata server endpoint. However, if you are using any of the libraries listed in the following table, make sure that you upgrade to the minimum supported version or a higher version.

   Language	Google library	Minimum supported version
   Java	com.google.api-client	1.18.0-rc
   Node.js	dialogflow	0.6.0
   Node.js	firebase-admin	8.2.0
   Node.js	google-cloud/debug-agent	3.0.0
   Node.js	google-cloud/profiler	0.2.0
   Node.js	google-cloud/trace-agent	2.11.0
   Node.js	gcp-metadata	0.5.0
   Node.js	gcs-resumable-upload	0.13.0
   Node.js	googleapis	27.0.0
   Node.js	google-auth-library	1.3.0
   Node.js	google-cloud/bigquery	2.0.0
   Node.js	google-cloud/bigquery-data-transfer	0.4.0
   Node.js	google-cloud/bigtable	0.15.0
   Node.js	google-cloud/cloud-container	0.3.0
   Node.js	google-cloud/compute	0.11.0
   Node.js	google-cloud/datastore	2.0.0
   Node.js	google-cloud/dlp	0.8.0
   Node.js	google-cloud/dns	0.8.0
   Node.js	google-cloud/error-reporting	0.5.0
   Node.js	google-cloud/firestore	0.16.1
   Node.js	google-cloud/language	2.0.0
   Node.js	google-cloud/logging-bunyan	0.9.0
   Node.js	google-cloud/logging	2.0.0
   Node.js	google-cloud/logging-winston	0.10.0
   Node.js	google-cloud/monitoring	0.6.0
   Node.js	google-cloud/os-login	0.3.0
   Node.js	google-cloud/pubsub	0.20.0
   Node.js	google-cloud/redis	0.2.0
   Node.js	google-cloud/resource	0.9.0
   Node.js	google-cloud/spanner	2.0.0
   Node.js	google-cloud/speech	2.0.0
   Node.js	google-cloud/storage	2.0.0
   Node.js	google-cloud/tasks	0.2.0
   Node.js	google-cloud/text-to-speech	0.3.0
   Node.js	google-cloud/translate	2.0.0
   Node.js	google-cloud/vision	0.21.0
   Node.js	google-cloud/profiler	0.2.0
   Node.js	google-cloud/trace-agent	2.11.0
   Node.js	google-gax	0.17.0
   Node.js	gce-images	1.0.0
   Node.js	gcp-metadata	0.5.0
   Node.js	gcs-resumable-upload	0.13.0
   Node.js	googleapis	27.0.0
   Node.js	google-auth-library	1.3.0
   Node.js	google-cloud/bigquery	2.0.0
   Node.js	google-cloud/bigquery-data-transfer	0.4.0
   Node.js	google-cloud/bigtable	0.15.0
   Node.js	google-cloud/cloud-container	0.3.0
   Node.js	google-cloud/common	0.18.0
   Node.js	google-cloud/common-grpc	0.7.0
   Node.js	google-cloud/compute	0.11.0
   Node.js	google-cloud/datastore	2.0.0
   Node.js	google-cloud/dlp	0.8.0
   Node.js	google-cloud/dns	0.8.0
   Node.js	google-cloud/error-reporting	0.5.0
   Node.js	google-cloud/firestore	0.16.1
   Node.js	google-cloud/language	2.0.0
   Node.js	google-cloud/logging	2.0.0
   Node.js	google-cloud/logging-bunyan	0.9.0
   Node.js	google-cloud/logging-winston	0.10.0
   Node.js	google-cloud/monitoring	0.6.0
   Node.js	google-cloud/os-login	0.3.0
   Node.js	googleapis/nodejs-pubsub	0.20.0
   Node.js	google-cloud/redis	0.2.0
   Node.js	google-cloud/resource	0.9.0
   Node.js	google-cloud/spanner	2.0.0
   Node.js	google-cloud/speech	2.0.0
   Node.js	google-cloud/storage	2.0.0
   Node.js	google-cloud/tasks	0.2.0
   Node.js	google-cloud/text-to-speech	0.3.0
   Node.js	google-cloud/translate	2.0.0
   Node.js	google-cloud/video-intelligence	1.3.0
   Node.js	google-cloud/vision	0.21.0
   PHP	google-cloud/video-intelligence	1.3.3
   Python	oauth2client	2.0.0
   Python	google-api-python-client	1.6.0
   Python	googleapis/google-cloud-python	0.10.0
   Python	google-cloud-happybase	0.20.0
   Ruby	gcloud	0.11.1
   Ruby	google-api-client	0.8.6