Migration overview
To migrate to v1.0, complete the following steps:
Identify the VM instances that are using the deprecated metadata server endpoints.
If you have Google Kubernetes Engine clusters, you need to identify the GKE nodes that are using the deprecated endpoint.
On the identified VM instances or nodes, find the processes, applications, or images that are using the deprecated metadata server endpoints.
For processes and applications, review the following information:
- If the process belongs to an application that you did not develop, update the application to use a v1 metadata server endpoint. For a list of known applications that require an update, see applications that require an update.
- If the process belongs to an application that you developed and your code
is calling any of the legacy endpoints, complete the following steps:
- Review the differences between v1 and the deprecated metadata server endpoints.
- For v1beta1, see Differences between v1beta1 and v1.0.
- For v0.1, see Differences between v0.1 and v1.0.
- Update queries to use the v1 metadata server endpoint.
- Review the differences between v1 and the deprecated metadata server endpoints.
- If the process belongs to an application that you developed but your code is not making requests to any of the legacy endpoints, then the requests might be made by SDKs or other dependencies. To fix this, update all SDKs and dependencies used by the application.To view the versions of the Google libraries that require an update, see supported library versions.
(Optional) Disable the v0.1 and v1beta1 metadata server endpoints.
Identifying the VM instances
To identify which VM instances are using v0.1 and v1beta1 endpoints, you can make requests to two new endpoints.
The output from querying these new endpoints tells you how many times a given VM instance accessed the deprecated endpoints. Note: Each time a VM instance is stopped, the counter resets.
To see if a VM instance has accessed the v0.1 and v1beta1 endpoints, run the following commands:
curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/legacy-endpoint-access/0.1 curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/legacy-endpoint-access/v1beta1
The following python sample shows how to programmatically watch these new endpoints:
Identifying the GKE nodes
To identify the nodes in your Google Kubernetes Engine clusters that are using the v0.1 and v1beta1 endpoints, see Identifying workloads using the legacy metadata server endpoints.
Identifying the processes
After you have identified the VM instances that are making requests to the deprecated metadata server endpoints, you can then find the processes on these VM instances that are making these requests.
To identify the processes, use logging or monitoring tools such as
ngrep
orauditd
.ngrep
You can use
ngrep
(networkgrep
) to collect packets and filter the contents of these packets. To collect packet information for processes that make requests to the deprecated endpoints, on your VM instance, run the following command:sudo ngrep -l -q "v1beta1|0\.1/meta-data" tcp \ and dst host 169.254.169.254 or metadata.google.internal \ and dst port 80
If any requests are made to the deprecated endpoints, the output resembles the following:
T 10.128.0.4:41312 -> 169.254.169.254:80 [AP] GET /computeMetadata/v1beta1/instance/id HTTP/1.1..Host: metadata.goog..User-Agent: curl/7.52.1..Accept: */*....
The output contains the following information:
- The root url:
/computeMetadata/v1beta1/instance/id
. - The user agent that is used to make the request. In this example, it is
curl/7.52.1
. - The IP address (source IP) and port for the VM instance that the request
originated from. In this example, the IP address is
10.128.0.4
and the port is41312
. - If the process is still alive, you can review the port information to find the process ID.
Reviewing port information
The following procedure might identify processes with outstanding hanging HTTP GET requests. This procedure finds the processes that are talking to both the current and deprecated metadata endpoints.
To list the sockets with open connections to the metadata server, on your VM instance, run the following command:
sudo lsof -n -P +c 0 -i @169.254.169.254
The output resembles the following:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME google_network_ 798 root 7u IPv4 1674967626 0t0 TCP 10.128.0.4:44876->169.254.169.254:80 (ESTABLISHED) google_accounts 805 root 5u IPv4 1674980506 0t0 TCP 10.128.0.4:44878->169.254.169.254:80 (ESTABLISHED) google_clock_sk 809 root 5u IPv4 1674982496 0t0 TCP 10.128.0.4:44880->169.254.169.254:80 (ESTABLISHED) google_clock_sk 809 root 6u IPv4 1674914460 0t0 TCP 10.128.0.4:44874->169.254.169.254:80 (CLOSE_WAIT)
The
NAME
column shows the source IP, destination IP, and ports. If this isn't enough to identify a process, you can use theps
command to view more information.ps 798 805 809
The output resembles the following:
PID TTY STAT TIME COMMAND 798 ? Ss 9:07 /usr/bin/python /usr/bin/google_network_daemon 805 ? Ss 14:19 /usr/bin/python /usr/bin/google_accounts_daemon 809 ? Ss 4:33 /usr/bin/python /usr/bin/google_clock_skew_daemon
auditd (Linux only)
Auditd
is a daemon that can write audit logs to disk based on a set of rules. By setting up a rule for theconnect
system call, you can identify which process creates each connection to the metadata server deprecated endpoints.To set up a rule to audit
connect
system calls, on your VM instance, run the following command:sudo auditctl -a exit,always -F arch=b64 -S connect -k CONNECT
To query for connections to the deprecated endpoints, on your VM instance, run the following command:
sudo ausearch -m SOCKADDR -k CONNECT --interpret | grep -C 3 'computeMetadata/v1beta1\|0\.1/meta-data'
Your output might resemble the following:
type=PROCTITLE msg=audit(09/26/2019 22:20:21.498:847) : proctitle=curl metadata.goog/computeMetadata/v1beta1/instance/id type=SOCKADDR msg=audit(09/26/2019 22:20:21.498:847) : saddr={ fam=inet laddr=169.254.169.254 lport=80 } type=SYSCALL msg=audit(09/26/2019 22:20:21.498:847) : arch=x86_64 syscall=connect success=no exit=EINPROGRESS(Operation now in progress) a0=0x3 a1=0x7fffd965bd50 a2=0x10 a3=0x14856d3402026e items=0 ppid=27641 pid=28977 auid=alice uid=alice gid=alice euid=alice suid=alice fsuid=alice egid=alice sgid=alice fsgid=alice tty=pts9 ses=2828 comm=curl exe=/usr/bin/curl key=CONNECT
The output contains the following information:
- The process title:
proctitle
- The request url:
curl metadata.goog/computeMetadata/v1beta1/instance/id
- The location of the executable:
exe=/usr/bin/curl
- The user:
uid=alice
After you have identified the processes, you can remove the rule. To remove the rule, run the following command:
sudo auditctl -d exit,always -F arch=b64 -S connect -k CONNECT
- The process title:
Differences between the v1beta1 and v1.0 metadata server endpoints
The v1 metadata server functions slightly differently than the previous v1beta1 server. The new metadata server requires that all requests provide the
Metadata-Flavor: Google
header, which indicates that the request was made with the intention of retrieving metadata values.Update your requests to include this new header. For example, a request to the
disks/
attribute now looks like the following:user@myinst:~$ curl "http://metadata.google.internal/computeMetadata/v1/instance/disks/" -H "Metadata-Flavor: Google"
Differences between the v0.1 and v1.0 metadata server endpoints
In general, the v0.1 metadata server endpoints differ from v1 in the following ways:
- The root used for querying metadata server endpoint has changed from
http://metadata.google.internal/0.1/meta-data/
tohttp://metadata.google.internal/computeMetadata/v1/
. v1 metadata server requests must include the
Metadata-Flavor: Google
header. For example, a request to thedisks/
attribute now looks like the following:curl "http://metadata.google.internal/computeMetadata/v1/instance/disks/" -H "Metadata-Flavor: Google"
Custom metadata can be queried at both the project and instance level.
In the v0.1 metadata server, for the
attributes/
property, both instance and project metadata values are in the same directory. If an attribute has both instance and project metadata values, the instance metadata value for a given key is returned.For v1 queries, the URL must specify whether instance or project metadata is requested. For example, to query the
sshKeys
attribute, run one or both of the following:For
sshKeys
that are set on the project, run the following command:curl "http://metadata.google.internal/computeMetadata/v1/project/attributes/sshKeys" -H "Metadata-Flavor: Google"
For
sshKeys
that are set on this instance, run the following command:curl "http://metadata.google.internal/computeMetadata/v1/instance/attributes/sshKeys" -H "Metadata-Flavor: Google"
Some metadata entries in v0.1 return JSON objects. In the v1 metadata server, these metadata entries are organized by directories. This organization of entries lets you query specific index entries and values for these previously nested fields. For more information about querying a directory listing or accessing JSON objects in v1, see Attributes as directory listings.
For a more detailed mapping of v0.1 to v1 metadata server endpoints, review the following tables.
Instance attributes
The following metadata entries are moved into the
instance/
directory:v0.1
root -http://metadata.google.internal/0.1/meta-data/
v1
root -http://metadata.google.internal/computeMetadata/v1
Metadata entry Version URI (relative to the root) Example output Description v0.1
description
My instance description
v1
instance/description
Hostname v0.1
hostname
my-instance.c.my-project.internal
v1
instance/hostname
Image v0.1
image
projects/eip-images/global/images/debian-9-drawfork-v20180109
v1
instance/image
Machine type v0.1
machine-type
projects/12345/machineTypes/e2-standard-2
v1
instance/machine-type
Instance tags v0.1
tags
["cheese", "lettuce"]
v1
instance/tags
Zone v0.1
zone
projects/12345/zones/us-central1-c
v1
instance/zone
Project attributes
The following metadata entries are moved into the
project/
directory:v0.1
root -http://metadata.google.internal/0.1/meta-data/
v1
root -http://metadata.google.internal/computeMetadata/v1
Metadata entry Version URI (relative to root) Example output Project ID v0.1
project-id
my-project
v1
project/project-id
Numeric Project ID v0.1
numeric-project-id
12345
v1
project/numeric-project-id
Attributes that are moved
The following metadata entries are moved to a new endpoint:
v0.1
root -http://metadata.google.internal/0.1/meta-data/
v1
root -http://metadata.google.internal/computeMetadata/v1
Metadata entry Version URI (relative to root) Example output Domain v0.1
domain
c.my-project.internal
v1
instance/hostname
my-instance.c.my-project.internal
Attributes that are renamed
The following metadata entries are renamed:
v0.1
root -http://metadata.google.internal/0.1/meta-data/
v1
root -http://metadata.google.internal/computeMetadata/v1
Metadata entry Version URI (relative to root) Example output Instance ID v0.1
instance-id
123456789
v1
instance/id
123456789
Attributes as directory listings
The following metadata entries returned JSON objects in v0.1 and are organized into directories in v1.
For these metadata entries, you can use either of the following methods to access the endpoints:
- Query the directory listings.
- Recursively query
the
v1
endpoints to get JSON objects similar to those inv0.1
.
1460
bytes. However, the network MTU can be set to1500
bytes. See Maximum transmission unit for background on network MTUs.v0.1
root -http://metadata.google.internal/0.1/meta-data/
v1
root -http://metadata.google.internal/computeMetadata/v1
Metadata entry Version URI (relative to root) Example output Disks v0.1
disks
{"disks":[{ "deviceName":"persistent-disk-0", "index":0, "mode":"READ_WRITE", "type":"PERSISTENT"}]}
v1
instance/attached-disks/?recursive=true
[{"deviceName": "persistent-disk-0", "index":0, "mode":"READ_WRITE", "type":"PERSISTENT"}]
Network interfaces v0.1
network
{"networkInterface": [{"accessConfiguration": [{"externalIp":"35.194.6.47", "type":"ONE_TO_ONE_NAT"}], "ip":"10.128.0.4", "mac":"42:01:0a:80:00:04","mtu":1460, "network":"projects/12345/networks/default"}]}
v1
instance/network-interfaces/?recursive=true
[{"accessConfigs": [{"externalIp":"35.194.6.47", "type":"ONE_TO_ONE_NAT"}], "dnsServers":["169.254.169.254"], "forwardedIps":[], "gateway":"10.128.0.1", "ip":"10.128.0.4", "ipAliases [], "mac":"42:01:0a:80:00:04","mtu":1460, "network":"projects/12345/networks/default", "subnetmask":"255.255.240.0","targetInstanceIps":[]}]
Service account information (all service accounts) v0.1
service-accounts
{"serviceAccounts": [{"scopes":["https://www.googleapis.com/auth/devstorage.read_only"], "serviceAccount":"12345-compute@developer.gserviceaccount.com"}]}
v1
instance/service-accounts/?recursive=true
{"123451-compute@developer.gserviceaccount.com": {"aliases":["default"], "email":"123451-compute@developer.gserviceaccount.com", "scopes":["https://www.googleapis.com/auth/devstorage.read_only"]}, "default":{"aliases":["default"], "email":"123451-compute@developer.gserviceaccount.com", "scopes":["https://www.googleapis.com/auth/devstorage.read_only"]}}
Service account information (single service account) v0.1
service-accounts/<email|default>
{"scopes":["https://www.googleapis.com/auth/devstorage.read_only"], "serviceAccount":"123451-compute@developer.gserviceaccount.com"}
v1
instance/service-accounts/<email|default>/?recursive=true
{"aliases":["default"], "email":"123451-compute@developer.gserviceaccount.com", "scopes":["https://www.googleapis.com/auth/devstorage.read_only"]}
Querying OAuth 2.0 access tokens
The following metadata entries return
OAuth2
access tokens.v0.1
root -http://metadata.google.internal/0.1/meta-data/
v1
root -http://metadata.google.internal/computeMetadata/v1
Metadata entry Version URI (relative to root) Example output OAuth2 Token v0.1
(Method 1)auth-token?service_account=<email|default>&scope=scopeA,scopeB
auth_token?service_account=<email|default>&scope=scopeA,scopeB
{"expires":1568854217, "oauth2_access_token":"ya29.c.KmyIB0i4tH1xLzKGrqeeK6TGWEW3b18Lcq...."}
v0.1
(Method 2)service-accounts/<email|default>/acquire
{"accessToken":"ya29.c.KmyIB0i4tH1xLNzKGrqeeK6TGWEW3b18Lcq.....", "expiresAt":1568854217,"expiresIn":2022}
v1
instance/service-accounts/<email|default>/token
{"access_token":"ya29.c.KmyIB0i4tH1xLNzKGrqeeK6TGWEW3b18Lcq....", "expires_in":2022,"token_type":"Bearer"}
Applications that require an update
To use the v1 metadata server endpoint, update the following applications to the minimum supported version or a higher version.
Application Minimum supported version Facter (developed by Puppet) Cloud Monitoring agent 5.5.2-375 RightScale's server agent (RightLink) RightLink 10 Images to update
If you have guest attributes enabled, the following images make calls to the v1beta1 metadata server to store SSH host keys. We recommend that you upgrade to later versions of the images that are listed.
Operating system Image versions CentOS - centos-6-v20190619
- centos-7-v20190619
Debian debian-9-stretch-v20190618 Red Hat Enterprise Linux (RHEL) - rhel-6-v20190618
- rhel-7-v20190618
- rhel-8-v20190521
- rhel-8-v20190522
- rhel-8-v20190618
- rhel-7-4-sap-v20190618
- rhel-7-6-sap-v20190618
SUSE Linux Enterprise Server (SLES) - sles-15-sp1-v20190625
- sles-15-sp1-sap-v20190625
Supported library versions
Some Google libraries are unaffected by the migration to the v1 metadata server endpoint. However, if you are using any of the libraries listed in the following table, make sure that you upgrade to the minimum supported version or a higher version.
Language Google library Minimum supported version Java com.google.api-client 1.18.0-rc Node.js dialogflow 0.6.0 Node.js firebase-admin 8.2.0 Node.js google-cloud/debug-agent 3.0.0 Node.js google-cloud/profiler 0.2.0 Node.js google-cloud/trace-agent 2.11.0 Node.js gcp-metadata 0.5.0 Node.js gcs-resumable-upload 0.13.0 Node.js googleapis 27.0.0 Node.js google-auth-library 1.3.0 Node.js google-cloud/bigquery 2.0.0 Node.js google-cloud/bigquery-data-transfer 0.4.0 Node.js google-cloud/bigtable 0.15.0 Node.js google-cloud/cloud-container 0.3.0 Node.js google-cloud/compute 0.11.0 Node.js google-cloud/datastore 2.0.0 Node.js google-cloud/dlp 0.8.0 Node.js google-cloud/dns 0.8.0 Node.js google-cloud/error-reporting 0.5.0 Node.js google-cloud/firestore 0.16.1 Node.js google-cloud/language 2.0.0 Node.js google-cloud/logging-bunyan 0.9.0 Node.js google-cloud/logging 2.0.0 Node.js google-cloud/logging-winston 0.10.0 Node.js google-cloud/monitoring 0.6.0 Node.js google-cloud/os-login 0.3.0 Node.js google-cloud/pubsub 0.20.0 Node.js google-cloud/redis 0.2.0 Node.js google-cloud/resource 0.9.0 Node.js google-cloud/spanner 2.0.0 Node.js google-cloud/speech 2.0.0 Node.js google-cloud/storage 2.0.0 Node.js google-cloud/tasks 0.2.0 Node.js google-cloud/text-to-speech 0.3.0 Node.js google-cloud/translate 2.0.0 Node.js google-cloud/vision 0.21.0 Node.js google-cloud/profiler 0.2.0 Node.js google-cloud/trace-agent 2.11.0 Node.js google-gax 0.17.0 Node.js gce-images 1.0.0 Node.js gcp-metadata 0.5.0 Node.js gcs-resumable-upload 0.13.0 Node.js googleapis 27.0.0 Node.js google-auth-library 1.3.0 Node.js google-cloud/bigquery 2.0.0 Node.js google-cloud/bigquery-data-transfer 0.4.0 Node.js google-cloud/bigtable 0.15.0 Node.js google-cloud/cloud-container 0.3.0 Node.js google-cloud/common 0.18.0 Node.js google-cloud/common-grpc 0.7.0 Node.js google-cloud/compute 0.11.0 Node.js google-cloud/datastore 2.0.0 Node.js google-cloud/dlp 0.8.0 Node.js google-cloud/dns 0.8.0 Node.js google-cloud/error-reporting 0.5.0 Node.js google-cloud/firestore 0.16.1 Node.js google-cloud/language 2.0.0 Node.js google-cloud/logging 2.0.0 Node.js google-cloud/logging-bunyan 0.9.0 Node.js google-cloud/logging-winston 0.10.0 Node.js google-cloud/monitoring 0.6.0 Node.js google-cloud/os-login 0.3.0 Node.js googleapis/nodejs-pubsub 0.20.0 Node.js google-cloud/redis 0.2.0 Node.js google-cloud/resource 0.9.0 Node.js google-cloud/spanner 2.0.0 Node.js google-cloud/speech 2.0.0 Node.js google-cloud/storage 2.0.0 Node.js google-cloud/tasks 0.2.0 Node.js google-cloud/text-to-speech 0.3.0 Node.js google-cloud/translate 2.0.0 Node.js google-cloud/video-intelligence 1.3.0 Node.js google-cloud/vision 0.21.0 PHP google-cloud/video-intelligence 1.3.3 Python oauth2client 2.0.0 Python google-api-python-client 1.6.0 Python googleapis/google-cloud-python 0.10.0 Python google-cloud-happybase 0.20.0 Ruby gcloud 0.11.1 Ruby google-api-client 0.8.6 - The root url: