Many software systems that depend on careful sequencing of events rely on a stable, consistent system clock. System logs written by most services include a timestamp, which helps debug issues that occur between various components of your system. To help keep system clocks in sync, Compute Engine instances are preconfigured to use network time protocol (NTP).
In addition to keeping server time in sync, NTP is helpful in the rare case of a leap second. A leap second is a one-second adjustment made to UTC time to account for changes in the Earth's rotation. Leap seconds don't happen at routine intervals, because the Earth's rotation speed varies irregularly in response to climatic and geological events. Previous leap seconds have noticeably impacted a variety of services and applications on the web. NTP servers help ensure that all servers report the same time during the event of a leap second.
This document describes how to configure NTP servers on your virtual machines (VMs) to behave properly in the case of a leap second.
Google NTP servers and leap smearing
Leap seconds for a Unix operating system are commonly implemented by repeating the last second of the day. This can cause problems with software that expects timestamps to only ever increase. To get around this problem, the time servers at Google Cloud "smear" the extra second over twenty-four hours—twelve before and twelve after the leap second event—so that computers don't see the extra second all at once as a repeated timestamp. This reduces risk in systems that depend on a consistent timestamp. It's recommended that all Compute Engine virtual machine (VM) instances are configured to use the internal Google NTP services.
Configure NTP for your instances
Google Cloud can't predict how external NTP services, such as pool.ntp.org
,
will handle the leap second. If at all possible, it's recommended that you don't
use external NTP sources with Compute Engine VMs. Even
worse, using both Google's NTP service and an external service can result in
unpredictable changes in the system time. Using only a single external NTP
source is preferable to using a mix, but external NTP services, such as
pool.ntp.org
, will likely use stepping to handle the leap second. As a result,
your VMs might see a repeated timestamp.
The safest approach is to configure your Compute Engine VMs to use a single NTP server—the internal NTP server provided by Google. Don't mix external NTP servers and Google NTP servers, as this could result in unexpected behavior. For more information about the internal NTP server provided by Google, see the Google Public NTP FAQ.
To ensure your VMs are correctly configured, follow these instructions.
Linux (chrony)
By default, the majority of new Linux releases use
chrony
to manage their
NTP settings and time synchronization. To make sure that
chrony
uses only the internal NTP service, check the chrony
configuration and remove external NTP servers.
Use
ssh
to connect to your instance.Console
To use the console to connect to the VM using SSH, follow these steps:
Go to the VM instances page in the Google Cloud console.
Click the SSH button for the VM that you want to configure.
gcloud
To use the Google Cloud CLI to connect to the VM using SSH, run the following command:
gcloud compute instances ssh VM_NAME
Replace
VM_NAME
with the name of the VM that you're connecting to.On your instance, run
chronyc sources
to check the current state of your NTP configuration:$ chronyc sources
The output looks similar to the following:
210 Number of sources = 2 MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* metadata.google.internal 2 6 377 4 -14us[ -28us] +/- 257us ^- 38.229.53.9 2 6 37 4 -283us[ -297us] +/- 28ms
If you see a single record pointing at
metadata.google
ormetadata.google.internal
, you don't need to make any changes. If you see multiple sources, mixed betweenmetadata.google
and a public source such aspool.ntp.org
, update your sources to remove any external NTP servers.In the example output, there are two records, one pointing at
metadata.google.internal
and another pointing to an external address. Because there are multiple sources, you would update your NTP servers to remove the38.229.53.9
address, as described in the next step.Configure your NTP servers to remove external NTP servers.
To remove the additional NTP server from the list, edit the
/etc/chrony/chrony.conf
file using your favorite text editor. Find all the lines that start withserver external_source_ip_or_name
and remove them.After editing your
/etc/chrony/chrony.conf
file, restart thechrony
service. The command to restart might vary depending on the Linux distribution, as shown in the following examples:sudo service chrony restart
sudo systemctl restart chrony
Verify your configuration by running the
chronyc sources
command again:$ chronyc sources
The output should look similar to the following:
210 Number of sources = 1 MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* metadata.google.internal 2 7 377 98 -1343ns[-1588ns] +/- 396us
Linux (ntpd)
Most older Linux distributions use
ntpd
to manage their NTP settings
and time synchronization. To make sure that ntpd
uses only the internal
NTP service, check the ntpd
configuration and remove external NTP
servers.
Use
ssh
to connect to your instance.Console
To use the console to connect to the VM using SSH, follow these steps:
Go to the VM instances page in the Google Cloud console.
Click the SSH button for the VM that you want to configure.
gcloud
To use the Google Cloud CLI to connect to the VM using SSH, run the following command:
gcloud compute instances ssh VM_NAME
Replace
VM_NAME
with the name of the VM that you're connecting to.On your instance, run
ntpq -p
to check the current state of your NTP configuration:$ ntpq -p
The output looks similar to following:
remote refid st t when poll reach delay offset jitter ============================================================================== *metadata.google 255.28.23.83 2 u 27 64 1 0.634 -2.537 2.285 *217.162.232.173 130.149.17.8 2 u 191 1024 176 79.245 3.589 27.454
If you see a single record pointing at
metadata.google
ormetadata.google.internal
, you don't need to make any changes. If you see multiple sources, mixed betweenmetadata.google
and a public source such aspool.ntp.org
, you need to update your sources to remove any external NTP servers.In the example output, there are two records, one pointing at
metadata.google
and another pointing to an external address. Because there are multiple sources, you would need to update your NTP servers to remove the*217.162.232.173
address, as described in the next step.Configure your NTP servers to remove external sources.
To configure your NTP servers, edit the
/etc/ntp.conf
file using your favorite text editor. Find theservers
section of the configuration, and remove all non-Google NTP sources, for example:vim /etc/ntp.conf
# You do need to talk to an NTP server or two (or three). #server ntp.your-provider.example ... server metadata.google.internal iburst
After editing your
/etc/ntp.conf
file, restart the NTP service. The command to restart might vary based on the Linux distribution:sudo service ntp reload
Verify your configuration by running the
ntpq -p
command again:ntpq -p
remote refid st t when poll reach delay offset jitter ============================================================================== *metadata.google 255.28.23.83 2 u 27 64 1 0.634 -2.537 2.285
Windows
Go to the VM instances page in the Google Cloud console.
Click the RDP button next to the Windows instance you want to connect to.
After you are logged in, right-click on the PowerShell icon and select Run as administrator.
When the command-prompt loads, run the following command to see the current NTP configuration:
w32tm /query /configuration
[Configuration] ... Type: NTP (Local) NtpServer: metadata.google.internal, ...
If you see a single record pointing at
metadata.google
ormetadata.google.internal
, you don't need to make any changes. If you see multiple sources, mixed betweenmetadata.google
and a public source, you need to remove the external server. Follow the Windows' guide for configuring your NTP server.To ensure the most widespread software compatibility on Windows VMs, Google recommends that you use the gVNIC driver to ensure sub-millisecond NTP accuracy with
metadata.google
.If you must use VirtIO with your Windows VM, to get sub-millisecond accuracy with NTP servers, Google recommends that you don't use the Windows Time Service (stop and unregister
w32tm
).Stop the Windows Time Service:
net stop w32time
Remove the Windows Time Service from the registry:
w32tm /unregister
After the Windows Time Service has been stopped and removed from the registry, install the Meinberg NTP client.
Follow the configuration instructions as provided in the Meinberg documentation.
Configure the NTP server for the Meinberg NTP client as
metadata.google.internal
.After you finish configuring NTP, wait between 5 and 15 minutes for the system clock in the VM to stabilize with the NTP server.
For information about why using w32tm is not recommended, see the Known Issues documentation.
Using leap smearing with systems external to Google Cloud
The leap smearing feature of Google's NTP servers is a convenient way to manage the risk involved with replaying a second on time sensitive systems. Other NTP services may provide an acceptable work around for most software systems. However, it's important that you don't mix Google leap smearing NTP services with public NTP stepping services.
To synchronize devices outside Google Cloud to smeared time, you can use Google Public NTP for those devices. The Google Public NTP uses the same leap second smearing that's provided to Compute Engine VMs.
What's next
- Learn more about PCI Data Security Standard compliance.
- Review the Google Public NTP FAQ.