Interacting with the Serial Console

This page describes how to enable interactive access to an instance's serial console to debug boot and networking issues, troubleshoot malfunctioning instances, interact with the GRand Unified Bootloader (GRUB), and perform other troubleshooting tasks.

A virtual machine instance has four virtual serial ports. Interacting with a serial port is similar to using a terminal window, in that input and output is entirely in text mode and there is no graphical interface or mouse support. The instance's operating system, BIOS, and other system-level entities often write output to the serial ports, and can accept input such as commands or answers to prompts. Typically, these system-level entities use the first serial port (port 1) and serial port 1 is often referred to as the serial console.

By default, you can call the getSerialPortOutput method to read information that your instance has written to its serial ports, but you cannot write information for your instance to read. However, if you run into problems accessing your instance through SSH or need to troubleshoot an instance that is not fully booted, you can enable interactive access to the serial console, which lets you connect to and interact with any of your instance's serial ports. For example, you can directly run commands and respond to prompts in the serial port.

Before you begin

Enabling interactive access on the serial console

To enable interactive access to an instance's serial console, you must apply a special metadata key/value pair. You can apply this metadata on specific instances or apply the metadata to the project, which enables interactive access to the serial console of all instances in that project.

Apply this special metadata to the project or instance to enable interactive access:

serial-port-enable=1

For example, using gcloud, you can apply this metadata to a specific instance like so:

gcloud compute instances add-metadata [INSTANCE_NAME]  \
    --metadata=serial-port-enable=1

To apply the metadata to the project:

gcloud compute project-info add-metadata --metadata=serial-port-enable=1

If you apply the metadata to the project, you can still disable it for a specific instance by setting serial-port-enable=0 in the metadata of the particular instance. This will override the project metadata.

Connecting to a serial console

After enabling interactive access for an instance's serial console, you can connect to the serial console using the Google Cloud Platform Console, gcloud, or a third-party SSH client.

The serial console authenticates users with SSH keys. Specifically, you must add your public SSH key to the project or instance metadata, and store your private key on the local machine from which you want to connect. gcloud and the Google Cloud Platform Console will automatically add SSH keys to the project for you. If you are using a third-party client, you might need to add SSH keys manually.

Console

  1. Go to the VM instances page.

    Go to the VM instances page

  2. Click the instance you want to connect to.
  3. Scroll to the bottom of the page and look for the Serial port section.
  4. If you want to connect to a serial port other than the default serial port 1, click the down arrow next to the Connect to serial port button and change the port number accordingly.
  5. Click the Connect to serial port button to connect to port 1 by default. For Windows instances, pull down the dropdown menu next to the button and connect to Port 2 to access the serial console.

gcloud

Use the gcloud compute connect-to-serial-port subcommand to connect using gcloud. For example:

gcloud compute connect-to-serial-port [INSTANCE_NAME]

where [INSTANCE_NAME] is the name of the instance for which you want to access the serial console.

By default, the connect-to-serial-port command connects to port 1 of the serial console. If you are connecting to a Windows VM instance, connect to port 2 instead:

gcloud compute connect-to-serial-port [INSTANCE_NAME] --port 2

To connect to any other port, provide a different port number using the --port flag. You can provide a port number from 1 through 4, inclusively. To learn more about port numbers, see Understanding serial port numbering.

Other SSH clients

You can connect to an instance's serial console using other third-party SSH clients, as long as the client allows you to connect to TCP port 9600.

For example, the following SSH command connects to the default serial port (1) of an instance named example-instance with the username jane in a project with the project ID myproject. The instance is in zone us-central1-f:

ssh -i [PRIVATE_SSH_KEY_FILE] -p 9600 \
myproject.us-central1-f.example-instance.jane@ssh-serialport.googleapis.com

In detail, you can connect to the serial console of an instance using the following login and address information:

[PROJECT_ID].[ZONE].[INSTANCE_NAME].[USERNAME].[OPTIONS]@ssh-serialport.googleapis.com

where:

  • [PROJECT_ID] is the project ID for this instance.
  • [ZONE] is zone of the instance.
  • [INSTANCE_NAME] is the name of the instance.
  • [USERNAME] is the username you are using to connect to your instance. Typically, this is the username on your local machine.
  • [OPTIONS] are additional options you can specify for this connection. For example, you can specify a certain serial port and specify any of the advanced options below. The port number can be 1 through 4, inclusively. To learn more about port numbers, see Understanding serial port numbering. If omitted, you will connect to serial port 1.

If you are connecting to a Windows VM instance, connect through port 2 using the following command:

ssh -i [PRIVATE_SSH_KEY_FILE] -p 9600 \
[PROJECT_ID].[ZONE].[INSTANCE_NAME].[USERNAME].port=2@ssh-serialport.googleapis.com

If you are having trouble connecting using a third-party SSH client, you can run gcloud compute connect-to-serial-port with the --dry-run command-line option to see the SSH command that it would have run on your behalf, and compare the options with the command you are using.

Setting up a secure connection

When you use a third-party SSH client that is not gcloud, you can ensure that you're protected against impersonation or man-in-the-middle attacks by checking Google's Serial Port server SSH key. Follow these instructions to set up your system to check the server SSH key:

  1. Download Google's Serial Port server SSH key.
  2. Open your known hosts file, generally located at ~/.ssh/known_hosts.
  3. Add the contents of the server SSH key, with ssh-serialport.googleapis.com prepended to the key. For example, if the server key contains the line ssh-rsa AAAAB3NzaC1yc..., then ~/.ssh/known_hosts should have a line:

    ssh-serialport.googleapis.com ssh-rsa AAAAB3NzaC1yc...
    

For security reasons, Google may occasionally change the Google Serial Port server SSH key. If your client fails to authenticate the server key, immediately abort the connection attempt and follow the directions above to download a new Google Serial Port server SSH key.

If, after updating the host key, you continue to receive a host authentication error from your client, stop attempts to connect to the serial port and contact Google support. Do not provide any credentials over a connection where host authentication has failed.

Disconnecting from the serial console

To disconnect from the serial console:

  1. Press the ENTER key.
  2. Type ~. (tilde, followed by a period).

You can discover other commands by typing ~? or by examining the man page for SSH:

man ssh

Do not try to disconnect using any of the following methods:

  • The CTRL+ALT+DELETE key combination or other similar combinations. This will not work because the serial console does not recognize PC keyboard combinations.

  • The exit or logout command does not work because the guest is not aware of any network or modem connections. Using this command causes the console to close and then reopen again, and you remain connected to the session. If you would like to enable exit and logout commands for your session, you can enable it by setting the on-dtr-low option.

Connecting to a serial console with a login prompt

If you are trying to troubleshoot an issue with an instance that has booted completely, or trying to troubleshoot an issue that occurs after the instance has booted past single user mode, you might be prompted for login information when trying to acccess the serial console.

By default, Google-supplied system images are not configured to allow password-based logins for local users. If your instance is running an image that is preconfigured with serial port logins, you need to set up a local password on the virtual machine instance so you can login to the serial console, if prompted.

Setting up a local password

The following instructions describe how to set up a local password for a user on a virtual machine instance so that the user can log on to the serial console of that instance using the specified password.

  1. Connect to the instance:

    gcloud compute ssh [INSTANCE_NAME]
    
  2. On the instance, create a local password with the following command. This sets a password for the user that you are currently logged in as.

    sudo passwd `whoami`
    
  3. Follow the prompts to create a password.

  4. Next, log out of the instance and connect to the serial console.
  5. Enter in your login information when prompted.

Setting up a login on other serial ports

Login prompts are enabled on port 1 by default on most Linux operating systems. However, port 1 can often be overwhelmed by logging data and other information being printed to the port. Alternatively, you can choose to enable a login prompt on another port instead, such as port 2 (ttyS1), by executing one of the following commands on your instance. You can see a list of available ports for an instance in Understanding serial port numbering.

The following table lists images preconfigured with a serial console login and the default ports.

Operating System Port(s) with a login prompt by default Service Management
CentOS 6 1 upstart
CentOS 7 1 systemd
CoreOS 1 systemd
Debian 7 None sysvinit
Debian 8 1 systemd
OpenSUSE 13 1 systemd
OpenSUSE Leap 1 systemd
RHEL 6 1 upstart
RHEL 7 1 systemd
SLES 11 1 sysvinit
SLES 12 1 systemd
Ubuntu 12.04 none upstart
Ubuntu 14.04 1 upstart
Ubuntu 15.10 1 systemd
Ubuntu 16.04 1 systemd
Windows COM2 N/A

To enable login prompts on additional serial ports, use the following instructions.

systemd

For Linux operating systems using systemd:

  • Enable the service temporarily till next reboot:

    sudo systemctl start serial-getty@ttyS1.service
    
  • Enable the service permanently, starting with the next reboot:

    sudo systemctl enable serial-getty@ttyS1.service
    

upstart

For Linux operating systems using upstart:

  1. Create a new /etc/init/ttyS1.conf file by copying and modifying an existing ttyS0.conf file to reflect ttyS1. For example:

    • On Ubuntu 14.04:

      sudo sh -c "sed -e s/ttyS0/ttyS1/g < /etc/init/ttyS0.conf > /etc/init/ttyS1.conf"
      
    • On RHEL 6.8 and CentOS 6.8

      sudo sh -c "sed -ne '/^# # ttyS0/,/^# exec/p'  < /etc/init/serial.conf  | sed -e 's/ttyS0/ttyS1/g' -e 's/^# *//' > /etc/init/ttyS1.conf"
      
  2. Start on a login prompt on ttyS1 without restarting:

    sudo start ttyS1
    

sysvinit

For Linux operating systems using sysvinit, run the following commands:

 sudo sed -i~ -e 's/^#T\([01]\)/T\1/' /etc/inittab
 sudo telinit q

Understanding serial port numbering

Each virtual machine instance has four serial ports. For consistency with the getSerialPortOutput API, each port is numbered 1 through 4. Linux and other similar systems number their serial ports 0 through 3. For example, on many operating system images, the corresponding devices are /dev/ttyS0 through /dev/ttyS3. Windows refers to serial ports as COM1 through COM4. To connect to what Windows considers COM3 and Linux considers ttyS2, you would specify port 3. Use the table below to help you figure out which port you want to connect to.

Virtual Machine Instance Serial Ports Standard Linux Serial Ports Windows COM Ports
1 /dev/ttyS0 COM1
2 /dev/ttyS1 COM2
3 /dev/ttyS2 COM3
4 /dev/ttyS3 COM4

Note that many Linux images use port 1 (/dev/ttyS0) for logging messages from the kernel and system programs.

Sending a serial break

The Magic SysRq key feature allows you to perform low-level tasks regardless of the system's state. For example, you can sync filestems, reboot the instance, kill processes, unmount filesystems and so on, using the Magic SysRq key feature.

To send a Magic SysRq command using a simulated serial break:

  1. Press the ENTER key.
  2. Type ~B (tilde, followed by uppercase B).
  3. Type the desired Magic SysRq command.

Viewing serial console logs

Compute Engine provides audit logs to track who has connected and disconnected from an instance's serial console. To view logs, you must have permissions for the Logs Viewer or be a project viewer or editor.

  1. Go to the Logs page in the Cloud Platform Console.

    Go to the Logs page

  2. Expand the drop-down menu and select GCE VM Instance.
  3. In the search bar, type ssh-serialport.googleapis.com and hit Enter.
  4. A list of audit logs describing connection and disconnections from a serial console appears. Expand any of the entries to get more information:

    Screenshot of audit logs for the serial console

For any of the audit logs, you can:

  1. Expand the protoPayload property.
  2. Look for methodName to see activity this log applies to (either a connection or disconnection request). For example, if this log tracks a disconnection from the serial console, the method name would say "google.ssh-serialport.v1.disconnect". Similarly, a connection log would say "google.ssh-serialport.v1.connect". An audit log entry is recorded at the beginning and end of each session on the serial console.

There are different audit log properties for different log types. For example, audit logs relating to connections will have some properties that are specific to connection logs, while audit logs for disconnections will have their own set of properties. There are certain audit log properties that are also shared between both log types.

All serial console logs

Property Value
requestMetadata.callerIp The IP address and port number from which the connection originated.
serviceName ssh-serialport.googleapis.com
resourceName A string containing the project ID, zone, instance name, and serial port number to indicate which serial console this pertains to. For example, projects/myproject/zones/us-east1-a/instances/example-instance/SerialPort/2 is port number 2, also known as COM2 or /dev/ttyS1, for the instance example-instance.
resource.labels Properties identifying the instance ID, zone, and project ID.
timestamp A timestamp indicating when the session began or ended.
severity NOTICE
operation.id A ID string uniquely identifying the session; you can use this to associate a Disconnect entry with the corresponding Connection entry.
operation.producer ssh-serialport.googleapis.com

Connection logs

Property Value
methodName google.ssh-serialport.v1.connect
status.message Connection succeeded.
request.serialConsoleOptions Any options that were specified with the request, including the serial port number.
request.@type type.googleapis.com/google.compute.SerialConsoleSessionBegin
request.username The username specified for this request. This is used to select the public key to match.
operation.first true
status.code For successful connection requests, a status.code value of google.rpc.Code.OK indicates that the operation completed successfully without any errors. Since the enum value for this property is 0, the status.code property will not be displayed in this case. However, any code that checks for a status.code value of google.rpc.Code.OK will work as expected.

Disconnection logs

Property Value
methodName google.ssh-serialport.v1.disconnect
response.duration The amount of time the session lasted, in seconds.
response.@type type.googleapis.com/google.compute.SerialConsoleSessionEnd
operation.last true

Failed connection logs

When a connection fails, Compute Engine creates an audit log entry. A failed connection log looks very similar to a successful connection entry, but has the following properties to indicate a failed connection.

Property Value
severity ERROR
status.code

The canonical Google API error code that best describes the error. The following are possible error codes that might appear:

status.message The human-readable message for this entry.

Tips and tricks

  • If you are having trouble connecting using a standard SSH client, but gcloud compute connect-to-serial-port connects successfully, it might be helpful to run gcloud compute connect-to-serial-port with the --dry-run command-line option to see the SSH command that it would have run on your behalf, and compare the options with the command you are using.

  • Setting the bit rate, also known as baud rate: You can set any bit rate you like, such as stty 9600, but the feature normally forces the effective rate to 115200 bps (~11.5kB/sec). This is because many OS images default to slow bit rates such as 9600 on the serial console, and would boot slowly.

  • Some OS images have inconvenient defaults on the serial port. For instance, on CentOS 7 stty icrnl is required to tell the console to do the right thing with the Enter key (which sends a CR, aka ^M). The bash shell might mask this until you try to set a password, and then wonder why it seems stuck at the password: prompt.

  • Some OS images have job control keys that are disabled by default if you attach a shell to a port in certain ways. Some examples of these keys include ^Z and ^C. The setsid command may fix this. Otherwise, if you see a job control is disabled in this shell message, be careful not to run commands that you will need to interrupt.

  • You might find it helpful to tell the system the size of the window you’re using, so that bash and editors can manage it properly. Otherwise, you might experience odd display behavior as bash or editors attempt to manipulate the display based on incorrect assumptions about the number of rows and columns available. Use the stty rows Y cols X command and stty -a to see what the setting is. For example: stty rows 60 cols 120 (if your window is 120 chars by 60 lines).

  • If you connect using SSH from machine A to machine B, and then to machine C (and so on), creating a nested SSH session, and you want to use ~ commands such as to disconnect, or send a serial break signal, you will need to add enough extra ~ characters to the command to get to the right SSH client. A command following a single ~ will be interpreted by the SSH client on machine A; two consecutive ~ (ENTER~~) will be interpreted by the client on machine B, and so forth. You only need to press ENTER once, because that is passed all the way through to the innermost SSH destination. This is true for any use of SSH clients which provide the ~ escape feature.

    If you lose track of how many ~ characters you need, press the ENTER key and then type ~ characters one at a time until the instance echoes the ~ back. This indicates that you have reached the end of the chain and you now know that to send a ~ command to the most nested SSH client, you need one less ~ than the number you typed.

Advanced options

Controlling max connections

You can set the max-connections property to control how many concurrent connections can be made to this serial port at a time. The default and maximum number of connections is 5. For example:

gcloud compute connect-to-serial-port [INSTANCE_NAME] --port [PORT_NUMBER] --extra-args max-connections=3
ssh -i [PRIVATE_SSH_KEY_FILE] -p 9600 [PROJECT_ID].[ZONE].[INSTANCE_NAME].[USERNAME].max-connections=3@ssh-serialport.googleapis.com

Setting replay options

By default, each time you connect to the serial console, you will receive a replay of the last 10 lines of data, regardless of whether the last 10 lines have been seen by another SSH client. You can change this setting and control how many and which lines are returned by providing the following options:

  • replay-lines=N: Set N to the number of lines you want replayed. For example, if N was 50, then the last 50 lines of the console output is included.
  • replay-bytes=N: Replays the most recent N bytes. You can also set N to new which replays all output that has not yet been sent to any client.
  • replay-from=N: Replays output starting from an absolute byte index that you provide. You can get the current byte index of serial console output by making a getSerialPortOutput request. If you set replay-from, all other replay options are ignored.

On gcloud, append the following to your connect-to-serial-port command, where N is the specified number of lines (or bytes or absolute byte index, depending on which replay option you are selecting):

--extra-args replay-lines=N

If you are using a third-party SSH client, provide this option in your SSH command:

ssh -i [PRIVATE_SSH_KEY_FILE] -p 9600 \
  myproject.us-central1-f.example-instance.jane.port=3.replay-lines=N@ssh-serialport.googleapis.com

You can also use a combination of these options as well. For example:

replay-lines=N and replay-bytes=new

Replay the specified number of lines OR replay all output not previously sent to any client, whichever is larger. The first client to connect with this flag combination will see all the output that has been sent to the serial port, and clients that connect subsquently will only see the last N lines. Examples:

gcloud compute connect-to-serial-port [INSTANCE_NAME] --port [PORT_NUMBER] --extra-args replay-lines=N,replay-bytes=new
ssh -i [PRIVATE_SSH_KEY_FILE] -p 9600 [PROJECT_ID].[ZONE].[INSTANCE_NAME].[USERNAME].replay-lines=N.replay-bytes=new@ssh-serialport.googleapis.com
replay-lines=N and replay-bytes=M

Replay lines up to, but not more than, the number of lines or bytes described by these flags, whichever is less. This option will not replay more than N or M bytes.

gcloud compute connect-to-serial-port [INSTANCE_NAME] --port [PORT_NUMBER] --extra-args replay-lines=N,replay-bytes=M
ssh -i [PRIVATE_SSH_KEY_FILE] -p 9600 [PROJECT_ID].[ZONE].[INSTANCE_NAME].[USERNAME].replay-lines=N.replay-bytes=M@ssh-serialport.googleapis.com

Handling dropped output

The most recent 1 MiB of output for each serial port is always available and generally, your SSH client should not miss any output from the serial port. If, for some reason, your SSH client stops accepting output for a period of time but does not disconnect, and more than 1 MiB of new data is produced, your SSH client might miss some output. In these scenarios, when your SSH client is not accepting data fast enough to keep up with the output on the serial console port, you can set the on-dropped-output property to determine how the console behaves.

Set any of the following applicable options with this property:

  • insert-stderr-note: Insert a note on the SSH client's stderr indicating that output was dropped. This is the default option.
  • ignore: Silently drops output and does nothing.
  • disconnect: Terminate the connnection.

For example:

gcloud compute connect-to-serial-port [INSTANCE_NAME] --port [PORT_NUMBER] --extra-args on-dropped-output=ignore
ssh -i [PRIVATE_SSH_KEY_FILE] -p 9600 [PROJECT_ID].[ZONE].[INSTANCE_NAME].[USERNAME].on-dropped-output=ignore@ssh-serialport.googleapis.com

Enabling disconnect using exit or logout commands

You can enable disconnecting on exit or logout commands by setting the on-dtr-low property to disconnect when you connect to the serial console.

On gcloud, append the following to your connect-to-serial-port command:

--extra-args on-dtr-low=disconnect

If you are using a third-party SSH client, provide this option in your SSH command:

ssh -i [PRIVATE_SSH_KEY_FILE] -p 9600 \
  myproject.us-central1-f.example-instance.jane.port=3.on-dtr-low=disconnect@ssh-serialport.googleapis.com

Enabling this option might cause your instance to disconnect one or more times when you are rebooting the instance, as the operating system resets the serial ports while booting up.

The default setting for this option is none, where nothing happens when the DTR line changes. If you change this to none, you can reboot your instance without being disconnected from the serial console but the console will not disconnect through normal means such as exit or logout commands, or normal key combinations like Ctrl+d.

What's next

Send feedback about...

Compute Engine Documentation