Troubleshooting SSH

Under certain conditions, a Compute Engine instance no longer accepts SSH connections. There are many reasons this could happen. Some common causes of SSH connection issues are as follows:

  • OS Login is enabled on the instance. You cannot use both metadata-based SSH keys and OS Login to connect to an instance. If OS Login is enabled, then connecting with metadata-based SSH keys is disabled because OS Login does not store SSH keys in authorized keys files.
  • OS Login is not enabled. When OS Login is not enabled, Google manages the authorized keys file for new user accounts based on SSH keys in metadata. A Compute Engine instance no longer accepts SSH connections using the SSH keys configured as part of your user account.
    • The accounts daemon stores a file in the guest to preserve state for the user accounts managed by Google.
    • The authorized keys file for a Google-managed user account is deleted when all SSH keys for the user account are removed from metadata.
    • User accounts that are not managed by Google are not modified by the accounts daemon.
  • The instance has a full disk. Check your disk space and clean it up as needed.
  • The sshd daemon is not configured properly. Review the user guide for your operating system to ensure that your ssh_d file is set up correctly.

This topic describes a number of tips and approaches to help troubleshoot and resolve some of the most common SSH issues.

Requirements

You can run most of your troubleshooting steps from your local workstation. To use a local Linux or Windows workstation to troubleshoot a VM instance, you must first prepare the workstation.

Prepare your workstation with the following steps:

  • Install or update to the latest version of the gcloud command-line tool.
  • Install the nmap network discovery and security auditing tool for your operating system. You will use this tool to test the network connection to your VM instance.
  • Set environment variables.

Set environment variables

You can set environment variables for any parameters that might be frequently used in this troubleshoot guide, such as the instance name and the name of the persistent boot disk for the affected instance.

Set environment variables on your local workstation.

Linux or macOS

On a Linux or macOS workstation, use the export command.

export PROB_INSTANCE='instance-name'
export BOOT_DISK='boot-disk-name'

Replace the following:

  • instance-name: The name of the instance that you are troubleshooting.
  • boot-disk-name: The name of the persistent boot disk for the instance that you are troubleshooting.

For example, if your instance is named instance1 and your boot disk is named disk1, run the following commands:

export PROB_INSTANCE='instance1'
export BOOT_DISK='disk1'

Windows

On the Windows OS, use the set command.

set PROB_INSTANCE='instance-name'
set BOOT_DISK='boot-disk-name'

Replace the following:

  • instance-name: The name of the instance that you are troubleshooting.
  • boot-disk-name: The name of the persistent boot disk for the instance that you are troubleshooting.

For example, if your instance is named instance1 and your boot disk is named disk1, run the following commands:

set PROB_INSTANCE='instance1'
set BOOT_DISK='disk1'

Test connectivity

You might not be able to SSH to a VM instance because of connectivity issues linked to firewalls, network connection, or the user account. Follow the steps in this section to identify any connectivity issues.

Check your firewall rules

Compute Engine provisions each project with a default set of firewall rules that permit SSH traffic. If the default firewall rule that permits SSH connections is somehow removed, you'll be unable to access your instance. Use the gcloud compute command-line tool to check your list of firewalls and ensure the default-allow-ssh rule is present.

On your local workstation, run the following command:

gcloud compute firewall-rules list

If the firewall rule is missing, add it back:

gcloud compute firewall-rules create default-allow-ssh
    --allow tcp:22

Test the network connection

You can use the nmap tool to connect to your instance on port 22, and see if the network connection is working. If you connect and see 22/tcp open ssh, your network connection is working, and you can rule out firewall problems.

  1. Use the gcloud tool to obtain the external natIP for your instance:

    gcloud compute instances describe $PROB_INSTANCE
        --format='get(networkInterfaces[0].accessConfigs[0].natIP)'
    198.51.100.1
    
  2. Test the network connection to your instance.

    Run the nmap command to test the network connection to your instance, replacing external-ip with the external IP of the instance:

    nmap external-ip
    

    For example, if the instance has the external IP 198.51.100.1, run the following command:

    nmap 198.51.100.1
    Starting Nmap 7.70 ( https://nmap.org ) at 2019-03-18 16:04 Greenwich Standard Time
    Nmap scan report for 229.30.196.35.bc.googleusercontent.com (198.51.100.1)
    Host is up (0.0061s latency).
    Not shown: 998 filtered ports
    PORT     STATE  SERVICE
    22/tcp   open   ssh
    Nmap done: 1 IP address (1 host up) scanned in 6.22 seconds
    

Connect as a different user

The issue that prevents you from logging in might be limited to your user account. For example, the permissions on the ~/.ssh/authorized_keys file on the instance might not be set correctly for the user.

Try logging in as a different user with the gcloud tool by specifying another-username with the SSH request. The gcloud tool will update the project's metadata to add the new user and allow SSH access.

gcloud compute ssh another-username@$PROB_INSTANCE

Debug the issue in the serial console

We recommend that you review the logs from the serial console for connection errors. You can access the serial console from your local workstation by using a browser.

Enable read/write access to an instance's serial console, so you can log into the console and troubleshoot problems with the instance. This is particularly useful when you cannot log in with SSH, or if the instance has no connection to the network. The serial console remains accessible in both of these situations.

To learn how to enable interactive access and connect to an instance's serial console, read Interacting with the serial console.

Inspect the VM instance without shutting it down

You might have an instance that you cannot connect to that continues to correctly serve production traffic. In this case, you might want to inspect the disk without interrupting the instance.

To inspect the disk, you need to take a snapshot of the boot disk, create a new disk from that snapshot, create a temporary instance, and then attach and mount the new persistent disk to your temporary instance to troubleshoot the disk.

  1. Create a new VPC network to host your cloned instance:

    gcloud compute networks create debug-network
    
  2. Add a firewall rule to allow SSH connections to the network:

    gcloud compute firewall-rules create debug-network-allow-ssh
       --allow tcp:22
    
  3. Create a snapshot of the boot disk.

    gcloud compute disks snapshot $BOOT_DISK
       --snapshot-name debug-disk-snapshot
    
  4. Create a new disk with the snapshot you just created:

    gcloud compute disks create example-disk-debugging
       --source-snapshot debug-disk-snapshot
    
  5. Create a new debugging instance without an external IP address:

    gcloud compute instances create debugger
       --network debug-network
       --no-address
    
  6. Attach the debugging disk to the instance:

    gcloud compute instances attach-disk debugger
       --disk example-disk-debugging
    
  7. Follow the instructions to connect to an instance without an external IP address.

  8. After you have logged into the debugger instance, troubleshoot the instance. For example, you can look at the instance logs:

    sudo su -
    
    mkdir /mnt/$PROB_INSTANCE
    
    mount /dev/disk/by-id/scsi-0Google_PersistentDisk_example-disk-debugging /mnt/$PROB_INSTANCE
    
    cd /mnt/$PROB_INSTANCE/var/log
    
    # Identify the issue preventing ssh from working
    ls
    

Use a startup script

If none of the preceding helped, you can create a startup script to collect information right after the instance starts. Follow the instructions for running a startup script.

Afterward, you also need to reset your instance before the metadata takes effect by using gcloud compute instances reset.

Alternatively, you can also recreate your instance by running a diagnostic startup script:

  1. Run gcloud compute instances delete with the --keep-disks flag.

    gcloud compute instances delete $PROB_INSTANCE
       --keep-disks boot
    
  2. Add a new instance with the same disk and specify your startup script.

    gcloud compute instances create new-instance
       --disk name=$BOOT_DISK,boot=yes
       --startup-script-url URL
    

As a starting point, you can use the compute-ssh-diagnostic script to collect diagnostics information for most common issues.

Use your disk on a new instance

If the other steps in this document do not work for you, and you need to recover data from your persistent boot disk, you can detach the boot disk and then attach that disk as a secondary disk on a new instance.

gcloud compute instances delete $PROB_INSTANCE
    --keep-disks=boot

gcloud compute instances create new-instance
    --disk name=$BOOT_DISK,boot=yes,auto-delete=no

gcloud compute ssh new-instance
هل كانت هذه الصفحة مفيدة؟ يرجى تقييم أدائنا:

إرسال تعليقات حول...

Compute Engine Documentation