Troubleshoot Linux VM boot issues due to fstab errors

Fstab (file system table) is a configuration file that is used to define how disk partition, file system, and file share mounts in your Linux VM persist across system reboots.

This document outlines multiple conditions where an incorrect fstab configuration may result in a boot problem and offers troubleshooting advice to fix the issue.

A few of the common issues with fstab misconfiguration are listed below:

Syntax error in fstab configuration file
An incorrect UUID
An entry exists for an unattached/unavailable device
Deprecated mount options like nobarrier for the xfs file system type in SUSE or RHEL images.

Before you begin

If you want to log serial port output in Cloud Logging, familiarize yourself with Cloud Logging.
If you haven't already, set up authentication. Authentication is the process by which your identity is verified for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine as follows.

Select the tab for how you plan to use the samples on this page:
Console

When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.
gcloud
1. Install the Google Cloud CLI, then initialize it by running the following command:
```
gcloud init
```
  Note: If you installed the gcloud CLI previously, make sure you have the latest version by running gcloud components update.
2. Set a default region and zone.
REST

To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

Identify fstab issues

For boot issues, Google highly recommendeds that you check the boot logs in the Serial Console of the Linux VM. The VM goes into emergency mode with an error message hinting that the issue is with fstab.

  [K[[1;31m TIME [0m] Timed out waiting for device dev-incorrect.device.
  [[1;33mDEPEND[0m] Dependency failed for /distribution.
  [[1;33mDEPEND[0m] Dependency failed for Local File Systems.
  ...
  Welcome to emergency mode! After logging in, type "journalctl -xb" to viewsystem logs, "systemctl reboot" to reboot, "systemctl default" to try again to boot into default mode.
  Give root password for maintenance
  (or type Control-D to continue)

As per the error message, there is dependency failure for the /distribution file system. The dependency failure for a file system mount point will differ based on the file system names used.

Resolution

Use the following options to solve the fstab error in Linux VMs hosted in Google Cloud. You can resolve the issue faster using the serial console than using the manual method.

To use the following methods to resolve the fstab issue, you must enable serial port access for your VM.

Option 1: Use the serial console to log into the VM in emergency mode

Log into the serial console of the VM from Google Cloud console.
In the current emergency mode, enter the root password to access your VM.
Use your favorite text editor to open the fstab file. Make the necessary changes and save the changes to the fstab file. In the following example, the vi editor is used.
```
vi /etc/fstab
```
In the previous example, the device for /distribution is detached from the VM. Comment the line that references the file system or remove the fstab entry for /distribution mount point.

Use man fstab for more details about the fstab configuration and syntax.
Save the file and exit the editor. If you are using the vi editor, use ESC + :wq! to save and exit the editor.
Resume the boot process by entering reboot at the prompt.
After successfully fixing the issue, the VM boot process should complete and the login prompt appears in the serial console.
Verify you can log in to the VM again using SSH.

Option 2 - Using Single User mode in Serial Console

This option can be used if the root password is not set.

Prerequisite: The GRUB_TIMEOUT parameter in the grub configuration file must be set to a non-zero value. Usually this file is at /etc/default/grub, but on some earlier distributions, it might be located in a non-standard directory.

Go to the VM instances page in the Google Cloud console. Go to VM instances
Log into the serial console of the Virtual Machine from the Google Cloud console.
Click RESET for virtual machine from Google Cloud Console.
In the serial console window, interrupt the boot process by clicking the arrow key to pause in the grub screen or menu.
In the list of kernels in the grub boot loader screen, choose the kernel and press e on the keyboard.
Add the rd.break parameter at the end of the kernel line and press ctrl+x. This helps in booting the VM into single user mode.

Note: The rd.break parameter works for RHEL, CENTOS, Oracle Linux, SUSE Operating Systems. This parameter isn't a persistent entry so no changes are required after boot. For Ubuntu 18.04, 20.04 and 22.04 LTS, you can add systemd.unit=rescue.target to boot into rescue mode which is similar to single user mode.
Mount the root file system in read write mode.
Analyze, make necessary change and save fstab file. Use man fstab for more details about configuration and syntax. Use your favorite text editor to open the fstab file. In the below example, the vi editor is used.
```
vi /etc/fstab
```
In this example, the device for /distribution is detached from the VM, so comment the line or remove the fstab entry for the /distribution mount point.
Save the file and exit from the editor. If you are using the vi editor, then use ESC + :wq! to save and exit from fstab.
Resume the boot process by entering reboot at the prompt.
After successfully fixing the issue, the VM boot process should complete and you should see the login prompt in the serial console.
Verify you can log in to the VM using SSH.