Jupyter Notebooks Cloud Dataproc cluster error

Problem

When you click on Jupyter Notebooks Web interface in Cloud Dataproc cluster, it returns the 504 error in the interfae.

Environment

  • Cloud Dataproc cluster
  • Component gateway enabled for Jupyter Notebooks

Solution

Check that the Notebooks instance disk is not out of space:

  1. Connect to the VM instance using SSH.
  2. Run the following command: df -h -T /home/jupyter
  3. If the Use% is above 85%, you need to manually delete files from /home/jupyter. As a first step, you can empty the trash with the following command: sudo rm -rf /home/jupyter/.local/share/Trash/*

Additionally, please check the memory utilization in the Notebooks instance:

  1. Connect to the VM instance using SSH.
  2. Run the following command: free -t -h
  3. If the used memory is above 85% of the total, you should consider changing the machine type.

You can install Cloud Monitoring agent to monitor if there is high memory usage in their Notebooks instance. See pricing information.

Cause

This can be due to disk running out of space or heavy memory utilization of the VM instance.