Troubleshooting notebooks

Nothing happens after clicking "Open JupyterLab"

Verify that your browser does not block pop-up tabs. JupyterLab opens in a new browser tab.

No proxy access to JupyterLab

AI Platform Notebooks instance settings, network configuration, and other factors can prevent proxy access to JupyterLab. Use SSH to connect to JupyterLab and learn more about why you might not have proxy access.

Downloading files from JupyterLab results in 403 (Forbidden) error

The "notebook" package in the M23 release of Deep Learning VM includes a bug that prevents you from downloading a file using the JupyterLab UI. You can read more about the bug at Cannot download files after JL update and Download file functionality is broken in notebook packages version 5.7.6+ (5.7.7, 5.7.8).

If you are using the M23 release of Deep Learning VM you can resolve the issue in one of two ways:

  • Use a Safari browser. The download functionality works for Safari.

  • Downgrade your notebook package to version 5.7.5.

    To downgrade your notebook package:

    1. Connect to your Deep Learning VM using SSH. For information on connecting to a VM using SSH, see Connecting to instances.

    2. Run the following commands:

      sudo pip3 install notebook==5.7.5
      sudo service jupyter restart
      

Opening a notebook results in a 403 (Forbidden) error

If you can't access a notebook, try the following:

  • Ensure the Google account trying to access the notebook instance has at least AI Platform Notebooks IAM roles, like Notebooks Runner or Notebooks Admin.

  • When you click Open JupyterLab to open a notebook, the notebook opens in a new browser tab. If you are signed in to more than one Google account, the new tab opens with your default Google account. If you did not create your notebook instance with your default Google account, the new browser tab will show a 403 (Forbidden) error.

GPU quota has been exceeded

Determine the number of GPUs available in your project by checking the quotas page. If GPUs are not listed on the quotas page, or you require additional GPU quota, you can request a quota increase. See Requesting additional quota on the Compute Engine Resource Quotas page.

New notebook is not created (insufficient permissions)

It usually takes about a minute to create a notebook instance. If your new notebook instance remains in "pending" state indefinitely, it might be because the service account used to start the notebook instance does not have the required Editor permission in your Google Cloud Platform (GCP) project.

You can start a notebook instance with a custom service account that you create or in single-user mode with a userid. If you start a notebook instance in single-user mode, then your notebook instance begins the boot process using Compute Engine default service account before turning control over to your userid.

To verify that a service account has the appropriate permissions, follow these steps:

Console

  1. Open the IAM page in the Cloud Console.

    Open the IAM page

  2. Determine the service account used with your notebook instance, which is one of the following:

    • A custom service account that you specified when you created your notebook instance.

    • The Compute Engine default service account for your GCP project, which is used when you start your notebook instance in single-user mode. The Compute Engine default service account for your GCP project is named project-number-compute@developer.gserviceaccount.com. For example: 113377992299-compute@developer.gserviceaccount.com.

  3. Verify that your service account is in the Editor role.

  4. If not, edit the service account and add it to the Editor role.

For more information, see Granting, changing, and revoking access to resources in the IAM documentation.

gcloud

  1. If you have not already, install the gcloud command-line tool.

  2. Get the name and project number for your GCP project with the following command. Replace project-id with the project ID for your GCP project.

    gcloud projects describe project-id
    

    You should see output similar to the following, which displays the name (name:) and project number (projectNumber:) for your project.

    createTime: '2018-10-18T21:03:31.408Z'
    lifecycleState: ACTIVE
    name: my-project-name
    parent:
     id: '396521612403'
     type: folder
    projectId: my-project-id-1234
    projectNumber: '113377992299'
    
  3. Determine the service account used with your notebook instance, which is one of the following:

    • A custom service account that you specified when you created your notebook instance.

    • The Compute Engine default service account for your GCP project, which is used when you start your notebook instance in single-user mode. The Compute Engine default service account for your GCP project is named project-number-compute@developer.gserviceaccount.com. For example: 113377992299-compute@developer.gserviceaccount.com.

  4. Add the roles/editor role to the service account with the following command. Replace project-name with the name of your project, and replace service-account-id with the service account ID for your notebook instance.

    gcloud projects add-iam-policy-binding project-name \
     --member serviceAccount:service-account-id \
     --role roles/editor
    

Creating an instance using the SDK results in a "Permission denied" error

When creating a new instance, verify that the user creating the instance has the iam.serviceAccounts.ActAs permission for the defined service account.

The service account on the instance provides access to other Google Cloud services. You can use any service account within the same project, but you must have the service account user permission (iam.serviceAccounts.actAs) to create the instance. If not specified, the Compute Engine default service account is used.

The following example shows how to specify a service account when you create an instance:

gcloud beta notebooks instances create nb-1 \
  --vm-image-project=deeplearning-platform-release \
  --vm-image-family=tf2-latest-cpu \
  --machine-type=n1-standard-1 \
  --service-account=your_service_account@project_id.iam.gserviceaccount.com \
  --location=us-west1-a

Notebook is unresponsive

If your notebook instance isn't executing cells or appears to be frozen, first try restarting the kernel by clicking Kernel from the top menu and then Restart Kernel. If that doesn't work, you can try the following:

  • From a terminal session in the notebook, run top to see if there are processes consuming the CPU
  • From the terminal, check the amount of free disk space using df or available RAM using free
  • Shut your instance down by selecting it from the Notebook instances page and clicking Stop. Once it has stopped completely, select it and click Start.

Restart the Jupyter service

To restart the Jupyter service, you can stop and start the VM from the Notebook instances page or you can log in to the notebook instance via SSH and enter:

sudo service jupyter restart