Troubleshooting notebooks

Connecting to and opening JupyterLab notebooks

Nothing happens after clicking Open JupyterLab

Verify that your browser does not block pop-up tabs. JupyterLab opens in a new browser tab.

No Inverting Proxy server access to JupyterLab

AI Platform Notebooks uses a Google internal Inverting Proxy server to provide access to JupyterLab. AI Platform Notebooks instance settings, network configuration, and other factors can prevent access to JupyterLab. Use SSH to connect to JupyterLab and learn more about why you might not have access through the Inverting Proxy.

Unable to SSH into notebook instance

AI Platform Notebooks uses OS Login to enable SSH access. This is done automatically at AI Platform Notebooks instance creation time by setting the metadata entry enable-oslogin value to TRUE. To enable SSH access for AI Platform Notebooks for users, complete the steps for configuring OS Login roles on user accounts.

Opening a notebook results in a 403 (Forbidden) error

There are 3 different options to access notebook JupyterLab:

Single User
Service Account
Project Editors

The access mode is configured during Notebook creation and it is defined in the notebook metadata:

Single User: proxy-mode=mail, proxy-user-mail=user@domain.com
Service Account: proxy-mode=service_account
Project Editors: proxy-mode=project_editors

If you can't access a notebook when you click Open JupyterLab, try the following:

Verify that the user accessing the instance has the iam.serviceAccounts.ActAs permission for the defined service account. The service account on the instance provides access to other Google Cloud services. You can use any service account within the same project, but you must have the Service Account User permission (iam.serviceAccounts.actAs) to access the instance. If not specified, the Compute Engine default service account is used and this permission is required as well.

The following example shows how to specify a service account when you create an instance:

gcloud notebooks instances create nb-1 \
  --vm-image-family=tf2-latest-cpu \
  --metadata=proxy-mode=mail,proxy-user-mail=user@domain.com \
  --service-account=your_service_account@project_id.iam.gserviceaccount.com \
  --location=us-west1-a

When you click Open JupyterLab to open a notebook, the notebook opens in a new browser tab. If you are signed in to more than one Google account, the new tab opens with your default Google account. If you did not create your notebook instance with your default Google account, the new browser tab will show a 403 (Forbidden) error.

Opening a notebook results in a 504 (Gateway Timeout) error

This is an indication of an internal proxy timeout or a backend server (Jupyter) timeout. This can be seen when:

The request never reached the internal Inverting Proxy server
Backend(Jupyter) returns a 504 error.

If you can't access a notebook:

Open a Google support case.

Opening a notebook results in a 524 (A Timeout Occurred) error

The internal Inverting Proxy server hasn't received a response from the Inverting Proxy agent for the request within the timeout period. Inverting Proxy agent runs inside your notebook instance as a Docker container. A 524 error is usually an indication that the Inverting Proxy agent isn't connecting to the Inverting Proxy server or the requests are taking too long on the backend server side (Jupyter). A typical case for this error is on the user side (e.g. A networking issue. Or the Inverting Proxy agent/Jupyter service isn't running)

If you can't access a notebook, try the following:

Verify that your notebook is started.
Verify that the Notebooks instance disk is not out of space.
1. Connect to your Deep Learning VM instance using SSH.
2. Run the following command:
```
df -h -T /home/jupyter
```
  If the Use% is above 85%, you need to manually delete files from /home/jupyter. As a first step, you can empty the trash with the following command:
```
sudo rm -rf  /home/jupyter/.local/share/Trash/*
```
Verify that the Docker service is started.
Verify that the Inverting Proxy agent is running. If the agent is started, try restarting it.
Make sure the Jupyter service is running. If it is, try restarting it.
Verify memory utilization in the Notebooks instance.
1. Connect to your Deep Learning VM instance using SSH.
2. Run the following command:
```
free -t -h
```
  If the used memory is above 85% of the total, you should consider changing the machine type.
3. You can install Cloud Monitoring agent to monitor if there is high memory usage in your Notebooks instance. See pricing information.
Verify that you are using Deep Learning VM version M55 or later.

Opening a notebook results in a 598 (Network read timeout) error

The Inverting Proxy server hasn't heard from the Inverting Proxy agent at all for more than 10 minutes, this is a strong indication of an Inverting Proxy agent/Jupyter issue.

If you can't access a notebook, try the following:

Verify that your notebook is started.
Verify that the Docker service is started.
Verify that the Inverting Proxy agent is running. If the agent is started, try restarting it.
Make sure the Jupyter service is running. If it is, try restarting it.
Verify that you are using AI Platform Deep Learning VM Image version M55 or later.

Notebook is unresponsive

If your notebook instance isn't executing cells or appears to be frozen, first try restarting the kernel by clicking Kernel from the top menu and then Restart Kernel. If that doesn't work, you can try the following:

From a terminal session in the notebook, run top to see if there are processes consuming the CPU
From the terminal, check the amount of free disk space using df or available RAM using free
Shut your instance down by selecting it from the Notebook instances page and clicking Stop. Once it has stopped completely, select it and click Start.

Working with files

Downloading files from JupyterLab results in 403 (Forbidden) error

The notebook package in the M23 release of Deep Learning VM includes a bug that prevents you from downloading a file using the JupyterLab UI. You can read more about the bug at Cannot download files after JL update and Download file functionality is broken in notebook packages version 5.7.6+ (5.7.7, 5.7.8).

If you are using the M23 release of Deep Learning VM you can resolve the issue in one of two ways:

Use a Safari browser. The download functionality works for Safari.
Downgrade your notebook package to version 5.7.5.

To downgrade your notebook package:
1. Connect to your Deep Learning VM using SSH. For information on connecting to a VM using SSH, see Connecting to instances.
2. Run the following commands:
```
sudo pip3 install notebook==5.7.5
sudo service jupyter restart
```

After restarting VM, local files cannot be referenced from notebook terminal

Sometimes after restarting an AI Platform Notebooks instance, local files cannot be referenced from within a notebook terminal.

This is a known issue. To reference your local files from within a notebook terminal, first re-establish your current working directory using the following command:

cd PWD

In this command, replace PWD with your current working directory. For example, if your current working directory was /home/jupyter/, use the command cd /home/jupyter/.

After re-establishing your current working directory, your local files can be referenced from within the notebook terminal.

GPU quota has been exceeded

Determine the number of GPUs available in your project by checking the quotas page. If GPUs are not listed on the quotas page, or you require additional GPU quota, you can request a quota increase. See Requesting additional quota on the Compute Engine Resource Quotas page.

Creating AI Platform Notebooks instances

New notebook is not created (insufficient permissions)

It usually takes about a minute to create a notebook instance. If your new notebook instance remains in the pending state indefinitely, it might be because the service account used to start the notebook instance does not have the required Editor permission in your Google Cloud Platform (GCP) project.

You can start a notebook instance with a custom service account that you create or in single-user mode with a userid. If you start a notebook instance in single-user mode, then your notebook instance begins the boot process using Compute Engine default service account before turning control over to your userid.

To verify that a service account has the appropriate permissions, follow these steps:

Console

Open the IAM page in the Cloud Console.
Open the IAM page
Determine the service account used with your notebook instance, which is one of the following:
- A custom service account that you specified when you created your notebook instance.
- The Compute Engine default service account for your GCP project, which is used when you start your notebook instance in single-user mode. The Compute Engine default service account for your GCP project is named PROJECT_NUMBER-compute@developer.gserviceaccount.com. For example: 113377992299-compute@developer.gserviceaccount.com.
Verify that your service account is in the Editor role.
If not, edit the service account and add it to the Editor role.

For more information, see Granting, changing, and revoking access to resources in the IAM documentation.

gcloud

If you have not already, install the gcloud command-line tool.
Get the name and project number for your GCP project with the following command. Replace PROJECT_ID with the project ID for your GCP project.
```
gcloud projects describe PROJECT_ID
```
You should see output similar to the following, which displays the name (name:) and project number (projectNumber:) for your project.
```
createTime: '2018-10-18T21:03:31.408Z'
lifecycleState: ACTIVE
name: my-project-name
parent:
 id: '396521612403'
 type: folder
projectId: my-project-id-1234
projectNumber: '113377992299'
```
Determine the service account used with your notebook instance, which is one of the following:
- A custom service account that you specified when you created your notebook instance.
- The Compute Engine default service account for your GCP project, which is used when you start your notebook instance in single-user mode. The Compute Engine default service account for your GCP project is named PROJECT_NUMBER-compute@developer.gserviceaccount.com. For example: 113377992299-compute@developer.gserviceaccount.com.
Add the roles/editor role to the service account with the following command. Replace project-name with the name of your project, and replace service-account-id with the service account ID for your notebook instance.
```
gcloud projects add-iam-policy-binding project-name \
 --member serviceAccount:service-account-id \
 --role roles/editor
```

Creating an instance results in a "Permission denied" error

When creating a new instance, verify that the user creating the instance has the iam.serviceAccounts.ActAs permission for the defined service account.

The service account on the instance provides access to other Google Cloud services. You can use any service account within the same project, but you must have the Service Account User permission (iam.serviceAccounts.actAs) to create the instance. If not specified, the Compute Engine default service account is used.

The following example shows how to specify a service account when you create an instance:

gcloud notebooks instances create nb-1 \
  --vm-image-family=tf2-latest-cpu \
  --service-account=your_service_account@project_id.iam.gserviceaccount.com \
  --location=us-west1-a

To grant the Service Account User permission, see Allowing a member to impersonate a single service account.

Creating a new instance results in an "already exists" error

When creating a new instance, verify that an AI Platform Notebooks instance with the same name was not deleted previously by Compute Engine and still exists in the AI Platform Notebooks API database.

The following example shows how to list instances using the AI Platform Notebooks API and verify their state.

gcloud notebooks instances list --location=LOCATION

If an instance's state is DELETED, run the following command to delete it permanently.

gcloud notebooks instances delete INSTANCE_NAME --location=LOCATION

Helpful procedures

To re-register with Inverting Proxy server

To re-register the Notebooks with the internal Inverting Proxy server, you can stop and start the VM from the Notebook instances page or you can log in to the notebook instance via SSH and enter:

cd /opt/deeplearning/bin
sudo ./attempt-register-vm-on-proxy.sh

Verify the Docker service status

To verify the Docker service status you can log in to the notebook instance via SSH and enter:

sudo service docker status

Verify that the Inverting Proxy agent is running

To verify if the notebook Inverting Proxy agent is running, log in to the notebook instance via SSH and enter:

# Confirm Inverting Proxy agent Docker container is running (proxy-agent)
sudo docker ps

# Verify State.Status is running and State.Running is true.
sudo docker inspect proxy-agent

# Grab logs
sudo docker logs proxy-agent

Verify the Jupyter service status and collect logs

To verify the Jupyter service status you can log in to the notebook instance via SSH and enter:

sudo service jupyter status

To collect Jupyter service logs:

sudo journalctl -u jupyter.service --no-pager

Verify the Jupyter internal API is active

To verify the Jupyter internal API is active you can log in to the notebook instance via SSH and enter:

curl http://127.0.0.1:8080/api/kernelspecs

Restart the Docker service

To restart the Docker service, you can stop and start the VM from the Notebook instances page or you can log in to the notebook instance via SSH and enter:

sudo service docker restart

Restart the Inverting Proxy agent

To restart the Inverting Proxy agent, you can stop and start the VM from the Notebook instances page or you can log in to the notebook instance via SSH and enter:

sudo docker restart proxy-agent

Restart the Jupyter service

To restart the Jupyter service, you can stop and start the VM from the Notebook instances page or you can log in to the notebook instance via SSH and enter:

sudo service jupyter restart