This page describes troubleshooting steps that you might find helpful if you run into problems when you use Vertex AI Workbench.
See also Troubleshooting Vertex AI for help using other components of Vertex AI.
To filter this page's content, click a topic:
Helpful procedures
This section describes procedures that you might find helpful.
Use SSH to connect to your user-managed notebooks instance
Use ssh to connect to your instance by typing the following command in either Cloud Shell or any environment where the Google Cloud CLI is installed.
gcloud compute ssh --project PROJECT_ID \
--zone ZONE \
INSTANCE_NAME -- -L 8080:localhost:8080
Replace the following:
PROJECT_ID
: Your project IDZONE
: The Google Cloud zone where your instance is locatedINSTANCE_NAME
: The name of your instance
Re-register with the Inverting Proxy server
To re-register the user-managed notebooks instance with the internal Inverting Proxy server, you can stop and start the VM from the User-managed notebooks page or you can use ssh to connect to your user-managed notebooks instance and enter:
cd /opt/deeplearning/bin sudo ./attempt-register-vm-on-proxy.sh
Verify the Docker service status
To verify the Docker service status you can use ssh to connect to your user-managed notebooks instance and enter:
sudo service docker status
Verify that the Inverting Proxy agent is running
To verify if the notebook Inverting Proxy agent is running, use ssh to connect to your user-managed notebooks instance and enter:
# Confirm Inverting Proxy agent Docker container is running (proxy-agent) sudo docker ps # Verify State.Status is running and State.Running is true. sudo docker inspect proxy-agent # Grab logs sudo docker logs proxy-agent
Verify the Jupyter service status and collect logs
To verify the Jupyter service status you can use ssh to connect to your user-managed notebooks instance and enter:
sudo service jupyter status
To collect Jupyter service logs:
sudo journalctl -u jupyter.service --no-pager
Verify that the Jupyter internal API is active
To verify that the Jupyter internal API is active you can use ssh to connect to your user-managed notebooks instance and enter:
curl http://127.0.0.1:8080/api/kernelspecs
Restart the Docker service
To restart the Docker service, you can stop and start the VM from the User-managed notebooks page or you can use ssh to connect to your user-managed notebooks instance and enter:
sudo service docker restart
Restart the Inverting Proxy agent
To restart the Inverting Proxy agent, you can stop and start the VM from the User-managed notebooks page or you can use ssh to connect to your user-managed notebooks instance and enter:
sudo docker restart proxy-agent
Restart the Jupyter service
To restart the Jupyter service, you can stop and start the VM from the User-managed notebooks page or you can use ssh to connect to your user-managed notebooks instance and enter:
sudo service jupyter restart
Make a copy of the user data on an instance
To store a copy of an instance's user data in Cloud Storage, complete the following steps.
Create a Cloud Storage bucket (optional)
In the same project where your instance is located, create a Cloud Storage bucket where you can store your user data. If you already have a Cloud Storage bucket, skip this step.
-
Create a Cloud Storage bucket:
gcloud storage buckets create gs://BUCKET_NAME
ReplaceBUCKET_NAME
with a bucket name that meets the bucket naming requirements.
Copy your user data
In your instance's JupyterLab interface, select File > New > Terminal to open a terminal window. For user-managed notebooks instances, you can instead connect to your instance's terminal by using SSH.
Use the
gsutil tool
to copy your user data to a Cloud Storage bucket. The following example command copies all of the files from your instance's/home/jupyter/
directory to a directory in a Cloud Storage bucket.gsutil cp -R /home/jupyter/* gs://BUCKET_NAMEPATH
Replace the following:
BUCKET_NAME
: the name of your Cloud Storage bucketPATH
: the path to the directory where you want to copy your files, for example:/copy/jupyter/