User-managed notebooks instances provide several methods for monitoring the health of your notebooks. This page describes how to use each method.
Methods for monitoring health status
You can monitor the health of your user-managed notebooks instances in a few different ways. This page describes how to use the following methods:
Set up the gcloud CLI
To complete some of the steps on this page, you need to use the Google Cloud CLI. Install and initialize the Google Cloud CLI.Use guest attributes to report system health
You can use guest attributes to report the system health of the following core services:
- Docker service
- Docker reverse proxy agent
- Jupyter service
- Jupyter API
Guest attributes are a specific type of custom metadata that applications can write to while running on your user-managed notebooks instance. To learn more about guest attributes, see Storing and retrieving metadata.
How instances use guest attributes to report system health
The notebooks-collection-agent
service runs a Python process
in the background that verifies the status of
the user-managed notebooks
instance's core services and updates the guest attributes as either
1
if no problems are detected or -1
if a failure is detected.
To use the notebooks-collection-agent
service to
report on your user-managed notebooks instance's health,
you must enable the following guest attributes while
creating a user-managed notebooks instance:
enable-guest-attributes=TRUE
: This enables guest attributes on your user-managed notebooks instance. All new instances enable this attribute by default.report-system-health=TRUE
: This records system health check results to your guest attributes.
The notebooks-collection-agent
service doesn't need
any special permissions to write to the instance's guest attributes.
Create a user-managed notebooks instance with system health guest attributes enabled
To use system health guest attributes to report on your user-managed notebooks instance's health, you must select the Enable system health report checkbox when you create a user-managed notebooks instance.
You can enable the system health report by using either the Google Cloud console, notebook.new (https://notebook.new), or the Google Cloud CLI.
Follow the steps in Before you begin to create a Google Cloud project and enable the Notebooks API.
Console
In the Google Cloud console, go to the User-managed notebooks page.
Click
New notebook, and then select Customize instance.The Create a user-managed notebook page opens.
For information about enabling system health report, see Environment upgrade and system health.
notebook.new
Go to notebook.new (https://notebook.new).
The Create a user-managed notebook dialog opens.
For information about enabling system health report, see Environment upgrade and system health.
gcloud
From Cloud Shell or any environment where the Google Cloud CLI is installed, enter the following Google Cloud CLI command:
gcloud notebooks instances create INSTANCE_NAME \ --vm-image-project=deeplearning-platform-release \ --vm-image-family=IMAGE_FAMILY \ --machine-type=MACHINE_TYPE \ --location=ZONE \ --metadata=enable-guest-attributes=TRUE,report-system-health=TRUE
Replace the following:
INSTANCE_NAME
: the name of your new instanceIMAGE_FAMILY
: the image family name that you want to use to create your instanceMACHINE_TYPE
: the machine type of your instance's VM; for example,n1-standard-4
ZONE
: the zone where you want your new instance to be located, for example,us-west1-a
Access your instance from the Cloud console.
Monitor system health through guest attributes
For user-managed notebooks instances that have the related guest attributes enabled, you can retrieve the values of your system health guest attributes by using either the Google Cloud console, the Google Cloud CLI, or the Notebooks API.
Console
In the Google Cloud console, go to the User-managed notebooks page.
Click the instance name that you want to view the system health status of.
On the Notebook details page, click the Health tab. Review the status of your instance and its core services.
gcloud
gcloud compute instances get-guest-attributes INSTANCE_NAME \
--zone ZONE
Replace the following:
INSTANCE_NAME
: the name of your instanceZONE
: the zone where your instance is locatedIf your core services are healthy, the results look like the following. A value of
1
means no failure was detected.NAMESPACE KEY VALUE notebooks docker_proxy_agent_status 1 notebooks docker_status 1 notebooks jupyterlab_api_status 1 notebooks jupyterlab_status 1 notebooks system-health 1 notebooks updated 2020-10-01 17:00:00.12345
If any of the four core services fail, system-health reports a
-1
value to indicate system failure. In most cases, a system failure means that JupyterLab is not accessible.An example of a failure result might look like the following.
NAMESPACE KEY VALUE notebooks docker_proxy_agent_status -1 notebooks docker_status -1 notebooks jupyterlab_api_status 1 notebooks jupyterlab_status 1 notebooks system-health -1 notebooks updated 2020-10-01 17:00:00.12345
Notebooks API
To monitor your system health, you can use the getInstanceHealth method to retrieve the values of your guest attributes.
The following example shows how to do this using the gcloud CLI.
gcloud notebooks instances is-healthy example-instance \
--location=ZONE
Replace ZONE
with the zone where your instance
is located, for example, us-west1-a
.
If your core services are healthy, the results look like the following.
A value of 1
means no failure was detected.
{ "health_state": HEALTHY, "docker-proxy-agent": 1, "docker-service": 1, "jupyter-service": 1, "jupyter-api": 1, "last-updated": "2020-10-01 17:00:30.12345" }
An example of a failure result might look like the following.
{ "healthy": UNHEALTHY, "docker-proxy-agent": 1, "docker-service": 1, "jupyter-service": -1, "jupyter-api": -1, "last-updated": "2020-10-01 17:00:30.12345" }
Report custom metrics to Monitoring
User-managed notebooks instances let you collect system status and JupyterLab metrics and report them to Cloud Monitoring. These custom metrics are different from the standard metrics that are reported when you install Monitoring on your user-managed notebooks instance.
The custom metrics reported to Monitoring include the following:
The system health of these user-managed notebooks core services:
- Docker service
- Docker reverse proxy agent
- Jupyter service
- Jupyter API
The following JupyterLab metrics:
- Number of kernels
- Number of terminals
- Number of connections
- Number of sessions
- Maximum memory
- High memory
- Current memory
How instances report custom metrics to Monitoring
To report custom metrics to Monitoring, you must enable
the report-notebook-metrics
metadata setting while
creating a user-managed notebooks instance.
You must also make sure that the user-managed notebooks
instance's service account
has Monitoring Metric Writer (roles/monitoring.metricWriter
)
permissions. For more information, see
Manage access to projects, folders, and organizations.
Create a user-managed notebooks instance that reports custom metrics to Monitoring
To report custom metrics to Monitoring, you must select the Report custom metrics to Cloud Monitoring checkbox when you create a user-managed notebooks instance.
You can enable reporting custom metrics to Cloud Monitoring by using either the Google Cloud console, notebook.new (https://notebook.new), or the Google Cloud CLI.
Follow the steps in Before you begin to create a Google Cloud project and enable the Notebooks API.
Console
In the Google Cloud console, go to the User-managed notebooks page.
Click
New notebook, and then select Customize instance.The Create a user-managed notebook page opens.
For information about enabling Report custom metrics to Cloud Monitoring, see Environment upgrade and system health.
notebook.new
Go to notebook.new (https://notebook.new).
The Create a user-managed notebook dialog opens.
For information about enabling Report custom metrics to Cloud Monitoring, see Environment upgrade and system health.
gcloud
From Cloud Shell or any environment where the Google Cloud CLI is installed, enter the following Google Cloud CLI command:
gcloud notebooks instances create INSTANCE_NAME \ --vm-image-project=deeplearning-platform-release \ --vm-image-family=IMAGE_FAMILY \ --machine-type=MACHINE_TYPE \ --location=ZONE \ --metadata=report-notebook-metrics=TRUE
Replace the following:
INSTANCE_NAME
: the name of your new instanceIMAGE_FAMILY
: the image family name that you want to use to create your instanceMACHINE_TYPE
: the machine type of your instance's VM, for example,n1-standard-4
ZONE
: the zone where you want your new instance to be located, for example,us-west1-a
Access your instance from the Cloud console.
Grant Monitoring Metric Writer permissions to the service account
After you've created
your new user-managed notebooks instance,
grant Monitoring Metric Writer permissions
(roles/monitoring.metricWriter
) to
the service account for
the user-managed notebooks instance.
For more information, see
Manage access to projects, folders, and organizations.
Monitor custom metrics through Monitoring
For user-managed notebooks instances that have reporting custom metrics enabled, you can monitor your custom metrics by using the Google Cloud console.
In the Google Cloud console, go to the User-managed notebooks page.
Click the instance name that you want to view the custom metrics of.
On the Notebook details page, click the Monitoring tab. Review the custom metrics for your instance.
Install Monitoring on an instance
This option automatically installs Monitoring. The installation requires 256 MB of disk space. An internet connection is required for the metrics to be reported to Monitoring.
How instances report system and application metrics
To report system and application metrics by installing
Cloud Monitoring on your user-managed notebooks instance,
you must select the Install Cloud Monitoring agent checkbox when you create
a user-managed notebooks instance.
These metrics are different from the custom metrics that are reported when
you enable the report-notebook-metrics
metadata
setting.
Create a user-managed notebooks instance that reports system and application metrics to Monitoring
To install Monitoring on your user-managed notebooks instance, you can use either the Google Cloud console, notebook.new (https://notebook.new), or the Google Cloud CLI.
Follow the steps in Before you begin to create a Google Cloud project and enable the Notebooks API.
Console
In the Google Cloud console, go to the User-managed notebooks page.
Click
New notebook, and then select Customize instance.The Create a user-managed notebook page opens.
For information about installing the Monitoring agent, see Environment upgrade and system health.
notebook.new
Go to notebook.new (https://notebook.new).
The Create a user-managed notebook dialog opens.
For information about installing the Monitoring agent, see Environment upgrade and system health.
gcloud
From Cloud Shell or any environment where the Google Cloud CLI is installed, enter the following Google Cloud CLI command:
gcloud notebooks instances create INSTANCE_NAME \ --vm-image-project=deeplearning-platform-release \ --vm-image-family=IMAGE_FAMILY \ --machine-type=MACHINE_TYPE \ --location=ZONE \ --metadata=install-monitoring-agent=TRUE
Replace the following:
INSTANCE_NAME
: the name of your new instanceIMAGE_FAMILY
: the image family name that you want to use to create your instanceMACHINE_TYPE
: the machine type of your instance's VM; for example,n1-standard-4
ZONE
: the zone where you want your new instance to be located, for example,us-west1-a
Access your instance from the Cloud console.
Monitor system and application metrics through Monitoring
For user-managed notebooks instances that have Monitoring installed, you can monitor your system and application metrics by using the Google Cloud console:
In the Google Cloud console, go to the User-managed notebooks page.
Click the instance name that you want to view the system and application metrics of.
On the Notebook details page, click the Monitoring tab. Review the system and application metrics for your instance.
Use the diagnostic tool to monitor system health
User-managed notebooks instances include a built-in diagnostic tool that can help you monitor the system health of your instances.
Tasks performed by the diagnostic tool
The diagnostic tool performs the following tasks:
Verifies the status of the following user-managed notebooks core services:
- Docker service
- Docker reverse proxy agent
- Jupyter service
- Jupyter API
Checks whether the disk space for boot and data disks is used beyond an 85% threshold.
Installs
lsof
(internet connection required).Collects the following instance logs:
- Network information (
ifconfig
,netstat
) - Logs in the
/var/log/
folder - Docker status information
lsof
(open files) data- Docker service status
- Proxy reverse agent status
- Jupyter service status
- Jupyter API status
- Proxy agent configuration file
- Python processes
- Network information (
Runs the following commands and collects the results:
- pip freeze
- conda list
- gcloud compute instances describe
INSTANCE_NAME
- gcloud config list
Run the diagnostic tool
To run the diagnostic tool, complete the following steps:
In the SSH terminal, run the following commands:
sudo -i cd /opt/deeplearning/bin/ ./diagnostic_tool.sh
The diagnostic tool collects the logs, compresses them in a
.tar.gz
file, and places the file in the/tmp/
folder.Extract the file and then evaluate the contents. The contents include:
log
folder: Logs from thevar/log/
folderreport.log
: Output for all commands collectedproxy-agent-config.json
: Proxy configuration information- Docker log: A
-json.log
file that includes Docker container logs
You can use the following options with the diagnostic tool.
Option | Description |
---|---|
-r | A repair option that tries to restore failed user-managed notebooks core services status |
-s | Runs without a confirmation |
-b |
Uploads the .tar.gz file
to a Cloud Storage bucket.
|
-v | A debug option for troubleshooting the tool in case of failures |
-c | Captures 30 seconds of packet traffic into your user-managed notebooks instance, filtering SSH |
-d | A destination folder in which to save the logs |
-h | Help |