Upgrade the environment of a user-managed notebooks instance

Upgrade the environment of a user-managed notebooks instance

Vertex AI Workbench user-managed notebooks is deprecated. On January 30, 2025, support for user-managed notebooks will end and the ability to create user-managed notebooks instances will be removed. Existing instances will continue to function but patches, updates, and upgrades won't be available. To continue using Vertex AI Workbench, we recommend that you migrate your user-managed notebooks instances to Vertex AI Workbench instances.

Vertex AI Workbench user-managed notebooks instances are Deep Learning VM Images instances with JupyterLab notebook environments enabled and ready for use. This page describes how to upgrade the environment of a user-managed notebooks instance.

Reasons to upgrade

You might want to upgrade the environment of your user-managed notebooks instance for any of the following reasons:

• To use new capabilities that are only available in a newer version of your environment.

• To benefit from framework updates, package updates, and bug fixes that have been implemented in a newer version of your environment.

Upgrade methods

There are two ways to upgrade a user-managed notebooks instance:

• Automatic upgrade: Enable auto upgrade when you create a user-managed notebooks instance. During a recurring time period that you specify, Vertex AI Workbench checks whether your instance can be upgraded, and if so, Vertex AI Workbench upgrades your instance.

• Manual upgrade: If an existing user-managed notebooks instance meets the requirements for upgrading, you can upgrade the instance manually.

Requirements and limitations

Backward compatibility with your user-managed notebooks isn't guaranteed. Make a copy of your data before upgrading a user-managed notebooks instance.

To determine whether you can upgrade a specific user-managed notebooks instance, see the following requirements and limitations:

• The Notebooks API must be enabled in the instance's Google Cloud project. For more information, see List enabled services and Enable an API.

• The user-managed notebooks instance's environment version must be M54 or higher.

• The user-managed notebooks instance must have been created using the Notebooks API or must have been registered with the Notebooks API.

• If your user-managed notebooks instance is container-based, Vertex AI Workbench upgrades the OS. The image version depends on the specific image pulled by your Dockerfile.

  To help make sure the upgrade uses the most recent version of the image, consider using the latest tag in your Dockerfile.

If upgrading your instance is not an option for you, consider migrating your data to a new user-managed notebooks instance.

How the upgrade works

User-managed notebooks instances that can be upgraded are dual-disk, with one boot disk and one data disk. The upgrade process upgrades the boot disk to a new image while preserving your data on the data disk.

Which components are upgraded or preserved?

The following table shows which components of your user-managed notebooks instance are upgraded and which are preserved.

Component	Upgrade result
Machine learning frameworks	Upgraded
Machine learning data	Preserved
Preinstalled dependencies	Upgraded
User-installed libraries	By default, must be reinstalled (see User-installed libraries)
Local files in the /home/jupyter directory	Preserved
Local files in any other /home/ directory	Not preserved
Preinstalled operating system packages	Upgraded
User-installed operating system packages	Not preserved
GPU drivers	Upgraded
Notebooks	Preserved
User configurations	Preserved

User-installed libraries

By default, user-managed notebooks instances store pip and Conda libraries in the boot disk, which is replaced during an upgrade. When you install pip libraries, you can include the --user flag to install them in the /home/jupyter/ directory, where they are preserved during an upgrade.

By default, if you install pip or Conda libraries in a kernel created from a custom container, the libraries only persist while the kernel is running. Each time the kernel is restarted, those libraries will need to be reinstalled. To install persistent libraries in a custom container, include the library installations in your Dockerfile. When installing pip libraries in a kernel created from a custom container, you can include the --user flag so that the libraries will persist until instance restart.

Environment versions

Vertex AI Workbench updates environments regularly (see the Deep Learning VM release notes), but with each released version, not all of the environments are updated. Vertex AI Workbench only upgrades an instance if there is a newer environment version for the VM image that your instance is based on.

For information about how to use a specific version to create a user-managed notebooks instance, see Create a specific version of a user-managed notebooks instance.

Before you upgrade

Before you upgrade, complete the following steps.

1. Check the release notes to learn about updates to newer versions.

2. Make a copy of your data as a backup.

Automatic upgrade

Vertex AI Workbench can automatically upgrade instances that are running. If your instance is stopped, it doesn't automatically upgrade your instance, even if you enabled auto upgrade when you created it.

When you enable automatic environment upgrades, you specify a recurring time period in which Vertex AI Workbench checks whether the instance can be upgraded, and if it can be, upgrades the instance.

The time period you specify is stored as a notebook-upgrade-schedule metadata entry, in unix-cron format, Greenwich Mean Time (GMT).

To check whether an instance can be upgraded, Vertex AI Workbench uses the API method isUpgradeable. This method checks for a newer version of the image on the instance's boot disk.

If the instance can be upgraded, Vertex AI Workbench uses an internal upgrade method to upgrade the instance.

Create a user-managed notebooks instance with auto upgrade enabled

To create a user-managed notebooks instance with auto upgrade enabled, select the Enable environment auto-upgrade checkbox and set a schedule when you create the instance.

You can specify auto-upgrade by using either the Google Cloud console or the Google Cloud CLI.

Before you begin

Before you can create a user-managed notebooks instance, you must have a Google Cloud project and enable the Notebooks API for that project.

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. Enable the Notebooks API.

   Enable the API

5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

6. Make sure that billing is enabled for your Google Cloud project.

7. Enable the Notebooks API.

   Enable the API

8. If you plan to use GPUs with your user-managed notebooks instance, check the quotas page in the Google Cloud console to ensure that you have enough GPUs available in your project. If GPUs are not listed on the quotas page, or you require additional GPU quota, you can request a quota increase. See Requesting an increase in quota on the Compute Engine Resource quotas page.

Required roles

If you created the project, you have the Owner (roles/owner) IAM role on the project, which includes all required permissions. Skip this section and start creating your user-managed notebooks instance. If you didn't create the project yourself, continue in this section.

To ensure that your user account has the necessary permissions to create a Vertex AI Workbench user-managed notebooks instance, ask your administrator to grant your user account the following IAM roles on the project:

• Notebooks Admin (roles/notebooks.admin)
• Service Account User (roles/iam.serviceAccountUser)

For more information about granting roles, see Manage access to projects, folders, and organizations.

Your administrator might also be able to give your user account the required permissions through custom roles or other predefined roles.

Create the instance

Console

1. In the Google Cloud console, go to the User-managed notebooks page. Or go to notebook.new (https://notebook.new) and skip the next step.

   Go to User-managed notebooks

2. Click  New notebook, and then select Customize.

3. On the Create a user-managed notebook page, in the Details section, provide the following information for your new instance:

   • Name: a name for your new instance
   • Region and Zone: Select a region and zone for the new instance. For best network performance, select the region that is geographically closest to you. See the available user-managed notebooks locations.

4. In the System health section, select Environment auto-upgrade.

5. Choose whether to upgrade your notebook Weekly or Monthly.

6. In the Weekday field, select the option that you want.

7. In the Hour field, choose an hour of the day.

8. Complete the rest of the instance creation dialog, and then click Create.

gcloud

1. From Cloud Shell or any environment where the Google Cloud CLI is installed, enter the following Google Cloud CLI command:

   gcloud notebooks instances create INSTANCE_NAME
       --metadata=notebook-upgrade-schedule=SCHEDULE
       --vm-image-project=deeplearning-platform-release
       --vm-image-family=VM_IMAGE_FAMILY
       --machine-type=MACHINE_TYPE
       --location=LOCATION
   

   Replace the following:

   • INSTANCE_NAME: the name of your new instance
   • SCHEDULE: the weekly or monthly schedule that you set, in unix-cron format; for example, "00 19 * * MON" means weekly on Monday, at 1900 hours Greenwich Mean Time (GMT)
   • VM_IMAGE_FAMILY: the image family name that you want to use to create your instance
   • MACHINE_TYPE: the machine type of your instance's VM
   • LOCATION: the Google Cloud location where you want your new instance to be

2. Access your instance from the Google Cloud console.

Edit the auto upgrade schedule

To edit the auto upgrade schedule after you have created your user-managed notebooks instance, complete the following steps:

1. In the Google Cloud console, go to the User-managed notebooks page.

   Go to User-managed notebooks

2. Click the instance name that needs the schedule change.

3. On the Notebook details page, in the Environment auto-upgrade section, edit the schedule.

4. Click Submit to save your changes.

Manual upgrade

You can manually upgrade user-managed notebooks instances that meet the requirements.

Check for a newer version of your instance's environment

To check whether a newer version of your instance's environment is available, access your instance from the Google Cloud console.

1. In the Google Cloud console, go to the User-managed notebooks page.

   Go to User-managed notebooks

2. If your instance isn't running, start the instance. Vertex AI Workbench can only upgrade instances when they're running.

3. Click the instance name that you want to check for availability of a newer environment version.

4. On the Notebook details page, next to VM details, click View in Compute Engine.

   If a newer version of the environment is available, a "This instance needs to be upgraded" message appears.

Upgrade your instance's environment to a newer version

You can manually upgrade a user-managed notebooks instance in the Google Cloud console or by using the Google Cloud CLI.

Note: Upgrades can affect user operations that are running.

Console

1. In the Google Cloud console, go to the User-managed notebooks page.

   Go to User-managed notebooks

2. If your instance isn't running, start the instance. Vertex AI Workbench can only upgrade instances when they're running.

3. Click the instance name that you want to upgrade.

4. On the Notebook details page, click  Upgrade.

5. Make sure you have made a copy of the data on your instance before continuing.

6. After your data is backed up, click Upgrade. Vertex AI Workbench upgrades and starts your instance.

gcloud

1. To check whether an instance can be upgraded, use the API method isUpgradeable. This method checks for a newer version of the image on the instance's boot disk.

   gcloud notebooks instances is-upgradeable INSTANCE_NAME \
       --location=LOCATION
   

   Replace the following:

   • INSTANCE_NAME: the name of your instance
   • LOCATION: the Google Cloud location where your instance is located

   If the instance is upgradable, the response is true. If the response is false, the instance cannot be upgraded, but you can still try to migrate your data to a new instance.

2. If your instance isn't running, start the instance. Vertex AI Workbench can only upgrade instances when they're running.

3. Make sure you have made a copy of the data on your instance before continuing.

4. If the instance is upgradable, use the API method upgrade to upgrade the instance:

   gcloud notebooks instances upgrade INSTANCE_NAME \
       --location=LOCATION
   

   Vertex AI Workbench upgrades the instance.

Roll back an upgrade

To roll back an upgrade, complete the following steps:

1. In the Google Cloud console, go to the User-managed notebooks page.

   Go to User-managed notebooks

2. Click the instance name that you would like to roll back.

3. On the Notebook details page, under Upgrade history, click Rollback.

4. Vertex AI Workbench rolls your instance back to the previous version.