Create notebooks

This document describes how to create Colab Enterprise notebooks in BigQuery. Notebooks are BigQuery Studio code assets powered by Dataform.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. Enable the BigQuery API.

   Enable the API

5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

6. Make sure that billing is enabled for your Google Cloud project.

7. Enable the BigQuery API.

   Enable the API

Enable BigQuery Studio

Follow the instructions at Enable BigQuery Studio for asset management to save, share, and manage versions of code assets such as notebooks.

Required permissions

Set the appropriate permissions to create, edit, or view notebooks.

All users with the Dataform Admin (roles/dataform.admin) role have owner access to all notebooks created in the project.

For more information about BigQuery Identity and Access Management (IAM), see Access control with IAM.

Permissions to create notebooks

To create and run notebooks, you need the following IAM permissions:

• resourcemanager.projects.get
• resourcemanager.projects.list
• bigquery.config.get
• bigquery.jobs.create
• bigquery.readsessions.create
• bigquery.readsessions.getData
• bigquery.readsessions.update
• resourcemanager.projects.get
• resourcemanager.projects.list
• dataform.locations.get
• dataform.locations.list

• dataform.repositories.create

  Note: Users who have the dataform.repositories.create permission can execute code using the default Dataform service account and all permissions granted to that service account. For more information, see Security considerations for Dataform permissions.

• dataform.repositories.list

• dataform.collections.create

• dataform.collections.list

• aiplatform.notebookRuntimeTemplates.apply

• aiplatform.notebookRuntimeTemplates.get

• aiplatform.notebookRuntimeTemplates.list

• aiplatform.notebookRuntimeTemplates.getIamPolicy

• aiplatform.notebookRuntimes.assign

• aiplatform.notebookRuntimes.get

• aiplatform.notebookRuntimes.list

• aiplatform.operations.list

You can get these permissions from the following IAM roles:

• BigQuery Read Session User (roles/bigquery.readSessionUser)

• BigQuery Studio User (roles/bigquery.studioUser)

  The BigQuery Studio User role combines the following IAM roles:

  • BigQuery Job User (roles/bigquery.jobUser)
  • Notebook Runtime User (roles/aiplatform.notebookRuntimeUser)

  • Code Creator (roles/dataform.codeCreator)

    Note: Users assigned the Code Creator role in a project can list the names of code assets in that project by using the Dataform API or the Dataform command-line interface (CLI). Non-administrators using BigQuery Studio can only see code assets that they created or that were shared with them.

Permissions to edit notebooks

To edit and run notebooks, you need the following IAM roles:

• BigQuery Job User (roles/bigquery.jobUser)
• BigQuery Read Session User (roles/bigquery.readSessionUser)
• Notebook Runtime User (roles/aiplatform.notebookRuntimeUser)
• Code Editor (roles/dataform.codeEditor)

Permissions to view notebooks

To view and run notebooks, you need the following IAM roles:

• BigQuery Job User (roles/bigquery.jobUser)
• BigQuery Read Session User (roles/bigquery.readSessionUser)
• Notebook Runtime User (roles/aiplatform.notebookRuntimeUser)
• Code Viewer (roles/dataform.codeViewer)

Create notebooks

Use the following sections to learn how to create a notebook.

Set the default region for code assets

If this is the first time you are creating a code asset, set the default region for code assets. You can't change the region for a code asset after it is created.

Note: If you create a saved query and choose a different default region than the one you have been using for code assets, for example choosing us-west1 when you have been using us-central1, then that query and all code assets you create afterwards use that new region by default. Existing code assets continue to use the region they were assigned when they were created.

Create a notebook from the BigQuery editor

To create a notebook containing a default query for a given table, follow these steps:

1. Go to the BigQuery page.

   Go to BigQuery

2. In the tab bar of the editor pane, click the arrow drop down next to the + sign and then click Create Python notebook.

   The new notebook opens, containing cells that show example queries against the bigquery-public-data.ml_datasets.penguins public dataset.

Create a notebook from a table

To create a notebook containing a default query for a specific table, follow these steps:

1. Go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and find the table that you want to query.

3. Click View actions next to the table, click Query in > Python notebook.

   The new notebook opens, containing cells that show example queries against the selected table.

Create a notebook to explore the result set of a query

To create a notebook to explore the result set of a query, follow these steps:

1. Go to the BigQuery page.

   Go to BigQuery

2. In the editor pane, run a query that generates a query result.

3. In the Query results pane, click Explore Data and then click Explore with Python notebook.

   The new notebook opens, containing cells with code to return the query SQL and the query results.

Create a notebook from an existing notebook

To open any version of an existing notebook as a new notebook, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Select a notebook.

3. Select the Activity pane.

4. Click View actions next to a notebook version and then click Open as new Python notebook.

   A copy of the notebook is opened as a new notebook.

Upload notebooks

You can upload a local notebook to use it in BigQuery Studio. The uploaded notebook is then visible in the BigQuery page of the Google Cloud console.

To upload a notebook, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project, and then do one of the following:

   • Next to Notebooks, click View actions > Upload to Notebooks.
   • Next to the Google Cloud project name, click View actions > Upload to project > Python notebook.

3. In the Upload Notebook dialog, in the Notebook field, click Browse, and then select the notebook that you want to upload.

4. Optional: In the Notebook name field, edit the name of the notebook.

5. In the Region field, select the region where you want to upload your notebook.

6. Click Upload.

Your notebook appears in the Explorer pane.

Connect to a runtime

Use the following sections to learn how to connect a notebook to a Vertex AI runtime. A runtime is a compute resource that runs the code in your notebook.

For more information about runtimes, see Runtimes and runtime templates.

Connect to the default runtime

The default runtime is a preset runtime that requires minimal setup.

To connect to the default runtime, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of a notebook to open it.

3. In the notebook, click Connect, or run any cell in the notebook.

   It might take several minutes to connect to the default runtime if you don't already have an active runtime.

Connect to a non-default runtime

If you want to use a runtime other than the default runtime, you must first create that additional runtime in Vertex AI.

To connect to non-default runtime, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of a notebook to open it.

3. In the notebook, click the drop-down next to Connect and then click Connect to a runtime.

4. Click Connect to an existing runtime.

5. In Runtimes, select the runtime to use.

6. Click Connect.

Connect to a new runtime

To connect to a new runtime, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of a notebook to open it.

3. In the notebook, click the drop-down next to Connect and then click Connect to a runtime.

4. Click Create new runtime.

5. In Runtime Template, select the Vertex AI runtime template to use.

6. In Runtime name, type a name for the runtime.

7. Click Connect.

Grant access to notebooks

To grant other users access to a notebook, add those users to an appropriate IAM role.

Important: Users who have access to the notebook can see all output generated by code in the notebook, even if this output contains data from tables they don't have access to. To prevent saved output from being shared, disable notebook output saving.

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Find the notebook that you want to grant access to.

3. Click View actions next to the notebook, and then click Share > Manage Permissions.

4. In the Manage permissions pane, click Add user/group.

5. In the New principals field, enter a principal.

6. In the Role list, select one of the following roles:

   • Code Owner: Can perform any action on the notebook, including deleting or sharing it.
   • Code Editor: Can edit the notebook.
   • Code Viewer: Can view the notebook.
   Note: The principal must also have the Notebook Runtime User (roles/aiplatform.notebookRuntimeUser) and BigQuery User (roles/bigquery.user) roles to run the notebook.

7. Optional: To view a complete list of roles and advanced sharing settings, click Advanced sharing.

8. Click Save.

9. To return to the notebook information page, click Close.

Share notebooks

To share a notebook with other users, you can generate and share a link to the notebook. For other users to see the notebook you share, you must first grant access to the notebook.

To run a notebook, users must have access to the data that the notebook accesses. For more information, see Grant access to a dataset.

Important: Users who have access to the notebook can see all output generated by code in the notebook, even if this output contains data from tables they don't have access to. To prevent saved output from being shared, disable notebook output saving.

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, expand the Shared notebooks folder. Find the notebook that you want to share.

3. Click View actions next to the notebook, and then click Share > Copy link.

4. Share the link with other users.

Disable notebook output saving

You can prevent sharing saved notebook output with other users who have access to the notebook file by disabling notebook output saving.

When you disable output saving for a selected notebook, BigQuery deletes all output saved in the notebook file and doesn't save the output of subsequent runs.

However, users who have access to the notebook can still view its output in the following ways:

• Run the notebook to view its current output. This output is not saved.
• View an archival version of the notebook and its output in revision history.

To disable saving of output for a selected notebook, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and then select the notebook for which you want to disable saving output.

3. Click Toggle header visibility > Edit > Notebook settings.

4. In the Notebook settings window, select Omit code cell output when saving this notebook.

5. Click Save.

6. Click Reload.

Resolve conflicts

If you and another user make conflicting changes in a notebook, the service raises the error Automatic saving failed. This file was updated remotely or in another tab. and provides a Show diff link. To resolve the conflict, follow these steps:

1. Click the Show diff link. The Review remote changes dialog opens.
2. Optional: To compare the notebook source code, select the Raw source checkbox.
3. Optional: To compare the versions inline instead of in separate panes, select the Inline diff checkbox.
4. Review the changes and decide which to keep, revising your input if necessary.
5. Click Save your changes.

Rename notebooks

To rename a notebook, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Find the notebook that you want to rename.

3. Click View actions next to the notebook, and then click Rename.

4. Type a name for the notebook, and then click Rename.

Troubleshooting

For more information, see Troubleshoot Colab Enterprise.

What's next

• Learn how to manage notebooks.
• Learn how to orchestrate notebooks.