Get started with Vertex AI TensorBoard

Open source TensorBoard (TB) is a Google open source project for machine learning experiment visualization. Vertex AI TensorBoard is an enterprise-ready managed version of TensorBoard.

Vertex AI TensorBoard provides various detailed visualizations, that includes:

• Tracking and visualizing metrics such as loss and accuracy over time
• Visualizing model computational graphs (ops and layers)
• Viewing histograms of weights, biases, or other tensors as they change over time
• Projecting embeddings to a lower dimensional space
• Displaying image, text, and audio samples

In addition to the powerful visualizations from TensorBoard, Vertex AI TensorBoard provides:

• A persistent, shareable link to your experiment's dashboard

• A searchable list of all experiments in a project

• Tight integrations with Vertex AI services for model training

• Enterprise-grade security, privacy, and compliance

With Vertex AI TensorBoard, you can track, visualize, and compare ML experiments and share them with your team.

Currently, Vertex AI TensorBoard charges a monthly fee of $300 per unique active user. Beginning in August 2023, Vertex AI TensorBoard will change from a per-user monthly license of $300 per month to $10 GiB per month for storage of logs and metrics. This means there are no more subscription fees, and you will only pay for the storage you've used. See the Vertex AI TensorBoard: Delete Outdated TensorBoard Experiments tutorial for how to manage storage.

Get Started

A Vertex AI TensorBoard instance, which is a regionalized resource storing your Vertex AI TensorBoard experiments, must be created before the experiments can be visualized. You can create multiple instances in a project.

Create a Vertex AI TensorBoard instance

Vertex AI SDK for Python

Create Vertex AI TensorBoard instance using the Vertex AI SDK for Python.

Vertex AI SDK for Python

View on GitHub Feedback

def create_tensorboard_sample(
    project: str,
    display_name: str,
    location: str,
):
    aiplatform.init(project=project, location=location)

    tensorboard = aiplatform.Tensorboard.create(
        display_name=display_name,
        project=project,
        location=location,
    )

    print(tensorboard.display_name)
    print(tensorboard.resource_name)
    return tensorboard

• project: Your project ID. You can find these IDs in the Google Cloud console welcome page.
  
• display_name: Provide a name for your TensorBoard instance.
• location: See List of available locations Be sure to use a region that supports TensorBoard if creating a TensorBoard instance.

Google Cloud console

Follow these steps to create a Vertex AI TensorBoard instance using the Google Cloud console.

1. If you're new to Vertex AI or starting a new project, set up your project and development environment.
2. In the Vertex AI section of the Google Cloud console, go to the Experiments page.
   
   Go to the Experiments page.
3. Navigate to the TensorBoard Instances tab.
4. Click Create at the top of the page.
5. Select a region from the Region drop-down list.
6. Add a description. (optional)
7. Click Create to create your TensorBoard instance

gcloud CLI

Use Google Cloud CLI to create a Vertex AI TensorBoard instance.

1. Install the gcloud CLI
2. Initialize the Google Cloud CLI by running gcloud init.
3. To confirm installation, explore the commands.
   

    gcloud ai tensorboards --help 

   
   The commands include create, describe, list, and delete. If needed, you can follow these steps to set default values for your project and region before proceeding.
   Now, you can create a Vertex AI TensorBoard instance.
4. Authenticate to the gcloud CLI.
   

   gcloud auth application-default login

5. Create a Vertex AI TensorBoard instance by providing a project name and a display name. This step might take a few minutes to complete for the first time in a project. Make note of the Vertex AI TensorBoard instance name (for example: projects/123/locations/us-central1/tensorboards/456) that is printed at the end of the following command. You will need it in the later steps.
   

   gcloud ai tensorboards create --display-name DISPLAY_NAME \
        --project PROJECT_NAME
      

   
   Replace the following:
   
   • PROJECT_NAME: the project that you want to create the TensorBoard instance in
   • DISPLAY_NAME: a descriptive name for the TensorBoard instance

Upload TensorBoard logs

Vertex AI TensorBoard offers Google Cloud CLI and Vertex AI SDK for Python for uploading TensorBoard logs. You can upload logs from any environment which can connect to Google Cloud.

Create a virtual environment (optional)

Optional, but recommended first step:

1. Create a dedicated virtual environment to install the Vertex AI TensorBoard Uploader Python CLI.

python3 -m venv PATH/TO/VIRTUAL/ENVIRONMENT
source PATH/TO/VIRTUAL/ENVIRONMENT/bin/activate

1. Replace PATH/TO/VIRTUAL/ENVIRONMENT with the path to your dedicated virtual environment.

Install the Vertex AI TensorBoard package through Vertex AI SDK

The uploader needs the latest version of pip in order to be installed properly.

pip install -U pip
pip install google-cloud-aiplatform[tensorboard]

Upload Vertex AI TensorBoard logs

Note: Don't upload an entire Cloud Storage bucket or Cloud Storage folder with many nested folders from multiple training runs into one experiment. Best practice is to upload training logs belonging to one training, evaluation, or hyperparameter tuning sweep into one Vertex AI TensorBoard experiment in the cloud.

Vertex AI SDK for Python

Continuous monitoring

Call aiplatform.start_upload_tb_log at the beginning of the training. The SDK opens a new thread for uploading. This thread keeps monitoring when there's new data in the directory. When training completes, call end_upload_tb_log which kills the uploader thread. If end_upload_tb_log isn't called, a time out plan closes the thread.

Vertex AI SDK for Python

View on GitHub Feedback

def upload_tensorboard_log_continuously_sample(
    tensorboard_experiment_name: str,
    logdir: str,
    tensorboard_id: str,
    project: str,
    location: str,
    experiment_display_name: Optional[str] = None,
    run_name_prefix: Optional[str] = None,
    description: Optional[str] = None,
) -> None:

    aiplatform.init(project=project, location=location)

    # Continuous monitoring
    aiplatform.start_upload_tb_log(
        tensorboard_id=tensorboard_id,
        tensorboard_experiment_name=tensorboard_experiment_name,
        logdir=logdir,
        experiment_display_name=experiment_display_name,
        run_name_prefix=run_name_prefix,
        description=description,
    )
    aiplatform.end_upload_tb_log()

• project: Your project ID. You can find these Project IDs in the Google Cloud console welcome page.
• location: See List of available locations
• tensorboard_id:
  • To find the TENSORBOARD_ID, go to the Experiments page Vertex AI section of the Google Cloud console, and then select the TensorBoard Instances tab.
    Go to the Experiments page
    
    
  If not set, the tensorboard_id in aiplatform.init is used.
• logdir: The location of the TensorBoard logs that reside either in your local file system or Cloud Storage.
• experiment_display_name: The display name of the experiment.
• run_name_prefix: If present, all runs created by this invocation will have their name prefixed by this value.
• description: A string description to assign to the experiment.

One time logging

Calling aiplatform.upload_tb_log to perform a one-time uploading of TensorBoard logs. It uploads only the existing data in the logdir and then returns immediately.

Vertex AI SDK for Python

View on GitHub Feedback

def upload_tensorboard_log_one_time_sample(
    tensorboard_experiment_name: str,
    logdir: str,
    tensorboard_id: str,
    project: str,
    location: str,
    experiment_display_name: Optional[str] = None,
    run_name_prefix: Optional[str] = None,
    description: Optional[str] = None,
    verbosity: Optional[int] = 1,
) -> None:

    aiplatform.init(project=project, location=location)

    # one time upload
    aiplatform.upload_tb_log(
        tensorboard_id=tensorboard_id,
        tensorboard_experiment_name=tensorboard_experiment_name,
        logdir=logdir,
        experiment_display_name=experiment_display_name,
        run_name_prefix=run_name_prefix,
        description=description,
    )

• project: Your project ID. You can find these Project IDs in the Google Cloud console welcome page.
• location: See List of available locations
• tensorboard_id:
  • To find the TENSORBOARD_ID, go to the Experiments page Vertex AI section of the Google Cloud console, and then select the TensorBoard Instances tab.
    Go to the Experiments page
    
    
  If not set, the tensorboard_id in aiplatform.init is used.
• logdir: The location of the TensorBoard logs that reside either in your local file system or Cloud Storage.
• experiment_display_name: The display name of the experiment.
• run_name_prefix: If present, all runs created by this invocation will have their name prefixed by this value.
• description: A string description to assign to the experiment.

gcloud CLI

tb-gcp-uploader --tensorboard_resource_name \
  TENSORBOARD_INSTANCE_NAME \
  --logdir=LOG_DIR \
  --experiment_name=TB_EXPERIMENT_NAME --one_shot=True

• TENSORBOARD_INSTANCE_NAME: there are two ways to identify the instance name:
  • The full name is printed at the end of the gcloud ai tensorboards create command that you used previously.
  • If the TensorBoard Instance was created using the Google Cloud console, the TENSORBOARD_INSTANCE_NAME is projects/PROJECT_ID_OR_NUMBER/locations/REGION/tensorboards/TENSORBOARD_INSTANCE_ID
  • To find the TENSORBOARD_INSTANCE_ID, go to the Experiments page Vertex AI section of the Google Cloud console, and then select the TensorBoard Instances tab.
    Go to the Experiments page
    
    
• LOG_DIR: the location of the event logs that resides either in the local file system or Cloud Storage
• TB_EXPERIMENT_NAME: the name of the experiment, for example test-experiment. The experiment name should be unique within a TensorBoard resource

The uploader CLI by default runs indefinitely, monitoring changes in the LOG_DIR, and uploads newly added logs. --one_shot=True disables the behavior. Run tb-gcp-uploader --help for more information.

View a Vertex AI TensorBoard experiment

In order to access the Vertex AI TensorBoard page in the Google Cloud console, you'll need the IAM role "Vertex AI TensorBoard Web App User". The IAM Administrator of the project can grant such access. Users with the Vertex AI Administrator role also have access.

Use the Google Cloud console

You can view your Vertex AI TensorBoard experiment from the Google Cloud console with the following steps.

1. In the Vertex AI section of the Google Cloud console, go to the Experiments page.

   Go to the Experiments page

2. From the experiments tab, scroll or filter the experiments list to find your experiment.

3. To open the Vertex AI TensorBoard page, click Open TensorBoard next to your experiment.

View appears. You can share the link to this view with others who have permission.

Alternatively, if you use Vertex AI TensorBoard with custom training, select the training job from the Training page. An Open TensorBoard button appears at the top of the Training Job details page.

Use the link from CLI output

In addition, when using the Vertex AI TensorBoard package, the CLI outputs a link to the Vertex AI TensorBoard instance in the first few lines of the log where you can view your experiment. For example: View your TensorBoard at https://us-central1.tensorboard.googleusercontent.com/experiment/projects+123+locations+us-central1+tensorboards+4567+experiments+my-experiment-name

Notebooks

To learn more, run the following Jupyter notebooks in the environment of your choice:

• "Vertex AI TensorBoard: Delete Outdated TensorBoard Experiments":

  Run in Colab | Open in Vertex AI Workbench user-managed notebooks | View on GitHub

• "Vertex AI TensorBoard integration with Vertex AI Pipelines":

  Run in Colab | Open in Vertex AI Workbench user-managed notebooks | View on GitHub

What's next

• Check out the TensorBoard API documentation.

• See Using Vertex AI TensorBoard with custom training

• See Profile model training performance using Profiler