Connecting to AI Platform Pipelines using the Kubeflow Pipelines SDK

You can use the Kubeflow Pipelines SDK to manage and automate activity on your AI Platform Pipelines cluster, such as:

  • Loading pipelines to your AI Platform Pipelines cluster
  • Running pipelines
  • Retrieving information about pipelines and runs

This guide describes how to connect to your AI Platform Pipelines cluster using the Kubeflow Pipelines SDK.

Before you start

Before you connect to your cluster using the Kubeflow Pipelines SDK, you must first set up your AI Platform Pipelines cluster, install the Kubeflow Pipelines SDK and ensure that you have sufficient permissions to access your AI Platform Pipelines cluster.

  1. Follow the guide to setting up your AI Platform Pipelines cluster.

  2. Install or upgrade the Kubeflow Pipelines SDK to version 0.5.2, 1.0.4, or later.

  3. To access an AI Platform Pipelines cluster using the Kubeflow Pipelines SDK, you must have the Service Account User Role for the Google Kubernetes Engine cluster's service account.

    Learn more about granting a user access to AI Platform Pipelines.

Using the Kubeflow Pipelines SDK to connect to an AI Platform Pipelines cluster

Use the following instructions to connect to your AI Platform Pipelines cluster using the Kubeflow Pipelines SDK.

  1. Open AI Platform Pipelines in the Google Cloud console.

    Go to AI Platform Pipelines

  2. Click Open pipelines dashboard for your Kubeflow Pipelines cluster. The Kubeflow Pipelines user interface opens in a new tab.

  3. Find the hostname and URL scheme in the URL of the Kubeflow Pipelines dashboard. The hostname and scheme are the portion of the URL between the beginning of the URL and /#/start. The hostname and scheme should match the pattern https://*.pipelines.googleusercontent.com.

  4. Use the Kubeflow Pipelines SDK to connect to your AI Platform Pipelines cluster from a Jupyter notebook or Python client.

    import kfp
client = kfp.Client(host='https://example.com')

    Replace https://example.com with the hostname and scheme for your cluster.

  5. Now you can use the Kubeflow Pipelines SDK to interact with your cluster. For example, the following command lists your cluster's pipelines.

    client.list_pipelines()

What's next