Set up an ML pipeline by using AI Platform Pipelines

Set up an ML pipeline by using AI Platform Pipelines

Beta

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This quickstart provides a brief introduction to AI Platform Pipelines. In this guide, you install Kubeflow Pipelines with TensorFlow Extended on a new Google Kubernetes Engine (GKE) cluster, then run an example pipeline.

This topic is intended for users who are new to AI Platform Pipelines.

Before you begin

Before following this guide, check that your Google Cloud project is correctly set up and that you have sufficient permissions to deploy AI Platform Pipelines.

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

5. Make sure that billing is enabled for your Google Cloud project.

6. Use the following instructions to check if you have been granted the roles required to deploy AI Platform Pipelines.

   1. Open a Cloud Shell session.

      Open Cloud Shell

      Cloud Shell opens in a frame at the bottom of the Google Cloud console.

   2. You must have the Viewer (roles/viewer) and Kubernetes Engine Admin (roles/container.admin) roles on the project, or other roles that include the same permissions such as the Owner (roles/owner) role on the project, to deploy AI Platform Pipelines. Run the following command in Cloud Shell to list the principals that have the Viewer and Kubernetes Engine Admin roles.

      gcloud projects get-iam-policy PROJECT_ID \
        --flatten="bindings[].members" --format="table(bindings.role, bindings.members)" \
        --filter="bindings.role:roles/container.admin OR bindings.role:roles/viewer"

      Replace PROJECT_ID with the ID of your Google Cloud project.

      Use the output of this command to verify that your account has the Viewer and Kubernetes Engine Admin roles.

   3. If you have not been granted the required roles, contact your Google Cloud project administrator for additional help.

      Learn more about granting Identity and Access Management roles.

Set up your AI Platform Pipelines instance

Use the following instructions to set up AI Platform Pipelines on a new GKE cluster.

1. Open AI Platform Pipelines in the Google Cloud console.

   Go to AI Platform Pipelines

2. Click Select project. A dialog prompting you to select a Google Cloud project appears.

3. Select the Google Cloud project you want to use for this quickstart, then click Open.

4. In the AI Platform Pipelines toolbar, click New instance. Kubeflow Pipelines opens in Google Cloud Marketplace.

5. Click Configure. A form opens for you to configure your Kubeflow Pipelines deployment.

6. If the Create a new cluster link is displayed, click Create a new cluster. Otherwise, continue to the next step.

   

7. Select us-central1-a as the Cluster zone where your GKE cluster should be created.

8. Check Allow access to the following Cloud APIs to grant applications that run on your GKE cluster access to Google Cloud resources. By checking this box, you are granting your cluster access to the https://www.googleapis.com/auth/cloud-platform access scope. This access scope provides full access to the Google Cloud resources that you have enabled in your project. Granting your cluster access to Google Cloud resources in this manner saves you the effort of creating a Kubernetes secret.

   

9. Click Create cluster to create your GKE cluster. This process takes several minutes to complete.

10. After your cluster has been created, supply the following information:

    • Namespace: Select default as the namespace.
    • App instance name: Enter pipelines-quickstart as the instance name.

11. Click Deploy to deploy Kubeflow Pipelines onto your new GKE cluster.

The deployment process takes several minutes to complete. After the deployment process is finished, continue to the next section.

Run an example pipeline

Use the following instructions to run an example pipeline in your new AI Platform Pipelines instance.

1. Open AI Platform Pipelines in the Google Cloud console.

   Go to AI Platform Pipelines

2. Find the AI Platform Pipelines cluster named pipelines-quickstart, then click Open pipelines dashboard to open Kubeflow Pipelines. The Kubeflow Pipelines dashboard opens, displaying the Getting Started page.

   

3. You can use the Getting Started page in the dashboard to learn more about the demonstration and tutorial pipelines that are provided with Kubeflow Pipelines, or learn more about how to create a pipeline.

   

   In the left navigation panel, click Pipelines.

4. Kubeflow Pipelines provides several example pipelines. Click [Tutorial] Data passing in python components. A graph displaying the steps in the pipeline opens.

5. To run the pipeline once, click Create run. In Kubeflow Pipelines, a run is a single execution of a pipeline. A form where you enter the run details opens.

6. Enter "Quickstart pipeline run" as the Run name.

7. Click Start. The pipelines dashboard displays a list of pipeline runs on the Experiments page. In Kubeflow Pipelines, you can use experiments to organize your pipeline runs into logical groups.

8. Click the run named Quickstart pipeline run. The graph of your run is displayed. While your run is still in progress, the graph changes as each step executes.

9. Click the pipeline steps to explore your run's inputs, outputs, logs, etc.

You have now run an example pipeline in your AI Platform Pipelines instance.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used on this page, follow these steps.

To avoid incurring more charges to your Google Cloud account, use the following instructions to delete the AI Platform Pipelines instance and GKE cluster that you created in the preceding sections.

1. Open AI Platform Pipelines in the Google Cloud console.

   Go to AI Platform Pipelines

2. Select the checkbox for the AI Platform Pipelines instance named pipelines-quickstart.

3. In the AI Platform Pipelines toolbar, click Delete.

4. In the Delete Kubeflow Pipelines from cluster dialog, select the Delete cluster checkbox. Selecting this checkbox indicates that you want to delete the GKE cluster you created for this quickstart.

5. Click Delete to delete your AI Platform Pipelines instance and GKE cluster.

What's next

• Learn more about AI Platform Pipelines and ML pipelines by reading the introduction to AI Platform Pipelines.
• Orchestrate your ML process as a pipeline.