Uploading a pipeline or component

Kubeflow pipelines are end-to-end machine learning (ML) workflows based on containers. Pipelines are built from components which are self-contained sets of code, packaged as container images, that perform a step in the ML workflow, such as preprocessing, data transformation, or training a model.

This guide describes how to upload a Kubeflow pipeline or component on AI Hub.

About uploading and sharing assets on AI Hub

You must sign in with a user account that is associated with a Cloud Identity or Google Workspace domain to upload and share assets on AI Hub. Assets that you upload to AI Hub are not shared by default. You can share these assets with individuals, groups, or your entire organization.

You are responsible for the assets that you upload on AI Hub. Follow your organization's guidelines and policies when uploading and sharing assets on AI Hub. Do not store credentials on AI Hub.

Prerequisites

To upload a pipeline or component on AI Hub, it must be created with Kubeflow 0.5 or higher. If you created your pipeline or component with a version of Kubeflow lower than 0.5, you must first upgrade Kubeflow before uploading your pipeline or component on AI Hub.

Prepare your pipeline or component for publication

Use the following instructions to prepare your pipeline or component before uploading it on AI Hub.

Pipeline

Before you can upload a pipeline on AI Hub, you must create the pipeline specification, and then use the Kubeflow Pipelines SDK to compile and compress it into a ZIP file.

Follow these steps to prepare your pipeline for the upload process:

  1. Use the Kubeflow Pipelines SDK to create the pipeline specification in the form of a Python program. See the Kubeflow Pipelines author's guide for details.
  2. Follow the Kubeflow Pipelines author's guide to compile the pipeline specification and compress it as a ZIP file.

Component

Before you can upload a pipeline on AI Hub, you must create the component specification as a YAML file and compress it into a ZIP file.

To learn more about creating Kubeflow pipeline components, read the Kubeflow Pipelines component specification.

Ensure your pipeline or component's containers are accessible

To ensure that the container images in your pipeline or component are accessible to the Kubeflow Pipelines system, one of the following must be true:

  • Your pipeline or component's containers must be publicly accessible.
  • Or the containers must be registered to the same Google Cloud project as the Kubeflow Pipelines clusters on deployment.
  • Or you must grant your Kubeflow Pipelines cluster's service account access to the pipeline or component's containers.

The recommended procedure is that you push your pipeline or component container images to a Container Registry location in a central Google Cloud project belonging to your organization.

For details on configuring the shared registry, see the guide to setting up a shared Container Registry.

Upload a pipeline or component on AI Hub

Follow this process to upload a pipeline or component on AI Hub:

  1. Go to AI Hub. Click Sign in in the upper-right corner of the window and follow the prompts to sign in with an account that is associated with your organization.
  2. Click Upload and select Kubeflow pipeline from the list. A form opens, prompting you to enter information about the pipeline or component that you want to upload.

  3. Supply the following information in the Metadata section:

    • To publish this asset as a component, select Publish as a Kubeflow pipeline component.
    • Pipeline name: A short description that helps you identify this asset on AI Hub.
    • Input data type: Select the types of data that your pipeline or component processes. This metadata is useful for people who search AI Hub for assets that work for their type of data.
    • ML Workflow: (Optional.) Select the steps in the ML workflow that characterize your pipeline or component. This metadata is useful for people who search AI Hub for assets that implement a specific ML workflow step.
    • Technique: (Optional.) Select the ML methodologies that were used to implement this pipeline or component. This metadata is useful for people who search AI Hub for assets that implement a specific ML technique.
    • Use case: (Optional.) Select the scenarios that characterize your pipeline or component. This metadata is useful for people who search AI Hub for assets that fulfill a specific use case.
    • Labels: (Optional.) Enter key terms that characterize your pipeline or component. This metadata is useful for people who search AI Hub for assets with a specific label.
    • Summary: A description of the functionality that this asset implements.

  4. Click Next.

  5. Supply the following information in the Version section:

    • Version: This field offers a way for you to name and manage the versions of your pipeline or component. A good first version number is: 1.
    • Pipeline .zip file: Click Upload and follow the prompts to upload the your-pipeline.zip file which you created in an earlier step of this guide.
    • Kubeflow version: (Optional.) Enter the version of Kubeflow that you used to create this pipeline or component. This information is useful for people who want to use this asset in their Kubeflow Pipelines environment.
    • Kubeflow Pipelines version: (Optional.) Enter the version of Kubeflow Pipelines that you used to create this pipeline or component. This information is useful for people who want to use this asset in their Kubeflow Pipelines environment.

  6. To share a link to your pipeline or component's source code, click Add URL under Source Code (optional). Supply the following for each link that you add:

    • URL: The location of your pipeline or component's source code.
    • Description: (Optional.) A description of the source code repository at the URL you specified.

  7. Click Next.

  8. Enter detailed information about your pipeline or component in the Documentation section. Follow the instructions in the documentation template to describe your pipeline or component. You can use Markdown to format your documentation.

  9. Click the Preview Changes tab to see the documentation rendered as HTML.

  10. Click Next.

  11. Check the information on the Review and finish section. If you need to change something, click one of the other sections and update the information, then return to the Review and finish section.

  12. Click Publish to upload the pipeline or component to your private assets on AI Hub.

Update a shared pipeline or component

Follow this process to update a shared pipeline's metadata, or to add a new version of a pipeline:

  1. Go to AI Hub.
  2. Click My assets. A page opens, listing the assets that you have shared with your organization.
  3. Find the pipeline or component that you would like to update, then click the asset's name. A form opens, prompting you to enter information about the pipeline or component.

  4. Update the information in the Metadata section:

    • To publish this asset as a component, select Publish as a Kubeflow pipeline component.
    • Pipeline name: A short description that helps you identify this asset on AI Hub.
    • Input data type: Select the types of data that your pipeline or component processes. This metadata is useful for people who search AI Hub for assets that work for their type of data.
    • ML Workflow: (Optional.) Select the steps in the ML workflow that characterize your pipeline or component. This metadata is useful for people who search AI Hub for assets that implement a specific ML workflow step.
    • Technique: (Optional.) Select the ML methodologies that were used to implement this pipeline or component. This metadata is useful for people who search AI Hub for assets that implement a specific ML technique.
    • Use case: (Optional.) Select the scenarios that characterize your pipeline or component. This metadata is useful for people who search AI Hub for assets that fulfill a specific use case.
    • Labels: (Optional.) Enter key terms that characterize your pipeline or component. Use commas to separate terms. This metadata is useful for people who search AI Hub for assets with a specific label.
    • Summary: A description of the functionality that this asset implements.

  5. Click Next. The Version section lists the published versions of this pipeline or component.

  6. To add a new version of this pipeline or component, click Add New Version and supply the following information:

    • Version: This field offers a way for you to name and manage the versions of your asset. This field is preset to your asset's next version number.
    • Pipeline .zip file: Click Upload and follow the prompts to upload your pipeline or component's new your-pipeline.zip file, which you created in an earlier step of this guide.
    • Kubeflow version: (Optional.) Enter the version of Kubeflow that you used to create this pipeline or component. This information is useful for people who want to use this asset in their Kubeflow Pipelines environment.
    • Kubeflow Pipelines version: (Optional.) Enter the version of Kubeflow Pipelines that you used to create this pipeline or component. This information is useful for people who want to use this asset in their Kubeflow Pipelines environment.

    • Review and update the links to the pipeline or component's source code.

      • To add a link to your pipeline or component's source code, click Add URL under Source Code (optional). Supply the following for each source code link:

        • URL: The location of the pipeline or component's source code.
        • Description: (Optional.) A description of the source code repository at the URL you specified.

      • To remove a link, click the icon in the Source Code URL pane of the link that you want to remove.

  7. Click Next.

  8. Review and update the Documentation for your pipeline or component. You can use Markdown to format your documentation.

  9. Click the Preview Changes tab to see the documentation rendered as HTML.

  10. Click Next.

  11. Check the information in the Review and finish section. If you need to change something, click one of the other sections and update the information, then return to the Review and finish section.

  12. Click Publish to save your changes.

What's next