Using the Python Client Library

This tutorial describes how to use the Google API client library to easily make calls to the Cloud Machine Learning Engine REST APIs in your Python applications. The code snippets and examples in the rest of this documentation use the Python client library.

You will be creating a model in your Google Cloud Platform project in this tutorial. That is a simple task that's easily contained to a small example. You can learn more about managing models in the model and job management overview.

Objectives

This is a basic tutorial designed to familiarize you with the Python client library. When you're finished you should be able to:

  • Use application default credentials in your Python applications.
  • Get a Python representation of the Cloud ML Engine services.
  • Use that representation to create a model in your project, which should help you understand how to call the other model and job management APIs.

Costs

You will not be charged for the operations in this tutorial. Refer to the pricing page for more information.

Before you begin

Complete the following steps to set up a GCP account with the Cloud ML Engine API activated, install and activate the Cloud SDK, and install and test TensorFlow.

Set up your GCP project

  1. Sign in to your Google account.

    If you don't already have one, sign up for a new account.

  2. Select or create a Cloud Platform project.

    Go to the Manage resources page

  3. Enable billing for your project.

    Enable billing

  4. Enable the Cloud Machine Learning Engine and Compute Engine APIs.

    Enable the APIs

Set up your environment

For macOS users, we recommend that you set up your environment using the MACOS tab below. Cloud Shell, shown on the CLOUD SHELL tab, is available on macOS, Linux, and Windows. Cloud Shell provides a quick way to try Cloud Machine Learning Engine, but isn’t suitable for ongoing development work.

macOS

  1. Install a virtual environment to create an isolated Python development environment for this guide. For example, the following commands install virtualenv and activate an environment named cmle-env.

    virtualenv cmle-env
    source cmle-env/bin/activate
  2. Confirm that you have Python installed and, if necessary, install it.

    python -V
  3. Install the Google Cloud SDK for macOS by following the instructions in the before you begin section of the Cloud SDK Quickstart.

  4. Initialize the Cloud SDK.

    gcloud init

    For more information about the initialization process, see the Cloud SDK documentation.

  5. Install pip, Python’s package manager. Check if you already have pip installed by running pip --version. Ensure that you have the latest version of pip and if not, upgrade it using the following command:

    pip install -U pip
  6. Install TensorFlow:

    pip install --upgrade tensorflow

    For more information about installing TensorFlow, see the TensorFlow documentation.

  7. Create application default credentials:

    gcloud auth application-default login

Cloud Shell

  1. Open the Google Cloud Platform Console.

    Google Cloud Platform Console

  2. Click the Activate Google Cloud Shell button at the top of the console window.

    Activate Google Cloud Shell

    A Cloud Shell session opens inside a new frame at the bottom of the console and displays a command-line prompt. It can take a few seconds for the shell session to be initialized.

    Cloud Shell session

    Your Cloud Shell session is ready to use.

  3. Configure the gcloud command-line tool to use your selected project.

            gcloud config set project [selected-project-id]
            

    where [selected-project-id] is your project ID. (Omit the enclosing brackets.)

Verify the Google Cloud SDK components

To verify that the Google Cloud SDK components are installed:

  1. List your models:

    gcloud ml-engine models list
  2. Verify that the command returns an empty list:

    Listed 0 items.

    After you start creating models, you can see them listed by using this command.

Run a simple TensorFlow Python program

Run a simple Python program to test your installation of TensorFlow. If you are using Cloud Shell, note that TensorFlow is already installed.

  1. Start a Python interactive shell.

    python
  2. Import TensorFlow.

    >>> import tensorflow as tf
  3. Create a constant that contains a string.

    >>> hello = tf.constant('Hello, TensorFlow!')
  4. Create a TensorFlow session.

    >>> sess = tf.Session()

    You can ignore the warnings that the TensorFlow library wasn't compiled to use certain instructions.

  5. Display the value of hello.

    >>> print(sess.run(hello))

    If successful, the system outputs:

    Hello, TensorFlow!
  6. Stop the Python interactive shell.

    >>> exit()

For more information about installing TensorFlow, go to Installing TensorFlow on the TensorFlow site.

Install the Google Python library

Install the Google APIs Client Library for Python.

Importing the required modules

When you want to use the API client library to use the Cloud ML Engine REST APIs in your Python code, you must import its package and the OAuth2 package. For this tutorial (and for most standard uses of Cloud ML Engine) you only need to import specific modules from both packages: GoogleCredentials from OAuth2 and discovery from Google API Client Library. Refer to the documentation for those packages to learn about the other available modules.

Create a new Python file using your favorite editor, and add these lines:

from oauth2client.client import GoogleCredentials
from googleapiclient import discovery

Getting your application credentials

You need credentials to access the APIs for your project. When you initialized the gcloud command-line tool during setup, you entered your user identification and selected a project to work on. You'll use those credentials, called application default credentials, for this tutorial, because they are the easiest to use. Real-world applications often require more complicated auth arrangements.

Here's what to add to your code:

credentials = GoogleCredentials.get_application_default()

Building a Python representation of the API

Now that you have your application default credentials, you can get your Python representation of the REST API. The method you call is build because the API client library uses service discovery to dynamically set up connections to the services as they exist when you make the call. Call your object that encapsulates the services ml and use the credentials object you just got:

ml = discovery.build('ml','v1', credentials=credentials)

Configuring your parameters and request body

To make a call to a service, you must create the parameters and request body that will be passed to the REST API. You pass parameters as regular Python parameters to the method that represents the call. The body is a JSON resource just as you would use if calling the API with an HTTP request directly.

Take a look at the REST API for creating a model in a new browser tab, projects.models.create:

  • Notice the path parameter parent, which is the part of the URI of the request that identifies the project. If you were making the HTTP POST request directly, you would use the following URI:

    https://ml.googleapis.com/v1/projects/your_project_ID/models
    

    When using the API client library, the variable part of the URI is represented as a string-typed parameter to the API call. You'll set it to 'projects/<var>your_project_ID</var>'. Store your project in a variable to make API calls cleaner:

    projectID = 'projects/{}'.format('your_project_ID')
    

  • The body of the request is a JSON resource representing the model information. You can see in the model resource definition that it has two values for input: name and (optionally) description. You can pass a Python dictionary in the place of JSON and the API client library will perform the necessary conversion.

    Create your Python dictionary:

    requestDict = {'name': 'your-model-name',
                   'description': 'This is a machine learning model entry.'}
    

Creating your request

Making calls to APIs with the Python client library has two steps: first you create a request, then you make the call using that request.

Create the request

Use the built client objects that you created earlier (if you followed the code snippet exactly, it's called ml) as the root of the API hierarchy and specify the API you want to use. Each collection in the API path behaves like a function that returns a list of the collections and methods within it. For example, the root of all the Cloud ML Engine APIs is projects, so your call begins with ml.projects().

Use this code to form your request:

request = ml.projects().models().create(parent=projectID, body=requestDict)

Send the request

The request that you constructed in the last step exposes an execute method that you call to send the request to the service:

response = request.execute()

It's common for developers to combine this step with the last one:

response = ml.projects().models().create(parent=projectID,
                                         body=requestDict).execute()

Handle simple errors

A lot of things can go wrong when you make API calls over the Internet. It's a good idea to handle common errors. The simplest way to deal with errors is to put your request in a try block and catch likely errors. Most of the errors you're likely to get from the service are HTTP errors, which are encapsulated in the HttpError class. To catch these errors, you first need to get the errors module from the googleapiclient package. Add this statement with the rest of your imports.

from googleapiclient import errors

Then wrap your request.execute() call in a try block. Also put a print statement in the block, so that you will try to print the response only if the call succeeds:

try:
    response = request.execute()
    print(response)

Finally, add a catch block to handle HTTP errors. You can use HttpError._get_reason() to get the reason text fields from the response:

except errors.HttpError, err:
    # Something went wrong, print out some information.
    print('There was an error creating the model. Check the details:')
    print(err._get_reason())

Of course, a simple print statement might not be the right approach for your application.

Putting it all together

Here is the complete example:

from oauth2client.client import GoogleCredentials
from googleapiclient import discovery
from googleapiclient import errors

# Store your full project ID in a variable in the format the API needs.
projectID = 'projects/{}'.format('your_project_ID')

# Get application default credentials (possible only if the gcloud tool is
#  configured on your machine).
credentials = GoogleCredentials.get_application_default()

# Build a representation of the Cloud ML API.
ml = discovery.build('ml', 'v1', credentials=credentials)

# Create a dictionary with the fields from the request body.
requestDict = {'name': 'your_model_name',
               'description': 'your_model_description'}

# Create a request to call projects.models.create.
request = ml.projects().models().create(
              parent=projectID, body=requestDict)

# Make the call.
try:
    response = request.execute()
    print(response)
except errors.HttpError, err:
    # Something went wrong, print out some information.
    print('There was an error creating the model. Check the details:')
    print(err._get_reason())

Generalizing to other methods

You can use the procedure you learned here to make any of the other REST API calls. Some of the APIs require much more complicated JSON resources than creating a model does, but the principles are the same:

  1. Import oauth2client.client.GoogleCredentials, googleapiclient.discovery, and googleapiclient.errors.

  2. Get credentials appropriate to your application (the simple application default credentials you used here provide an easy auth solution while you experiment with the API).

  3. Use the discovery module to build a Python representation of the API.

  4. Use the API representation as a series of nested objects to get to the API you want and create a request. For example,

    projects().models().versions().delete(
        name='projects/myproject/models/mymodel/versions/myversion')
    

  5. Call request.execute() to send the request, handling exceptions in an appropriate way for your application.

  6. When there is a response body, you can treat it like a Python dictionary to get at the JSON objects specified in the API reference. Note that many of the objects in responses have fields that are present only in some circumstances. You should always check to avoid key errors:

    response = request.execute()

    some_value = response.get('some_key') or 'default_value'

What's next

Send feedback about...

Cloud Machine Learning Engine (Cloud ML Engine)