Serving Machine Learning Models Using Apigee Edge and AI Platform

This tutorial teaches you how to set up Apigee Edge with AI Platform Prediction.

When serving machine learning models, data preparation and model training are just two factors to consider. You also need to look at the following:

  • Serving predictions with scalability, performance, and availability in mind.
  • Restricting access to functionality depending on who the user is—for example, an internal user in your company or a user from an external partner organization.
  • Enforcing quotas and usage policies using API keys.
  • Managing the lifecycle of API keys, in particular distribution and revocation.
  • Granting access to your ML APIs through a manual or automated approval process.
  • Rolling out new model versions as machine learning models are updated.

Google Cloud provides managed services for model hosting and API management for accessing the prediction features of your hosted models.

AI Platform is a managed scalable machine learning service available on Google Cloud. In addition to training using Google's distributed infrastructure, AI Platform provides model hosting to serve large-scale online and batch predictions. The underlying infrastructure leverages Google's global load balancing to provide high-performance predictions to users worldwide.

For low-latency use cases, AI Platform online prediction enables predictions for your hosted models by exposing a REST API that returns predictions directly in the response. The serving infrastructure to host your model scales automatically according to demand, all without the need to maintain complex configurations or manage infrastructure provisioning. AI Platform also simplifies the process of updating prediction models by providing model versioning features.

To enable wider API management features for your service, you can use AI Platform online prediction in conjunction with Apigee Edge to enable the following:

  • Granular security and access control policies for your API.
  • API analytics for understanding usage.
  • API rate limiting and other quota enforcement.
  • Third-party access to your API through developer keys.
  • Monetization of your AI Platform API hosted in AI Platform online prediction.
  • Transparent model changes by directing prediction requests through an API facade.

Objectives

  • Deploy API proxies using Apigee Edge for your AI Platform–hosted models.
  • Secure access to your hosted models using API keys.
  • Specify manual and automatic approval processes for API key distribution.
  • Create API products to enforce different usage policies, such as time-based quotas according to a subscriber type.

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

In addition to the Google Cloud billable components, this tutorial uses Apigee Edge.

Before you begin

Create a Google Cloud project

  1. Sign in to your Google Account.

    If you don't already have one, sign up for a new account.

  2. In the Cloud Console, on the project selector page, select or create a Cloud project.

    Go to the project selector page

  3. Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

  4. Enable the AI Platform API.

    Enable the API

If you don't have the Cloud SDK installed, use Cloud Shell to deploy the application. Cloud Shell provides a shell environment with the Cloud SDK preinstalled. The rest of this tutorial uses Cloud Shell.

Create an Apigee Edge account

Follow the steps in Creating an Apigee Edge Account. Apigee provides an evaluation account at no cost to test drive the solution, which you can use to perform the steps in this tutorial.

Solution architecture

In this tutorial, you deploy TensorFlow models to AI Platform and use online prediction to make them available for serving. For each hosted model, you deploy an API proxy to Apigee Edge, which uses a service account to authenticate requests from client applications. Apigee Edge then forwards the requests to AI Platform online prediction. Apigee Edge enforces API policies such as usage quotas, rate limiting, and API key verification, and can further enforce policy variations, such as quota variations by API key, by using API products that you configure. The following diagram illustrates this architecture.

Deploying TensorFlow models to AI Platform and using online
prediction to make them available for serving

The following table explains the diagram concepts in further detail:

Concept Description
Application An application calls predictions on your machine learning model through an API hosted in Apigee Edge. The HTTP request is made to an endpoint managed by an API proxy. The API proxy in Apigee Edge performs API key verification for authentication, and applies specific levels of access and functionality as defined by the API product associated with the API key.
API proxy An API proxy forwards requests from a proxy endpoint (exposed to client applications) to a target endpoint. The target endpoint is your hosted model's endpoint for requesting AI Platform online predictions.
API product An API product bundles Apigee resources (such as API proxies) in order to provide a specific level of access and functionality to client app developers. For example, the API product can enforce access limits, API key approval methods, and other configurations for the API proxies used for accessing online predictions on hosted models.
Hosted model A hosted model is a trained ML model deployed to AI Platform. To request predictions from your hosted models, you use AI Platform online prediction and batch prediction.

Instead of exposing the hosted model in AI Platform directly to users, this solution's architecture uses Apigee Edge to allow external users to call hosted models through an API proxy. Proxying requests through Apigee Edge enables different usage policies to be enforced.
Model assets In this tutorial, to deploy a hosted TensorFlow model to AI Platform, you export your model in TensorFlow's SavedModel format and store the resulting model assets in a Cloud Storage bucket. Then AI Platform online or batch predictions can make predictions on the model.

Example scenario

In this tutorial, you use Apigee Edge to create two example API products for a simple hosted model (income prediction from census data) deployed to AI Platform.

The following diagram explains the two example API products: an Income Prediction Trial product with low usage quotas, and an Income Prediction Premium product with elevated quotas. In the example scenario, you register two developers and applications to test the API under different restrictions enforced by the two API products.

Two example API products for a
simple hosted model deployed to AI Platform

Creating a Google Cloud service account

Apigee Edge needs a Google Cloud service account to authenticate its requests to AI Platform online prediction.

GCLOUD

  1. Open Cloud Shell:

    GO TO Cloud Shell

    Unless otherwise noted, you use Cloud Shell throughout this tutorial.

  2. Set up an environment variable for your service account name:

    SERVICE_ACCOUNT=income-prediction-api@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com
    
  3. Create a service account:

    gcloud iam service-accounts create \
        income-prediction-api \
        --display-name "Apigee service account"
    
  4. Create a private key for the service account:

    gcloud iam service-accounts keys create \
        --iam-account ${SERVICE_ACCOUNT} \
        ~/key.json
    

    This command saves a private key file at ${HOME}/key.json. Keep this file secure; you need it for later steps.

Console

  1. In the Cloud Console, go to the IAM & admin > Service accounts page for your project:

    GO TO THE SERVICE ACCOUNTS PAGE

  2. Click Create service account, and enter the details for the service account:

    1. Provide a service account name, such as Income Prediction API.
    2. Provide a service account ID, such as income-prediction-api.
    3. Click Create.
  3. On the Grant this service account access to project (optional) page, leave the Role drop-down list blank and click Continue.

  4. On the Grant users access to this service account (optional) page, scroll down and click Create Key:

    1. Select the key type as JSON, which is the default.
    2. Click Create. Your browser initiates the download of your private key file. Keep this file secure; you need it later.
    3. Click Done.

Deploying the income prediction model to AI Platform

This tutorial uses an existing code sample for training an income prediction model with census data. For convenience, a trained model is included in this tutorial's code repository on GitHub.

Complete the following steps to deploy the pre-trained income prediction model to AI Platform:

  1. Open Cloud Shell:

    GO TO Cloud Shell

  2. In Cloud Shell, define your environment variables, which you use in later commands:

    GCS_MODEL_BUCKET=${GOOGLE_CLOUD_PROJECT}-ml-models
    MODEL_NAME=census
    MODEL_VERSION=v1
    MODEL_REGION=us-central1
    
  3. Clone the tutorial files from GitHub:

    cd ${HOME}
    
    git clone https://github.com/GoogleCloudPlatform/cloudml-apigee-serving.git
    
  4. Create a Cloud Storage regional storage bucket to store the prediction model:

    gsutil mb -c regional -l ${MODEL_REGION} \
        -p ${GOOGLE_CLOUD_PROJECT} gs://${GCS_MODEL_BUCKET}
    
  5. Extract and copy the model to Cloud Storage:

    cd ${HOME}/cloudml-apigee-serving/model
    
    tar zxvf census.tar.gz
    
    gsutil -m cp -rv census gs://${GCS_MODEL_BUCKET}
    
  6. Create the prediction model in AI Platform:

    gcloud ml-engine models create ${MODEL_NAME} \
         --regions ${MODEL_REGION}
    
  7. Create a new version of the prediction model:

    gcloud ml-engine versions create ${MODEL_VERSION} \
        --model ${MODEL_NAME} \
        --origin gs://${GCS_MODEL_BUCKET}/census \
        --runtime-version 1.6
    
  8. Get information about your new version:

    gcloud ml-engine versions describe ${MODEL_VERSION} \
        --model ${MODEL_NAME}
    

    The output looks similar to the following:

    deploymentUri: your_bucket_path
    framework: TENSORFLOW
    isDefault: true
    lastUseTime: '2018-05-07T04:41:22Z'
    name: projects/your_project/models/census/versions/v1
    pythonVersion: '2.7'
    runtimeVersion: '1.6'
    state: READY
    
  9. Before proceeding, ensure that you can run predictions successfully:

    gcloud ml-engine predict \
        --model ${MODEL_NAME} \
        --version ${MODEL_VERSION} \
        --json-instances ${HOME}/cmle-op-apigee-tutorial/model/test.json
    

    The output looks similar to the following. This output verifies that your hosted model is deployed correctly to AI Platform and can be called through AI Platform online prediction.

    CLASS_IDS  CLASSES  LOGISTIC               LOGITS                PROBABILITIES
    [0]        [u'0']   [0.23072466254234314]  [-1.204223871231079]  [0.7692753076553345, 0.23072466254234314]
    

Deploying the API proxy to Apigee Edge

An API proxy provides a way for clients to call your hosted model through an external API. The API proxy performs this task by proxying client requests from an external API interface hosted by Apigee Edge to the AI Platform predict API for your hosted model. The interface implements an API Facade Pattern, shielding users from target endpoint changes as new models and model versions are deployed. See API proxies for more information.

In an API proxy configuration, you use flows to configure the processing steps executed in your API request and response. Processing steps can include activities such as generating and signing JSON web tokens (JWT) tokens, adding request and response headers, or using specific Apigee Edge features. Apigee Edge provides a number of built-in API policies for performing these activities. You can add these policies to your API flows.

Apigee Edge provides a number of ways to deploy API proxies; these ways include a web management interface (sign-in required), management APIs, and command-line tools.

For this tutorial, you use apigeetool to create the required Apigee resources and deploy the API proxy.

  1. Open Cloud Shell:

    GO TO Cloud Shell

  2. In Cloud Shell, set up environment variables. You need these variables later. Replace [YOUR_APIGEE_USER_EMAIL], [YOUR_APIGEE_ORGANIZATION], [YOUR_APIGEE_ENVIRONMENT], and [YOUR_APIGEE_USER_PASSWORD] with your values.

    • Your Apigee account is automatically granted the Organization Administrator role, and has super-user access to all resources within your Apigee organization. Learn more about the function of different Apigee roles.
    • Your Apigee organization is automatically created. You can find the name of your organization in the Apigee web management interface, in the top-left corner of the page where your account name is displayed.
    • Two Apigee environments named test and prod have been automatically created for you. You can use either environment in the following steps.

    1. Export environment variables required by apigeetool:

      export APIGEE_USERNAME=[YOUR_APIGEE_USER_EMAIL]
      export APIGEE_ORGANIZATION=[YOUR_APIGEE_ORGANIZATION]
      
    2. Optionally, export an environment variable for your Apigee password:

      export APIGEE_PASSWORD=[YOUR_APIGEE_USER_PASSWORD]
      
    3. Set up other environment variables needed to work with your Apigee environment:

      APIGEE_ENVIRONMENT=[YOUR_APIGEE_ENVIRONMENT]
      
    4. Keep the default Apigee cache name as 'google':

      APIGEE_CACHE=google
    5. Set up environment variables for your AI Platform model name and version. These variables should be the same as the previous values you used for deploying the model version:

      MODEL_NAME=census
      MODEL_VERSION=v1
      
    6. Set up an environment variable for the service account used by Apigee to call AI Platform online prediction:

      SERVICE_ACCOUNT=income-prediction-api@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com
      
  3. Install apigeetool, which you use to create Apigee resources and deploy the API proxy:

    npm install -g apigeetool
    
    alias apigeetool=$(npm config get prefix)/bin/apigeetool
    
  4. Extract your service account private key from the JSON credentials file that you saved earlier:

    PRIVATE_KEY=`sed -rne 's/^.*\"private_key\": \"(.*)\",$/\1/p' <  ${HOME}/key.json`
    
  5. Create an encrypted key value map (KVM) to securely store the service account name and private key in Apigee Edge:

    apigeetool createkvmmap \
        -e ${APIGEE_ENVIRONMENT} \
        --encrypted \
        --mapName "secrets"
    
  6. Add the service account name to the encrypted KVM:

    apigeetool addEntryToKVM \
        -e ${APIGEE_ENVIRONMENT} \
        --mapName "secrets" \
        --entryName "service-account" \
        --entryValue "${SERVICE_ACCOUNT}"
    
  7. Add the private key to the encrypted KVM:

    apigeetool addEntryToKVM \
        -e ${APIGEE_ENVIRONMENT} \
        --mapName "secrets" \
        --entryName "private-key" \
        --entryValue "`echo -e ${PRIVATE_KEY}`"
    
  8. Update the API proxy endpoint to your deployed model in AI Platform. The API proxy forwards requests to the projects.predict method for your hosted model in AI Platform online prediction.

    sed -i \
        -e "s/\[project\]/${GOOGLE_CLOUD_PROJECT}/" \
        -e "s/\[model-name\]/${MODEL_NAME}/" \
        -e "s/\[model-version\]/${MODEL_VERSION}/" \
        ${HOME}/cmle-op-apigee-tutorial/apiproxy/targets/default.xml
    
  9. Create an Apigee cache for temporary storage of OAuth2 access tokens returned from the Google Authorization Server:

    apigeetool createcache \
        -e ${APIGEE_ENVIRONMENT} \
        -z ${APIGEE_CACHE}
    
  10. Grant Identity and Access Management (IAM) permissions to allow your service account to request predictions on your model. You grant these permissions by adding the IAM policy binding for the census model, as follows:

    gcloud ml-engine models add-iam-policy-binding \
        ${MODEL_NAME} \
        --member=serviceAccount:${SERVICE_ACCOUNT} \
        --role=roles/ml.modelUser
    
  11. Deploy the API proxy:

    apigeetool deployproxy  \
        -e ${APIGEE_ENVIRONMENT} \
        -n income_prediction -d ${HOME}/cmle-op-apigee-tutorial
    

Creating API products to distribute your APIs

You can use an Apigee API product to configure specific levels of access and functionality exposed to developers. For example, you might create two API products to reflect "Trial" and "Premium" tiers of service for your hosted models in AI Platform, where each tier is differentiated by allowed quota, accessible functionality, and other features.

In this section, you create two API products: Income Prediction Trial and Income Prediction Premium. You configure both API products to use the API proxy that you deployed in the previous step, but you enforce different usage policies, as explained here:

  • The Income Prediction Trial API product enforces a low daily usage quota, but allows developers to obtain API keys through an automated process. This policy might reflect a "free trial" tier, where you provide simple access to your API and relatively low usage limits.

  • The Income Prediction Premium API product provides a higher daily usage quota, but requires developers to go through a manual approval process. This policy might reflect a paid tier for your API that provides access and elevated usage limits.

To create the API products, follow these steps.

  1. Create the Income Prediction Trial API product:

    apigeetool createProduct \
        --approvalType "auto" \
        --environments ${APIGEE_ENVIRONMENT} \
        --proxies income_prediction \
        --productName "income_prediction_trial" \
        --productDesc "Free trial API for income prediction." \
        --quota 10 \
        --quotaInterval 1 \
        --quotaTimeUnit "day"
    
  2. Create the Income Prediction Premium API product:

    apigeetool createProduct \
        --approvalType "manual" \
        --environments ${APIGEE_ENVIRONMENT} \
        --proxies income_prediction \
        --productName "income_prediction_premium" \
        --productDesc "Premium API for income prediction." \
        --quota 10000 \
        --quotaInterval 1 \
        --quotaTimeUnit "day"
    

Creating developer profiles

Developer profiles can either be created by an Apigee Edge administrator, or by individuals using a self-service sign up flow.

  1. Create a developer profile for John:

    apigeetool createDeveloper \
        --email john@example.com \
        --firstName John \
        --lastName Citizen \
        --userName john@example.com
    
  2. Create a second developer profile for Sally:

    apigeetool createDeveloper \
        --email sally@example.com \
        --firstName Sally \
        --lastName Resident \
        --userName sally@example.com
    

Registering developer applications

In this section, you manually register developer applications for the two developer profiles.

On successful completion of each of the following commands, you see a JSON response printed to the terminal. For each command, note the value of the consumerKey property, which is the API key generated for the developer application.

  1. Register the developer application FooApp and associate it with John, to use with the Income Prediction Trial API product. This command uses jq to extract the value of the consumerKey property from the response. Note that value.

    FOOAPP_API_KEY="$(apigeetool createApp \
        --apiProducts "income_prediction_trial" \
        --name "FooApp" \
        --email john@example.com \
        | jq -r '.credentials[0].consumerKey')"
    
  2. Register the developer application BarApp and associate it with Sally, to use with the Income Prediction Premium API product. Note the consumerKey value in the output.

    BARAPP_API_KEY="$(apigeetool createApp \
        --apiProducts "income_prediction_premium" \
        --name "BarApp" \
        --email sally@example.com \
        | jq -r '.credentials[0].consumerKey')"
    

Testing the API proxy

To test the API proxy, you can now use the API key to call the Income Prediction API.

  1. Test the API key (the value of the consumerKey property) for the FooApp application, which is registered to developer John.

    curl -s -X POST \
        -H "Content-Type: application/json" \
        -H "X-APIKEY: ${FOOAPP_API_KEY}" \
        -d '{
            "instances": [{
                "age": 25,
                "workclass": "Private",
                "education": "Some-college",
                "education_num": 1,
                "marital_status": "Never-married",
                "occupation": "Other-service",
                "relationship": "Other-relative",
                "race": "Other",
                "gender": "Female",
                "capital_gain": 0,
                "capital_loss": 0,
                "hours_per_week": 40,
                "native_country": "United-States"}]
        }' \
        https://${APIGEE_ORGANIZATION}-${APIGEE_ENVIRONMENT}.apigee.net/ml/census/v1/predict
    

    If the test succeeds, the response looks similar to the following:

    {
        "predictions":[
            {
                "probabilities":[
                    0.7692753076553345,
                    0.23072466254234314
                ],
                "class_ids":[
                    0
                ],
                "classes":[
                    "0"
                ],
                "logits":[
                    -1.204223871231079
                ],
                "logistic":[
                    0.23072466254234314
                ]
            }
        ]
    }
    

    Repeat the request ten more times to exhaust the quota:

    for i in `seq 10`; do !! | jq '.' ; done
    

    When the quota is exceeded, Apigee Edge returns an error:

    {
        "fault":{
            "detail":{
                "errorcode": "policies.ratelimit.QuotaViolation"
            },
            "faultstring": "Rate limit quota violation. Quota limit exceeded. Identifier : \_default"
        }
    }
    

    The request is rejected if the API key is removed from the request header:

    {
        "fault":{
            "faultstring": "Failed to resolve API Key variable request.header.X-APIKEY",
            "detail":{
               "errorcode": "steps.oauth.v2.FailedToResolveAPIKey"
            }
        }
    }
    
  2. Test the API key (the value of the consumerKey property) for the BarApp application, which is registered to developer Sally.

    curl -s -X POST \
        -H "Content-Type: application/json" \
        -H "X-APIKEY: ${BARAPP_API_KEY}" \
        -d '{
            "instances": [{
                "age": 25,
                "workclass": "Private",
                "education": "Some-college",
                "education_num": 1,
                "marital_status": "Never-married",
                "occupation": "Other-service",
                "relationship": "Other-relative",
                "race": "Other",
                "gender": "Female",
                "capital_gain": 0,
                "capital_loss": 0,
                "hours_per_week": 40,
                "native_country": "United-States"}]
        }' \
        https://${APIGEE_ORGANIZATION}-${APIGEE_ENVIRONMENT}.apigee.net/ml/census/v1/predict | jq '.'
    

    The output looks similar to the following, because the API product for the API key has not yet been approved:

    {
        "fault": {
            "faultstring": "Invalid ApiKey for given resource",
            "detail": {
                "errorcode": "oauth.v2.InvalidApiKeyForGivenResource"
            }
        }
    }
    

Approving and revoking API products

In the previous step, the API proxy returned an error when the API key for the BarApp application was used, because the API product had not been approved for use with the API key.

To approve the API product for use with the BarApp API key, you can use the Apigee Edge Management API. Alternatively, you can use the Apigee Edge web interface to approve the API product. Learn more about manually approving or revoking API keys using the web interface.

Complete the following steps in Cloud Shell to approve the API product using the Apigee Edge Management API.

  1. Set up environment variables, which you need in later commands:

    DEVELOPER_EMAIL=sally@example.com
    APP_NAME=BarApp
    API_PRODUCT=income_prediction_premium
    
  2. Call the Apigee Edge Management API to approve the API product for use with the API key:

    curl -s -X POST \
        -u "${APIGEE_USERNAME}" \
        --header "Content-Type: application/octet-stream" \
        "https://api.enterprise.apigee.com/v1/organizations/${APIGEE_ORGANIZATION}/developers/${DEVELOPER_EMAIL}/apps/${APP_NAME}/keys/${BARAPP_API_KEY}/apiproducts/${API_PRODUCT}?action=approve"
    
  3. Retry the API request using the API key:

    curl -s -X POST \
        -H "Content-Type: application/json" \
        -H "X-APIKEY: ${BARAPP_API_KEY}" \
        -d '{
            "instances": [{
                "age": 25,
                "workclass": "Private",
                "education": "Some-college",
                "education_num": 1,
                "marital_status": "Never-married",
                "occupation": "Other-service",
                "relationship": "Other-relative",
                "race": "Other",
                "gender": "Female",
                "capital_gain": 0,
                "capital_loss": 0,
                "hours_per_week": 40,
                "native_country": "United-States"}]
        }' \
        https://${APIGEE_ORGANIZATION}-${APIGEE_ENVIRONMENT}.apigee.net/ml/census/v1/predict | jq '.'
    

    The output looks similar to the following, because the API product for the API key is approved:

    {
        "predictions":[
            {
                "probabilities":[
                    0.7692753076553345,
                    0.23072466254234314
                ],
                "class_ids":[
                    0
                ],
                "classes":[
                    "0"
                ],
                "logits":[
                    -1.204223871231079
                ],
                "logistic":[
                    0.23072466254234314
                ]
            }
        ]
    }
    

Enabling self-service registration using the developer portal

In this tutorial, you used an Apigee administrative account to create developer profiles, applications, and API keys.

You might want to enable self-service registration for your API product by using a Developer Services Portal. This portal allows users to register developer profiles and applications, and generate API keys through a self-service flow. If you enable a self-service portal, individual users can gain access to your API through an automated flow. Depending on your needs, API key approval through such an automated flow can either occur automatically, or still require manual approval.

Apigee Edge provides a graphical web-based tool for building your developer portal. Learn more about how to build a developer portal for your API products.

Cleaning up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:

  • Delete the project.
  • Close your Apigee Edge Trial account.
  • Stop serving API proxy requests.

Delete the project

  1. In the Cloud Console, go to the Manage resources page.

    Go to the Manage resources page

  2. In the project list, select the project that you want to delete and then click Delete .
  3. In the dialog, type the project ID and then click Shut down to delete the project.

Close your Apigee Edge Trial account

Your Apigee Edge Trial account is automatically closed when the trial period expires. You don't need to take any specific actions to delete your trial account.

Stop serving API proxy requests

To stop serving API proxy requests, undeploy or delete the API proxy. Also, to prevent new registrations, delete any Developer Portals.

  1. In Cloud Shell, undeploy the API proxy using apigeetool:

    apigeetool undeploy \
        -u ${APIGEE_USERNAME} \
        -o ${APIGEE_ORGANIZATION} \
        -n income_prediction
        -e ${APIGEE_ENVIRONMENT} \
        -r 1 \
        -n income_prediction
    

    The output looks similar to the following:

    "income_prediction" Revision 1
      undeployed
      environment = test
      base path = /
    

What's next