Recommendations in TensorFlow: Deploy the recommendation system

This article is the fourth part of a multi-part tutorial series that shows you how to implement a machine learning (ML) recommendation system with TensorFlow and AI Platform. This part shows you how to deploy a production system on Google Cloud to serve recommendations and perform periodic updates to the recommendation model.

The series consists of these parts:

The recommendation system in this tutorial uses the weighted alternating least squares (WALS) algorithm. WALS is included in the contrib.factorization package of the TensorFlow codebase, and is used to factorize a large matrix of user and item ratings. For details, see the Overview.


  • Deploy an App Engine endpoint to serve recommendations.
  • Deploy a Cloud Composer environment to orchestrate model training updates.


This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

The projected cost for this tutorial is $0.15.

Before you begin

This tutorial relies on the steps performed in Part 3 of the tutorial set.

  1. Select the Google Cloud project you used for Part 3:


  2. Make sure that billing is enabled for your project:


  3. Enable the Cloud Resource Manager, Source Repositories, AI Platform, App Engine Admin, and Kubernetes Engine APIs.

    Enable the APIs

In earlier parts of this series, you saw how to train a TensorFlow model to generate web page recommendations from Google Analytics data. To make recommendations available, you can provide a REST API endpoint that serves the recommendations to a client application. This REST endpoint takes a Google Analytics client ID and number of recommendations K as inputs, and returns K number of article IDs that are recommended for that user.

This tutorial provides sample code for a REST API endpoint for serving article recommendations, and shows you how to deploy the code as an App Engine flexible environment application. Service discovery and authentication for the API is provided by Endpoints.

To incorporate new website articles and user-click data into the recommendations, you must periodically retrain and redeploy the model. This involves these steps:

  1. Query training data from Google Analytics tables in BigQuery.
  2. Export that data to Cloud Storage as a CSV file.
  3. Train a new recommendation model on the data.
  4. Deploy the updated model to the API.

You can use Cloud Composer to implement and manage this sequence. Cloud Composer is a managed service for Apache Airflow on Google Cloud. Apache Airflow is an orchestration manager for sequences of tasks in which each step involves an operation performed by a cloud service, and in which each step depends on the successful completion of the previous step. The tutorial shows you how to deploy a Cloud Composer environment and provides sample configuration of a workflow represented in a directed acyclic graph (DAG) to perform the model update steps.

The following diagram shows the complete architecture of the deployed solution.

Archicture of the deployed recommendation system
Figure 1. Architecture of the deployed recommendation system

Deploying the recommendation service

  1. Open a command shell on a computer where you've installed the Cloud SDK and the tutorial code.

  2. Change to the top-level directory for the solution code.

  3. Make a Cloud Storage bucket with the name recserve_[YOUR-PROJECT-ID]

    export BUCKET=gs://recserve_$(gcloud config get-value project 2> /dev/null)
    gsutil mb ${BUCKET}

Set up WALS model package and model data

The training task in the Airflow DAG requires the code package for the model to be deployed in a static location in Cloud Storage.

  1. Create a distributable package for the model code using the setup script in the model directory:

    cd wals_ml_engine
    python sdist
  2. Copy the package to the code folder in the bucket you created:

    gsutil cp dist/wals_ml_engine-0.1.tar.gz ${BUCKET}/code/
  3. When the recommendation service is launched, it requires the numpy array model files that were discussed in Part 3.

    To create these initial model files, run an AI Platform job on the initial training data, with an output directory argument pointing to your Cloud Storage bucket:

    ./ local ../data recommendation_events.csv \
        --data-type web_views --use-optimized --output-dir ${BUCKET} 

    This process takes a few minutes to complete.

Install the API endpoint and application

  1. Create the App Engine application:

    gcloud app create --region=us-east1
    gcloud app update --no-split-health-checks

    You can choose any Google Cloud region for the application. In this example, the command creates the application in the us-east1 region.

  2. Prepare to deploy the API endpoint service:

    cd scripts

    The following output appears:

    To deploy:  gcloud endpoints services deploy /var/folders/1m/r3slmhp92074pzdhhfjvnw0m00dhhl/T/tmp.Xr4t4ici.yaml

    The script prepares the endpoint service configuration file from the template app/openapi.yaml, by making a temporary copy of the template file and substituting your project ID in the file.

  3. Run the command that is output by the script:

    gcloud endpoints services deploy [TEMP_FILE_PATH]
  4. Prepare to deploy the App Engine app:


    The following output appears:

    ERROR: ( The project [lramsey-goog-com-csa-airflow] already contains an App Engine application...
    To deploy:  gcloud -q app deploy ../app/app_template.yaml_deploy.yaml

    You can ignore the error that's displayed by the script; this is the expected behavior.

    The script similarly prepares the app configuration file from the template app/app_template.yaml, by making a temporary copy of the template file and substituting your application ID and endpoint service ID in the file.

  5. Run the command that is output by the script:

    gcloud -q app deploy ../app/app_template.yaml_deploy.yaml

Deploying Cloud Composer

This section describes the orchestration tasks performed by Cloud Composer, and how to deploy the Airflow components to your project.

The recommendation service DAG

Apache Airflow manages sequences of tasks with dependencies represented in a DAG. The following diagram shows the DAG for the recommendation service.

DAG flow showing 4 tasks
Figure 2. The Airflow DAG for this solution

The DAG consists of four tasks:

  1. bq_rec_training_data: Runs the SQL query from Part 3 against the BigQuery dataset that contains the tables imported from Google Analytics. Dumps that data into the recommendation_events table.

  2. bq_export_op: Exports the data from the recommendation_events table into a CSV file in your Cloud Storage bucket.

  3. ml_engine_training_op: Trains the model using the CSV file of event data as input. The training task writes new model files to the Cloud Storage model directory that is read by the App Engine endpoint.

  4. app_engine_deploy_version: Deploys a new version of the App Engine endpoint app, so that the new model files are loaded. Migrates traffic to the new app version.

The DAG is defined in the airflow/dags/ file. Task 4 uses an Airflow plugin for App Engine, defined in the airflow/plugins/ file.

Deploy the Airflow service

  1. Create a Cloud Composer environment in your project:

    gcloud composer environments create $CC_ENV --location us-central1

    This process takes a few minutes to complete.

  2. Get the name of the Cloud Storage bucket created for you by Cloud Composer:

    gcloud composer environments describe $CC_ENV \
        --location us-central1 --format="csv[no-heading](config.dagGcsPrefix)" | sed 's/.\{5\}$//'

    This bucket contains subfolders for DAGs and plugins. In the output, you see the location of the Cloud Storage bucket, like this:

  3. Set a shell variable that contains the path to that output:

    export AIRFLOW_BUCKET="gs://[region-environment_name-random_id-bucket]"
  4. Copy the DAG file to the dags folder in your Cloud Composer bucket:

    gsutil cp airflow/dags/ ${AIRFLOW_BUCKET}/dags
  5. Import the solution plugins to your composer environment:

    gcloud composer environments storage plugins import \
        --location us-central1 --environment ${CC_ENV} --source airflow/plugins/

Using the recommendation service

This section shows you how to use the recommendation service, including accessing the endpoint API and the Airflow web console.

Recommendation endpoint service

The OpenAPI service specification file for the recommendation endpoint is app/openapi.yaml. For more information on using Endpoints see the documentation.

To test the recommendation service endpoint, you can use the script:

cd scripts

The output of that script shows a sample call to the API, with the JSON return value:

curl "https://[PROJECT_ID]"

    "articles": [

Accessing the Airflow web console

The Airflow web console allows you to manage the configuration and execution of the DAG. For example, using the console you can:

  • Inspect and change the schedule of the DAG execution.
  • Manually execute tasks within the DAG.
  • Inspect task logs.

For more information on how to use the console, see the Airflow documentation.

  1. Access the Airflow web console:

    gcloud composer environments describe $CC_ENV \
        --location us-central1 --format="csv[no-heading](config.airflow_uri)"

    You see the URL for the console website, which looks like the following:
  2. To access the Airflow console for your Cloud Composer instance, go to the URL displayed in the output.

Cleaning up

If you don't want to continue this series, we recommend that you delete the resources you created to avoid incurring charges to your Google Cloud account. The easiest way to delete all resources is to delete the project you created for this tutorial.

Delete the project

The easiest way to eliminate billing is to delete the project that you created for the tutorial.

To delete the project:

  1. In the Cloud Console, go to the Manage resources page.

    Go to the Manage resources page

  2. In the project list, select the project that you want to delete and then click Delete .
  3. In the dialog, type the project ID and then click Shut down to delete the project.

What's next

  • Learn more about recommendation systems. There are several annual conferences on this topic. See these papers for machine learning models for recommendation developed by Google researchers.
  • Try out other Google Cloud features for yourself. Have a look at our tutorials.