Using AI Explanations with AI Platform Prediction

This is a general guide on how to deploy and use a model on AI Platform Prediction with AI Explanations.

Before you begin

You must do several things before you can train and deploy a model in AI Platform:

  • Set up your local development environment.
  • Set up a GCP project with billing and the necessary APIs enabled.
  • Create a Cloud Storage bucket to store your training package and your trained model.

To set up your GCP project, follow the instructions provided in the sample notebooks.

Saving a model

TensorFlow 1.15, 2.1, and 2.2 are supported. Use to save models in TensorFlow 2.2, but do not use it for TensorFlow 1.15.

Learn more about how to save models for use with AI Explanations, and how to use the Explainable AI SDK.

Explanation metadata file

Before deploying your model, you must submit a metadata file with information about your inputs, outputs and baseline, so that AI Explanations provides explanations for the correct parts of your model. You can use the Explainable AI SDK to generate the inputs and outputs for you.

  • Identify the names of the input and output tensors you want explanations for. Learn more about how to identify your input and output tensors.
  • Pass your baseline to input_baselines.
  • Specify "tensorflow" for your framework.
  • Name the file explanation_metadata.json.
  • Upload your explanation_metadata.json file to the same Cloud Storage bucket where your SavedModel is stored.

The following example shows what an explanation_metadata.json file looks like:

    "inputs": {
      "data": {
        "input_tensor_name": "YOUR_INPUT_TENSOR_NAME"
        "input_baselines": [0]
    "outputs": {
      "duration": {
        "output_tensor_name": "YOUR_OUTPUT_TENSOR_NAME"
    "framework": "tensorflow"

In this example, "data" and "duration" are meaningful names for the input and output tensors that you can assign in the process of building and training the model. The actual input and output tensor names follow the format name:index. For example, x:0 or Placeholder:0.

For input_baselines, you can start by specifying one baseline. In this example, [0] represents an all-black image. Learn more about adjusting baselines.

Locating input and output tensor names manually

In most cases, you can use the Explainable AI SDK to generate the inputs and outputs for your model. You only need to find your input and output tensor names manually if you're using a pretrained TensorFlow 1.15 model.

The best way to find your input and output tensor names depends on the types of your input and output data, as well as how you build your model. For a more in-depth explanation of each case, as well as examples, refer to the guide on [understanding inputs and outputs][understanding-inputs-outputs].

Input data type(s) Output data type Other criteria Recommended approach(es)
Numeric or string Numeric Inputs are not in serialized form. Outputs are not numeric data treated as categorical data (for example, numeric class IDs). Use the SavedModel CLI to find the names of your input and output tensors. Alternatively, build the explanation metadata file while training and saving the model, where your program or environment still has access to the training code.
Any serialized data Any Add a TensorFlow parsing operation to your serving input function when you export the model. Use the output of the parsing operation to help identify input tensors.
Any Any Model includes preprocessing operations To get the names of the input tensors after the preprocessing steps, use the name property of tf.Tensor to get the input tensor names.
Any Not probabilities, logits, or other types of floating point tensors You want to get explanations for outputs that are not probabilities, logits, or other types of floating point tensors. Inspect your graph with TensorBoard to find the correct output tensors.
Any non-differentiable data Any You want to use integrated gradients, which require differentiable inputs. Encode non-differentiable inputs as differentiable tensors. Add the names of both the original input tensor and the encoded input tensor to your explanation metadata file.

Deploy models and versions

AI Platform Prediction organizes your trained models using model and version resources. An AI Platform Prediction model is a container for the versions of your machine learning model.

To deploy a model, you create a model resource in AI Platform Prediction, create a version of that model, then link the model version to the model file stored in Cloud Storage.

Create a model resource

AI Platform Prediction uses model resources to organize different versions of your model.

Create a model resource for your model versions. In the following gcloud command-line tool command, replace MODEL_NAME with your desired name for your model. Your model name must start with a letter, and it must contain only letters, numbers, and underscores.

You must create your model on a regional endpoint in order to use AI Explanations.

gcloud ai-platform models create MODEL_NAME \
  --region us-central1

See the AI Platform Prediction model API for more details.

Create a model version

Now you are ready to create a model version with the trained model you previously uploaded to Cloud Storage. When you create a version, specify the following parameters:

  • name: must be unique within the AI Platform Prediction model.
  • deploymentUri: the path to your SavedModel directory in Cloud Storage.
  • framework (required): tensorflow only
  • runtimeVersion: use any of the following runtime versions that support AI Explanations: 1.15, 2.1, and 2.2.
  • pythonVersion: must be set to "3.5" if you're using runtime version 1.14. Use "3.7" for runtime version 1.15.
  • machineType (required): the type of virtual machine that AI Platform Prediction uses for the nodes that serve predictions and explanations. Select a supported machine type for AI Explanations.
  • explanation-method: the type of feature attribution method to use: "sampled-shapley", "integrated-gradients", or "xrai".
  • Paths or steps: use --num-paths for sampled Shapley, and use --num-integral-steps for integrated gradients or XRAI.

See more information about each of these parameters in the AI Platform Training and Prediction API for a version resource.

  1. Set environment variables to store the path to the Cloud Storage directory where your SavedModel is located, your model name, your version name and your framework choice.

  2. Create the version:

    gcloud beta ai-platform versions create $VERSION \
    --model $MODEL \
    --origin $MODEL_DIR \
    --runtime-version 1.15 \
    --framework $FRAMEWORK \
    --python-version 3.5 \
    --machine-type n1-standard-4 \
    --explanation-method $EXPLAIN_METHOD \
    --num-integral-steps 25 \
    --region us-central1

    Creating the version takes a few minutes. When it is ready, you should see the following output:

    Creating version (this might take a few minutes)......done.
  3. Check the status of your model deployment, and ensure that your model deployed correctly:

    gcloud ai-platform versions describe $VERSION_NAME \
      --model $MODEL_NAME \
      --region us-central1

    Check that the state is READY. You should see output similar to this:

    createTime: '2018-02-28T16:30:45Z'
    deploymentUri: gs://your_bucket_name
    framework: TENSORFLOW
    machineType: mls1-c1-m2
    name: projects/your_project_id/models/your_model_name/versions/your_version_name
    pythonVersion: '3.5'
    runtimeVersion: '1.15'
    state: READY

Supported machine types for explanations

For AI Explanations requests, you must deploy your model version with one of the following machine types. If you don't specify a machine type, the deployment fails.

The following table compares the available machine types:

Name vCPUs Memory (GB)
n1-standard-2 2 7.5
n1-standard-4 4 15
n1-standard-8 8 30
n1-standard-16 16 60
n1-standard-32 32 120

These machine types are only available on regional endpoints. Learn more about their support for other AI Platform Prediction features.

Learn about pricing for each machine type. Read more about the detailed specifications of Compute Engine (N1) machine types in the Compute Engine documentation.

Format input data

The basic format for online prediction is a list of data instances. These can be either plain lists of values or members of a JSON object, depending on how you configured your inputs in your training application. Learn how to format complex inputs and binary data for prediction.

This example shows an input tensor and an instance key to a TensorFlow model:

{"values": [1, 2, 3, 4], "key": 1}

The makeup of the JSON string can be complex as long as it follows these rules:

  • The top level of instance data must be a JSON object: a dictionary of key/value pairs.

  • Individual values in an instance object can be strings, numbers, or lists. You cannot embed JSON objects.

  • Lists must contain only items of the same type (including other lists). You may not mix string and numerical values.

You pass input instances for online prediction as the message body for the projects.explain call. Learn more about the prediction request body's formatting requirements.

  1. To submit your request with gcloud, ensure that your input file is a newline-delimited JSON file, with each instance as a JSON object, one instance per line.

    {"values": [1, 2, 3, 4], "key": 1}
    {"values": [5, 6, 7, 8], "key": 2}

Request predictions and explanations

Request your predictions and explanations:

gcloud beta ai-platform explain \
  --model $MODEL \
  --version $VERSION \
  --json-instances='your-data.txt' \
  --region us-central1

Understand the explanations response

After you have deployed your model, you can use the Explainable AI SDK to request explanations and visualize feature attribution results for both tabular and image data:

import explainable_ai_sdk

m = explainable_ai_sdk.load_model_from_ai_platform(PROJECT_ID, MODEL_NAME, VERSION)
explanations = m.explain([instance_dict])

For tabular models, the attributions are plotted in a bar chart. For image models, the attributions are displayed on the input image, using the same visualization settings you specified when you deployed the model.

For details on each field in the explanations response, refer to the full example response in the API reference.

Learn how to parse the explanations response by referring to the example notebooks:

Check your explanations

The following code example helps you to check a batch of explanations and see if you need to adjust your baselines.

In the code, you only need to update your input key value according to what you specified in your explanation_metadata.json file.

    "inputs": {
        "input_tensor_name": "YOUR_INPUT_TENSOR_NAME"
        "input_baselines": [0]

For example, if your input key value was "data", then the same value would be "data" in line 4 of the following code snippet:

def check_explanations(example, mean_tgt_value=None, variance_tgt_value=None):
  passed_test = 0
  total_test = 1
  attribution_vals = example['attributions_by_label'][0]['attributions']['YOUR-INPUT-KEY-VALUE']

  baseline_score = example['attributions_by_label'][0]['baseline_score']
  sum_with_baseline = np.sum(attribution_vals) + baseline_score
  predicted_val = example['attributions_by_label'][0]['example_score']

  # Check 1
  # The prediction at the input is equal to that at the baseline.
  #  Please use a different baseline. Some suggestions are: random input, training
  #  set mean.
  if abs(predicted_val - baseline_score) <= 0.05:
    print('Warning: example score and baseline score are too close.')
    print('You might not get attributions.')
    passed_test += 1

  # Check 2 (only for models using Integrated Gradient explanations)
  # Ideally, the sum of the integrated gradients must be equal to the difference
  # in the prediction probability at the input and baseline. Any discrepancy in
  # these two values is due to the errors in approximating the integral.
  if explain_method == 'integrated-gradients':
    total_test += 1
    want_integral = predicted_val - baseline_score
    got_integral = sum(attribution_vals)
    if abs(want_integral-got_integral)/abs(want_integral) > 0.05:
        print('Warning: Integral approximation error exceeds 5%.')
        print('Please try increasing the number of integrated gradient steps.')
        passed_test += 1

  print(passed_test, ' out of ', total_test, ' sanity checks passed.')

When parsing your explanations, you can run these checks on each attribution you have received:

for i in attributions_resp['explanations']:

Using the approximation error to improve results

The AI Explanations feature attribution methods (sampled Shapley, integrated gradients, and XRAI) are all based on variants of Shapley values. Because Shapley values are very computationally expensive, AI Explanations provides approximations instead of the exact values. Along with feature attribution results, AI Explanations also returns an approximation error. If your approximation error exceeds 0.05, consider adjusting your inputs to reduce the error.

You can reduce the approximation error and get closer to the exact values by changing the following inputs:

  • Increasing the number of integral steps or number of paths.
  • Changing the input baseline(s) you select.
  • Adding more input baselines. With the integrated gradients and XRAI methods, using additional baselines increases latency. Using additional baselines with the sampled Shapley method does not increase latency.

Increasing steps or paths

To reduce approximation error, you can increase:

  • the number of paths for sampled Shapley
  • the number of integral steps for integrated gradients or XRAI

You set these parameters when you create a version resource during model deployment.

Adjusting baselines

You can set input_baselines in your explanation_metadata.json file. This section provides examples for tabular and image data. Input baselines can represent median, minimum, maximum, or random values in relation to your training data.

In general:

  • Start with one baseline representing median values.
  • Change this baseline to one representing random values.
  • Try two baselines, representing the minimum and maximum values.
  • Add another baseline representing random values.

Example for tabular data

The following Python code creates the contents of an explanation metadata file for tabular data. You can use either sampled Shapley or integrated gradients to get feature attributions for tabular data. This code is part of the example notebook for tabular data.

Notice that input_baselines is a list where you can specify multiple baselines. This example sets just one baseline. The baseline is a list of median values for the training data (train_data in this example).

explanation_metadata = {
    "inputs": {
      "data": {
        "input_baselines": [train_data.median().values.tolist()],
        "encoding": "bag_of_features",
        "index_feature_mapping": train_data.columns.tolist()
    "outputs": {
      "duration": {
  "framework": "tensorflow"

To set two baselines representing minimum and maximum values, set input_baselines as follows: [train_data.min().values.tolist(), train_data.max().values.tolist()]

Example for image data

The following Python code creates the contents of an explanation metadata file for image data. You can use integrated gradients to get feature attributions for image data. This code is part of the example notebook for image data.

Notice that input_baselines is a list where you can specify multiple baselines. This example sets just one baseline. The baseline is a list of random values. Using random values for an image baseline is a good approach if the images in your training dataset contain a lot of black and white.

Otherwise, set input_baselines to [0, 1] to represent black and white images.

random_baseline = np.random.rand(192,192,3)

explanation_metadata = {
    "inputs": {
      "data": {
        "input_tensor_name": "input_pixels:0",
        "modality": "image",
        "input_baselines": [random_baseline.tolist()]
    "outputs": {
      "probability": {
        "output_tensor_name": "dense/Softmax:0"
  "framework": "tensorflow"

What's next