Limitations of AI Explanations

As you consider the explanations returned from the service, you should keep in mind these high-level limitations. For an in-depth explanation, refer to the AI Explainability Whitepaper.

Meaning and scope of feature attributions

Consider the following when analyzing feature attributions provided by AI Explanations:

  • Each attribution only shows how much the feature affected the prediction for that particular example. A single attribution might not reflect the overall behavior of the model. To understand approximate model behavior on an entire dataset, aggregate attributions over the entire dataset.
  • The attributions depend entirely on the model and data used to train the model. They can only tell the patterns the model found in the data, and can't detect any fundamental relationships in the data. So, the presence or absence of a strong attribution to a certain feature doesn't mean there is or is not a relationship between that feature and the target. The attribution merely shows that the model is or is not using the feature in its predictions.
  • Attributions alone cannot tell if your model is fair, unbiased, or of sound quality. Carefully evaluate your training dataset, procedure, and evaluation metrics in addition to the attributions.

Improving feature attributions

The following factors have the highest impact on feature attributions:

  • The attribution methods approximate the Shapley value. You can increase the precision of the approximation by increasing the number of integral steps or paths for the integrated gradients or sampled Shapley methods, respectively. As a result, the attributions could change dramatically.
  • The attributions only express how much the feature affected the change in prediction value, relative to the baseline value. Take care to choose a meaningful baseline, relevant to the question you're asking of the model. Attribution values and their interpretation might change significantly as you switch baselines.