Agent validation

As discussed on the Agent design page, the design of your agent can greatly influence the quality of your agent. To help agent designers create high-quality agents, Dialogflow provides a validation feature. Agent validation results are available automatically whenever agent training is performed and completed. You can access the results of validation from either the Dialogflow Console or the API.

The validation results are informational only. They provide a list of errors that you should correct to improve the quality and performance of your agent. If your agent has errors, you can choose to ignore them and launch your agent. Agent validation does not affect the behavior of an agent in any way.

Here are some example results that you might find:

  • Intent has training phrases which are too similar.
  • Intent contains a parameter that is not used in enough training phrases.
  • There are no negative examples for the fallback intent.
  • This text is annotated in some training phrases but not others.

Enable and disable automatic validation

By default, agent validation is automatically performed whenever the agent is trained. You can enable and disable this setting:

  1. Go to the Dialogflow ES Console.
  2. Select your agent.
  3. Click the settings icon next to the agent name.
  4. Select the ML Settings tab.
  5. Toggle the Agent Validation setting on (default) or off.

Agent validation screenshot

Agent validation page

To access the validation data for your agent:

  1. Go to the Dialogflow ES Console.
  2. Select your agent.
  3. Click Validation in the sidebar menu.

Agent validation screenshot

Validation results for intents and entities

When you visit either the intents list or the entities list pages, any intents or entities with validation errors show an error indicator next to the name.

Agent validation error screenshot

When you visit a page for a specific intent or entity that has validation errors, an error indicator is shown near the Save button.

Agent validation error screenshot

Clicking this button shows a list of errors for the intent or entity. By default, only errors with a severity of CRITICAL or ERROR are shown. You can toggle the types of severities to also show WARNING and INFO.

Agent validation error screenshot

You may also see error indicators for specific training phrases and parameters, and you can hover over these indicators to see the details.

Agent validation error screenshot

Validation results for versions and environments

When viewing versions and environments, each agent version may have unique validation errors. If any errors are present, the validation icon appears next to the version.

Agent validation error screenshot

Clicking the icon will show the details.

Agent validation error screenshot

Validation results with the API

In most cases, you will view validation results using the Dialogflow Console. However, you can access this data from the API as well.

REST

Call the getValidationResult method on the Agents type.

Before using any of the request data, make the following replacements:

  • PROJECT_ID: your Google Cloud project ID

HTTP method and URL:

GET https://dialogflow.googleapis.com/v2/projects/PROJECT_ID/agent/validationResult?languageCode=en

To send your request, expand one of these options:

You should receive a JSON response similar to the following:

{
  "validationErrors": [
    {
      "severity": "ERROR",
      "entries": [
        "projects/my-project/agent/intents/58b44b2d-4967-4a81-b017-12623dcd5d28/parameters/bf6fdf55-b862-4101-b5b1-36f1423629d0"
      ],
      "errorMessage": "Parameter 'test' has an empty value."
    },
    {
      "severity": "WARNING",
      "entries": [
        "projects/my-project/agent/intents/271e3808-3c91-4e6b-89e8-47951abcec8d"
      ],
      "errorMessage": "Intent 'app.current.update' does not have enough unique training phrases. Consider adding more different examples."
    },
    {
      "severity": "ERROR",
      "entries": [
        "projects/my-project/agent/intents/26e64b1b-eaa7-4ce2-be46-631a501fccbe/trainingPhrases/a650375e-083c-4bb5-9794-ba9453e51282",
        "projects/my-project/agent/intents/58b44b2d-4967-4a81-b017-12623dcd5d28/trainingPhrases/1d947780-22d3-4f80-8d7a-3f86efbf0be3"
      ],
      "errorMessage": "Multiple intents share training phrases which are too similar:\n - Intent 'app.notifications.open': training phrase 'open allo notifications settings'\n - Intent 'app.current.notifications.open': training phrase 'open notifications settings'"
    },
  ]
}

Severity States

Validation can show the following severity states:

Severity Description
INFO The agent doesn't follow the best practice.
WARNING The agent might not behave as expected.
ERROR The agent may experience partial failures.
CRITICAL The agent may completely fail.
SEVERITY_UNSPECIFIED Not specified. This value should never be used. (This severity is possible through the API)

Force validation

Agent validation results are available automatically whenever agent training is performed and completed. To force new validation results, you can disable automatic validation, and trigger it manually. See the Agent ML settings page on the console.

Handling many issues

When a large number of issues are found, you should consider the following:

  • Fix issues in small batches. If there are many similar issues, fixing one issue may fix similar issues after you retrain the agent.
  • At most 5000 issues are shown at a time. If you have over 5000 issues, you may not see a count reduction until less than 5000 remain.