Tools

Using tools, you can connect generative agents to external systems. These systems can augment the knowledge of generative agents and empower them to execute complex tasks efficiently.

You can use built-in tools or build customized tools to suit your requirements.

Built-in tools

Built-in tools are hosted by Google. You can activate these tools in generative agents without the need for manual configuration.

The supported built-in tools are:

  • Code Interpreter: A Google first party tool that combines the capability of code generation and code execution and allow the user to perform various tasks, including: data analysis, data visualization, text processing, solving equations or optimization problems.

Your agent is optimized to determine how and when these tools should be invoked, but you can provide additional examples to fit your use cases.

Examples should have a schema like the following:

{
  "toolUse": {
    "tool": "projects/PROJECT_ID/locations/LOCATION_ID/agents/AGENT_ID/tools/df-code-interpreter-tool",
    "action": "generate_and_execute",
    "inputParameters": [
      {
        "name": "generate_and_execute input",
        "value": "4 + 4"
      }
    ],
    "outputParameters": [
      {
        "name": "generate_and_execute output",
        "value": {
          "output_files": [
            {
              "name": "",
              "contents": ""
            }
          ],
          "execution_result": "8",
          "execution_error": "",
          "generated_code": "GENERATED_CODE"
        }
      }
    ]
  }
}

Customized tools

The following sections describe customized tools.

OpenAPI tools

A generative agent can connect to an external API using an OpenAPI tool by providing the OpenAPI schema. By default, the generative agent will call the API on your behalf. Alternatively, you can execute OpenAPI tools on the client side.

Example schema:

openapi: 3.0.0
info:
  title: Simple Pets API
  version: 1.0.0
servers:
  - url: 'https://api.pet-service-example.com/v1'
paths:
  /pets/{petId}:
    get:
      summary: Return a pet by ID.
      operationId: getPet
      parameters:
        - in: path
          name: petId
          required: true
          description: Pet id
          schema:
            type: integer
      responses:
        200:
          description: OK
  /pets:
    get:
      summary: List all pets
      operationId: listPets
      parameters:        
        - name: petName
          in: query
          required: false
          description: Pet name
          schema:
            type: string
        - name: label
          in: query
          description: Pet label
          style: form
          explode: true
          required: false
          schema:
            type: array
            items:
              type: string
        - name: X-OWNER
          in: header
          description: Optional pet owner provided in the HTTP header
          required: false
          schema:
            type: string
        - name: X-SESSION
          in: header
          description: Dialogflow session id
          required: false
          schema:
            $ref: "@dialogflow/sessionId"
      responses:
        '200':
          description: An array of pets
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/Pet'
    post:
      summary: Create a new pet
      operationId: createPet
      requestBody:
        description: Pet to add to the store
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Pet'
      responses:
        '201':
          description: Pet created
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Pet'
components:
  schemas:
    Pet:
      type: object
      required:
        - id
        - name        
      properties:
        id:
          type: integer
          format: int64
        name:
          type: string
        owner:
          type: string
        label:
          type: array
          items:
            type: string

You can optionally use the internal schema reference @dialogflow/sessionId as parameter schema type. With this parameter schema type, the Dialogflow session ID for the current conversation will be supplied as a parameter value. For example:

- name: X-SESSION
   in: header
   description: Dialogflow session id
   required: false
   schema:
     $ref: "@dialogflow/sessionId"

OpenAPI tool limitations

The following limitations apply:

  • Supported parameter types are path, query, header. The cookie parameter type is not supported yet.
  • Parameters defined by OpenAPI schema support the following data types: string, number, integer, boolean, array. The object type is not supported yet.
  • You currently can't specify query parameters in the console example editor.
  • Request and response body must be empty or JSON.

OpenAPI tool API authentication

The following authentication options are supported when calling an external API:

  • Dialogflow Service Agent auth
    • Dialogflow can generate an ID token or access token using Dialogflow Service Agent. The token is added in the authorization HTTP header when Dialogflow calls an external API.
    • An ID token can be used to access Cloud Functions and Cloud Run services after you grant the roles/cloudfunctions.invoker and roles/run.invoker roles to service-agent-project-number@gcp-sa-dialogflow.iam.gserviceaccount.com. If the Cloud Functions and Cloud Run services are in the same resource project, you don't need additional IAM permission to call them.
    • An access token can be used to access other Google Cloud APIs after you grant required roles to service-agent-project-number@gcp-sa-dialogflow.iam.gserviceaccount.com.
  • API key
    • You can configure API key authentication by providing the key name, request location (header or query string) and API key so that Dialogflow passes the API key in the request.
  • OAuth
    • OAuth Client Credential flow is supported for server-to-server authentication. Client ID, Client Secret and Token endpoint from OAuth provider need to be configured in Dialogflow. Dialogflow exchanges an OAuth access token and passes it in the auth header of the request.
    • For other OAuth flows, you need to use the Function Tool to integrate with your own sign-in UI to exchange the token.
  • Mutual TLS authentication
  • Custom CA certificate

Data store tools

Data store tools can be used by a generative agent for answers to end-user's questions from your data stores. You can set up one data store of each type per tool, and the tool will query each of these data stores for answers. By default, the generative agent will call the data store tool on your behalf. Alternatively, you can execute data store tools on the client side.

The data store type can be one of the following:

  • PUBLIC_WEB: A data store that contains public web content.
  • UNSTRUCTURED: A data store that contains unstructured private data.
  • STRUCTURED: A data store that contains structured data (for example FAQ).

The following example shows how to reference a data store:

"dataStoreConnections": [
  {
    "dataStoreType": "PUBLIC_WEB",
    "dataStore": "projects/PROJECT_NUMBER/locations/LOCATION_ID/collections/default_collection/dataStores/DATASTORE_ID"
  },
  {
    "dataStoreType": "UNSTRUCTURED",
    "dataStore": "projects/PROJECT_NUMBER/locations/LOCATION_ID/collections/default_collection/dataStores/DATASTORE_ID"
  },
  {
    "dataStoreType": "STRUCTURED",
    "dataStore": "projects/PROJECT_NUMBER/locations/LOCATION_ID/collections/default_collection/dataStores/DATASTORE_ID"
  }
]

Data store tool responses might also contain snippets about the content source that was used to generate the response. The generative agent can further provide instructions on how to proceed with the answer from the data stores or how to respond when there is no answer. Examples can be used to further enhance the generative agent behavior. The example should have the following schemas:

{
  "toolUse": {
    "tool": "projects/PROJECT_ID/locations/LOCATION_ID/agents/AGENT_ID/tools/TOOL_ID",
    "action": "TOOL_DISPLAY_NAME",
    "inputParameters": [
      {
        "name": "TOOL_DISPLAY_NAME input",
        "value": {
          "query": "QUERY"
        }
      }
    ],
    "outputParameters": [
      {
        "name": "TOOL_DISPLAY_NAME output",
        "value": {
          "answer": "ANSWER",
          "snippets": [
            {
              "title": "TITLE",
              "text": "TEXT_FROM_DATASTORE",
              "uri": "URI_OF_DATASTORE"
            }
          ]
        }
      }
    ]
  }
}

Function tools

If you have functionality accessible by your client code, but not accessible by OpenAPI tools, you can use function tools. Function tools are always executed on the client side, not by the generative agent.

The process is as follows:

  1. Your client code sends a detect intent request.
  2. The generative agent detects that a function tool is required, and the detect intent response contains the name of the tool along with input arguments. This session is paused until another detect intent request is received with the tool result.
  3. Your client code calls the tool.
  4. Your client code sends another detect intent request that provides the tool result as output arguments.

The following example shows the input and output schema of a function tool:

{
  "type": "object",
  "properties": {
    "location": {
      "type": "string",
      "description": "The city and state, for example, San Francisco, CA"
    }
  },
  "required": [
    "location"
  ]
}
{
  "type": "object",
  "properties": {
    "temperature": {
      "type": "number",
      "description": "The temperature"
    }
  }
}

The following example shows the initial detect intent request and response using REST:

HTTP method and URL:
POST https://REGION_ID-dialogflow.googleapis.com/v3/projects/PROJECT_ID/locations/LOCATION_ID/agents/AGENT_ID/sessions/SESSION_ID:detectIntent
{
  "queryInput": {
    "text": {
      "text": "what is the weather in Mountain View"
    },
    "languageCode": "en"
  }
}
{
  "queryResult": {
    "text": "what is the weather in Mountain View",
    "languageCode": "en",
    "responseMessages": [
      {
        "source": "VIRTUAL_AGENT",
        "toolCall": {
          "tool": "<tool-resource-name>",
          "action": "get-weather-tool",
          "inputParameters": {
            "location": "Mountain View"
          }
        }
      }
    ]
  }
}

The following example shows the second detect intent request, which provides the tool result:

{
  "queryInput": {
    "toolCallResult": {
      "tool": "<tool-resource-name>",
      "action": "get-weather-tool",
      "outputParameters": {
        "temperature": 28.0
      }
    },
    "languageCode": "en"
  }
}

Client side execution

Like function tools, OpenAPI and data store tools can be executed on the client side by applying an API override when interacting with the session.

For example:

DetectIntentRequest {
  ...
  query_params {
    playbook_state_override {
      playbook_execution_mode: ALWAYS_CLIENT_EXECUTION
    }
  }
  ...
}

The process is as follows:

  1. Your client code sends a detect intent request that specifies client execution.
  2. The generative agent detects that a tool is required, and the detect intent response contains the name of the tool along with input arguments. This session is paused until another detect intent request is received with the tool result.
  3. Your client code calls the tool.
  4. Your client code sends another detect intent request that provides the tool result as output arguments.