Define data agent context for BigQuery data sources

Authored context is guidance that data agent owners can provide to shape the behavior of a data agent and to refine the API's responses. Effective authored context provides your Conversational Analytics API data agents with useful context for answering questions about your data sources.

This page describes how to provide authored context for BigQuery data sources. For BigQuery data sources, you can provide authored context through a combination of structured context and system instructions. Whenever possible, provide context through structured context fields. You can then use the system_instruction parameter for supplemental guidance that isn't covered by the structured fields.

While both structured context fields and system instructions are optional, providing robust context enables the agent to give more accurate and relevant responses.

After you define the structured fields and system instructions that make up your authored context, you can provide that context to the API in one of the following calls:

Define structured context fields

This section describes how to provide context to a data agent by using structured context fields. You can provide the following information to an agent as structured context:

Table-level structured context

Use the tableReferences key to provide an agent with details about the specific tables that are available for answering questions. For each table reference, you can use the following structured context fields to define a table's schema:

  • description: A summary of the table's content and purpose
  • synonyms: A list of alternative terms that can be used to refer to the table
  • tags: A list of keywords or tags that are associated with the table

The following examples show how to provide these properties as structured context within direct HTTP requests and with the Python SDK.

HTTP

In a direct HTTP request, you provide these table-level properties within the schema object for the relevant table reference. For a complete example of how to structure the full request payload, see Connect to BigQuery data.

"tableReferences": [
  {
    "projectId": "bigquery-public-data",
    "datasetId": "thelook_ecommerce",
    "tableId": "orders",
    "schema": {
        "description": "Data for orders in The Look, a fictitious ecommerce store.",
        "synonyms": ["sales"],
        "tags": ["sale", "order", "sales_order"]
    }
  },
  {
    "projectId": "bigquery-public-data",
    "datasetId": "thelook_ecommerce",
    "tableId": "users",
    "schema": {
        "description": "Data for users in The Look, a fictitious ecommerce store.",
        "synonyms": ["customers"],
        "tags": ["user", "customer", "buyer"]
    }
  }
]

Python SDK

When you use the Python SDK, you can define these table-level properties on the schema property of a BigQueryTableReference object. The following example shows how to create table reference objects that provide context for the orders and users tables. For a complete example of how to build and use table reference objects, see Connect to BigQuery data.

# Define context for the 'orders' table
bigquery_table_reference_1 = geminidataanalytics.BigQueryTableReference()
bigquery_table_reference_1.project_id = "bigquery-public-data"
bigquery_table_reference_1.dataset_id = "thelook_ecommerce"
bigquery_table_reference_1.table_id = "orders"

bigquery_table_reference_1.schema = geminidataanalytics.Schema()
bigquery_table_reference_1.schema.description = "Data for orders in The Look, a fictitious ecommerce store."
bigquery_table_reference_1.schema.synonyms = ["sales"]
bigquery_table_reference_1.schema.tags = ["sale", "order", "sales_order"]

# Define context for the 'users' table
bigquery_table_reference_2 = geminidataanalytics.BigQueryTableReference()
bigquery_table_reference_2.project_id = "bigquery-public-data"
bigquery_table_reference_2.dataset_id = "thelook_ecommerce"
bigquery_table_reference_2.table_id = "users"

bigquery_table_reference_2.schema = geminidataanalytics.Schema()
bigquery_table_reference_2.schema.description = "Data for users in The Look, a fictitious ecommerce store."
bigquery_table_reference_2.schema.synonyms = ["customers"]
bigquery_table_reference_2.schema.tags = ["user", "customer", "buyer"]

Column-level structured context

The fields key, which is nested within a table reference's schema object, takes a list of field objects to describe individual columns. Not all fields need additional context; however, for commonly used fields, including additional details can help improve the agent's performance.

For each field object, you can use the following structured context fields to define a column's fundamental properties:

  • description: A brief description of the column's contents and purpose
  • synonyms: A list of alternative terms that can be used to refer to the column
  • tags: A list of keywords or tags that are associated with the column

The following examples show how you can provide these properties as structured context for the status field within the orders table and for the first_name field within the users table with direct HTTP requests and with the Python SDK.

HTTP

In a direct HTTP request, you can define these column-level properties by providing a list of fields objects within the schema object of a table reference.

"tableReferences": [
  {
    "projectId": "bigquery-public-data",
    "datasetId": "thelook_ecommerce",
    "tableId": "orders",
    "schema": {
      "fields": [{
          "name": "status",
          "description": "The current status of the order.",
      }]
    }
  },
  {
    "projectId": "bigquery-public-data",
    "datasetId": "thelook_ecommerce",
    "tableId": "users",
    "schema": {
      "fields": [{
          "name": "first_name",
          "description": "The first name of the user.",
          "tags": "person",
      }]
    }
  }
]

Python SDK

When you use the Python SDK, you can define these column-level properties by assigning a list of Field objects to the fields property of a table's schema property.

# Define column context for the 'orders' table
bigquery_table_reference_1.schema.fields = [
    geminidataanalytics.Field(
        name="status",
        description="The current status of the order.",
    )
]

# Define column context for the 'users' table
bigquery_table_reference_2.schema.fields = [
    geminidataanalytics.Field(
        name="first_name",
        description="The first name of the user.",
        tags=["person"],
    )
]

Example queries

The example_queries key takes a list of example_query objects to define natural language queries that help the agent provide more accurate and relevant responses to common or important questions. By providing the agent with both a natural language question and its corresponding SQL query, you can guide the agent to provide higher quality and more consistent results.

For each example_query object, you can provide the following fields to define a natural language question and its corresponding SQL query:

  • natural_language_question: The natural language question that a user might ask
  • sql_query: The SQL query that corresponds to the natural language question

The following examples show how to provide example queries for the orders table both with direct HTTP requests and with the Python SDK.

HTTP

In a direct HTTP request, provide a list of example_query objects in the example_queries field. Each object must contain a naturalLanguageQuestion key and a corresponding sqlQuery key.

"example_queries": [
  {
    "naturalLanguageQuestion": "How many orders are there?",
    "sqlQuery": "SELECT COUNT(*) FROM `bigquery-public-data.thelook_ecommerce.orders`"
  },
  {
    "naturalLanguageQuestion": "How many orders were shipped?",
    "sqlQuery": "SELECT COUNT(*) FROM `bigquery-public-data.thelook_ecommerce.orders` WHERE status = 'shipped'"
  }
]

Python SDK

When using the Python SDK, you can provide a list of ExampleQuery objects. For each object, provide values for the natural_language_question and sql_query parameters.

example_queries = [
    geminidataanalytics.ExampleQuery(
        natural_language_question="How many orders are there?",
        sql_query="SELECT COUNT(*) FROM `bigquery-public-data.thelook_ecommerce.orders`",
    ),
    geminidataanalytics.ExampleQuery(
        natural_language_question="How many orders were shipped?",
        sql_query="SELECT COUNT(*) FROM `bigquery-public-data.thelook_ecommerce.orders` WHERE status = 'shipped'",
    )
]

Define additional context in system instructions

You can use the system_instruction parameter to provide supplemental guidance for context that isn't supported by structured context fields. By providing this additional guidance, you can help the agent better understand the context of your data and use case.

System instructions consist of a series of key components and objects that provide the data agent with details about the data source and guidance about the agent's role when answering questions. You can provide system instructions to the data agent in the system_instruction parameter as a YAML-formatted string.

The following template shows a suggested YAML structure for the string, which you can provide to the system_instruction parameter for a BigQuery data source, including available keys and expected data types. While this template provides a suggested structure with important components for defining system instructions, it does not include all possible system instruction formats.

- system_instruction: str # A description of the expected behavior of the agent. For example: You are a sales agent.
- tables: # A list of tables to describe for the agent.
  - table: # Details about a single table that is relevant for the agent.
    - name: str # The name of the table.
    - fields: # Details about columns (fields) within the table.
      - field: # Details about a single column within the current table.
        - name: str # The name of the column.
        - aggregations: list[str] # Commonly used or default aggregations for the column.
  - relationships: # A list of join relationships between tables.
    - relationship: # Details about a single join relationship.
      - name: str # The name of this join relationship.
      - description: str # A description of the relationship.
      - relationship_type: str # The join relationship type: one-to-one, one-to-many, many-to-one, or many-to-many.
      - join_type: str # The join type: inner, outer, left, right, or full.
      - left_table: str # The name of the left table in the join.
      - right_table: str # The name of the right table in the join.
      - relationship_columns: # A list of columns that are used for the join.
        - left_column: str # The join column from the left table.
        - right_column: str # The join column from the right table.
- glossaries: # A list of definitions for glossary business terms, jargon, and abbreviations.
  - glossary: # The definition for a single glossary item.
    - term: str # The term, phrase, or abbreviation to define.
    - description: str # A description or definition of the term.
    - synonyms: list[str] # Alternative terms for the glossary entry.
- additional_descriptions: # A list of any other general instructions or content.
  - text: str # Any additional general instructions or context not covered elsewhere.

The following sections contain examples of key components of system instructions:

system_instruction

Use the system_instruction key to define the agent's role and persona. This initial instruction sets the tone and style for the API's responses and helps the agent understand its core purpose.

For example, you can define an agent as a sales analyst for a fictitious ecommerce store as follows:

- system_instruction: >-
    You are an expert sales analyst for a fictitious ecommerce store. You will answer questions about sales, orders, and customer data. Your responses should be concise and data-driven.

tables

While you define a table's fundamental properties (such as its description and synonyms) as structured context, you can also use the tables key within system instructions to provide supplemental business logic. For BigQuery data sources, this includes using the fields key to define default aggregations for specific columns.

The following sample YAML code block shows how you can use the tables key within your system instructions to nest fields that provide supplemental guidance for the table bigquery-public-data.thelook_ecommerce.orders:

- tables:
  - table:
    - name: bigquery-public-data.thelook_ecommerce.orders
    - fields:
      - field:
        - name: num_of_items
        - aggregations: 'sum, avg'

relationships

The relationships key in your system instructions contains a list of join relationships between tables. By defining join relationships, you can help the agent understand how to join data from multiple tables when answering questions.

As an example, you can define an orders_to_user relationship between the bigquery-public-data.thelook_ecommerce.orders table and the bigquery-public-data.thelook_ecommerce.users table as follows:

- relationships:
  - relationship:
    - name: orders_to_user
    - description: >-
        Connects customer order data to user information with the user_id and id fields to allow an aggregated view of sales by customer demographics.
    - relationship_type: many-to-one
    - join_type: left
    - left_table: bigquery-public-data.thelook_ecommerce.orders
    - right_table: bigquery-public-data.thelook_ecommerce.users
    - relationship_columns:
      - left_column: user_id
      - right_column: id

glossaries

The glossaries key in your system instructions lists definitions for business terms, jargon, and abbreviations that are relevant to your data and use case. By providing glossary definitions, you can help the agent accurately interpret and answer questions that use specific business language.

As an example, you can define terms like common business statuses and "OMPF" according to your specific business context as follows:

- glossaries:
  - glossary:
    - term: complete
    - description: Represents an order status where the order has been completed.
    - synonyms: 'finish, done, fulfilled'
  - glossary:
    - term: shipped
    - description: Represents an order status where the order has been shipped to the customer.
  - glossary:
    - term: returned
    - description: Represents an order status where the customer has returned the order.
  - glossary:
    - term: OMPF
    - description: Order Management and Product Fulfillment

additional_descriptions

Use the additional_descriptions key to provide any general instructions or context that doesn't fit into other structured context or system instruction fields. By providing additional descriptions in your system instructions, you can help the agent better understand the context of your data and use case.

As an example, you can use the additional_descriptions key to provide information about your organization as follows:

- additional_descriptions:
  - text: All the sales data pertains to The Look, a fictitious ecommerce store.
  - text: 'Orders can be of three categories: food, clothes, and electronics.'

Example: Authored context for a sales agent

The following example for a fictitious sales analyst agent demonstrates how to provide authored context by using a combination of structured context and system instructions.

Example: Structured context

You can provide structured context with details about tables, columns, and example queries to guide the agent, as shown in the following HTTP and Python SDK examples.

HTTP

The following example shows how to define structured context in an HTTP request:

{
  "bq": {
    "tableReferences": [
      {
        "projectId": "bigquery-public-data",
        "datasetId": "thelook_ecommerce",
        "tableId": "orders",
        "schema": {
          "description": "Data for orders in The Look, a fictitious ecommerce store.",
          "synonyms": ["sales"],
          "tags": ["sale", "order", "sales_order"],
          "fields": [
            {
              "name": "status",
              "description": "The current status of the order."
            },
            {
              "name": "num_of_items",
              "description": "The number of items in the order."
            }
          ]
        }
      },
      {
        "projectId": "bigquery-public-data",
        "datasetId": "thelook_ecommerce",
        "tableId": "users",
        "schema": {
          "description": "Data for users in The Look, a fictitious ecommerce store.",
          "synonyms": ["customers"],
          "tags": ["user", "customer", "buyer"],
          "fields": [
            {
              "name": "first_name",
              "description": "The first name of the user.",
              "tags": ["person"]
            },
            {
              "name": "last_name",
              "description": "The last name of the user.",
              "tags": ["person"]
            },
            {
              "name": "age_group",
              "description": "The age demographic group of the user."
            },
            {
              "name": "email",
              "description": "The email address of the user.",
              "tags": ["contact"]
            }
          ]
        }
      }
    ]
  },
  "example_queries": [
    {
      "naturalLanguageQuestion": "How many orders are there?",
      "sqlQuery": "SELECT COUNT(*) FROM `bigquery-public-data.thelook_ecommerce.orders`"
    },
    {
      "naturalLanguageQuestion": "How many orders were shipped?",
      "sqlQuery": "SELECT COUNT(*) FROM `bigquery-public-data.thelook_ecommerce.orders` WHERE status = 'shipped'"
    },
    {
      "naturalLanguageQuestion": "How many unique customers are there?",
      "sqlQuery": "SELECT COUNT(DISTINCT id) FROM `bigquery-public-data.thelook_ecommerce.users`"
    },
    {
      "naturalLanguageQuestion": "How many users in the 25-34 age group have a cymbalgroup email address?",
      "sqlQuery": "SELECT COUNT(DISTINCT id) FROM `bigquery-public-data.thelook_ecommerce.users` WHERE users.age_group = '25-34' AND users.email LIKE '%@cymbalgroup.com'"
    }
  ]
}

Python SDK

The following example shows how to define structured context with the Python SDK:

# Define context for the 'orders' table
bigquery_table_reference_1 = geminidataanalytics.BigQueryTableReference()
bigquery_table_reference_1.project_id = "bigquery-public-data"
bigquery_table_reference_1.dataset_id = "thelook_ecommerce"
bigquery_table_reference_1.table_id = "orders"

bigquery_table_reference_1.schema = geminidataanalytics.Schema()
bigquery_table_reference_1.schema.description = "Data for orders in The Look, a fictitious ecommerce store."
bigquery_table_reference_1.schema.synonyms = ["sales"]
bigquery_table_reference_1.schema.tags = ["sale", "order", "sales_order"]
bigquery_table_reference_1.schema.fields = [
    geminidataanalytics.Field(
        name="status",
        description="The current status of the order.",
    ),
    geminidataanalytics.Field(
        name="num_of_items",
        description="The number of items in the order."
    )
]

# Define context for the 'users' table
bigquery_table_reference_2 = geminidataanalytics.BigQueryTableReference()
bigquery_table_reference_2.project_id = "bigquery-public-data"
bigquery_table_reference_2.dataset_id = "thelook_ecommerce"
bigquery_table_reference_2.table_id = "users"

bigquery_table_reference_2.schema = geminidataanalytics.Schema()
bigquery_table_reference_2.schema.description = "Data for users in The Look, a fictitious ecommerce store."
bigquery_table_reference_2.schema.synonyms = ["customers"]
bigquery_table_reference_2.schema.tags = ["user", "customer", "buyer"]
bigquery_table_reference_2.schema.fields = [
    geminidataanalytics.Field(
        name="first_name",
        description="The first name of the user.",
        tags=["person"],
    ),
    geminidataanalytics.Field(
        name="last_name",
        description="The last name of the user.",
        tags=["person"],
    ),
    geminidataanalytics.Field(
        name="age_group",
        description="The age demographic group of the user.",
    ),
    geminidataanalytics.Field(
        name="email",
        description="The email address of the user.",
        tags=["contact"],
    )
]

# Define example queries
example_queries = [
  geminidataanalytics.ExampleQuery(
      natural_language_question="How many orders are there?",
      sql_query="SELECT COUNT(*) FROM `bigquery-public-data.thelook_ecommerce.orders`",
  ),
  geminidataanalytics.ExampleQuery(
      natural_language_question="How many orders were shipped?",
      sql_query="SELECT COUNT(*) FROM `bigquery-public-data.thelook_ecommerce.orders` WHERE status = 'shipped'",
  ),
  geminidataanalytics.ExampleQuery(
      natural_language_question="How many unique customers are there?",
      sql_query="SELECT COUNT(DISTINCT id) FROM `bigquery-public-data.thelook_ecommerce.users`",
  ),
  geminidataanalytics.ExampleQuery(
      natural_language_question="How many users in the 25-34 age group have a cymbalgroup email address?",
      sql_query="SELECT COUNT(DISTINCT id) FROM `bigquery-public-data.thelook_ecommerce.users` WHERE users.age_group = '25-34' AND users.email LIKE '%@cymbalgroup.com'",
  )
]

Example: System instructions

The following system instructions supplement structured context by defining the agent's persona and providing guidance that's not supported by structured fields, such as relationship definitions, glossary terms, additional descriptions, and supplemental orders table details. In this example, because the users table is fully defined with structured context, it doesn't need to be redefined in the system instructions.

- system_instruction: >-
    You are an expert sales analyst for a fictitious ecommerce store. You will answer questions about sales, orders, and customer data. Your responses should be concise and data-driven.
- tables:
    - table:
        - name: bigquery-public-data.thelook_ecommerce.orders
        - fields:
            - field:
                - name: num_of_items
                - aggregations: 'sum, avg'
- relationships:
    - relationship:
        - name: orders_to_user
        - description: >-
            Connects customer order data to user information with the user_id and id fields.
        - relationship_type: many-to-one
        - join_type: left
        - left_table: bigquery-public-data.thelook_ecommerce.orders
        - right_table: bigquery-public-data.thelook_ecommerce.users
        - relationship_columns:
            - left_column: user_id
            - right_column: id
- glossaries:
    - glossary:
        - term: complete
        - description: Represents an order status where the order has been completed.
        - synonyms: 'finish, done, fulfilled'
    - glossary:
        - term: OMPF
        - description: Order Management and Product Fulfillment
- additional_descriptions:
    - text: All the sales data pertains to The Look, a fictitious ecommerce store.