Check grounding

Preview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This feature is in Public preview.

This page describes how to use the check grounding API to determine how grounded a piece of text (called an answer candidate) is in a given set of reference texts (called facts).

The check grounding API returns an overall support score of 0 to 1, which indicates how much the answer candidate agrees with the given facts. The response also includes citations to the facts supporting each claim in the answer candidate.

Perfect grounding requires that every claim in the answer candidate must be supported by one or more of the given facts. In other words, the claim is wholly entailed by the facts. If the claim is only partially entailed, it is not considered grounded. For example, the claim "Google was founded by Larry Page and Sergey Brin in 1975" is only partially correct—the names of the founders are correct but the date is wrong—and as such the whole claim is considered ungrounded. In this version of the check grounding API, a sentence is considered a single claim.

You can also enable anti-citations, as an Experimental feature, to get a contradiction score, which indicates how much the answer candidate contradicts the given facts. The response also include anti-citations to the contradicting facts for each claim.

You can use the check grounding API for checking any piece of text. It could be a human-generated blurb or a machine-generated response. A typical use case would be to check an LLM-generated response with respect to a given set of facts. Among other things, the citations generated by the API would help distinguish hallucinated claim in the response from grounded claims.

Before you begin

The check grounding API is a Public preview feature. The anti-citations feature that generates the contradiction score is an Experimental feature. To try the anti-citations feature, contact your Google account team and ask to be added to the allowlist.

Terms defined and explained

Before you use the check grounding API, it helps to understand the inputs and outputs, and how to structure your grounding facts for best results.

Input data

The check grounding API requires the following inputs in the request.

• Answer candidate: An answer candidate can be any piece of text whose grounding you want to check. For example, in the context of Vertex AI Search, the answer candidate might be the generated search summary that answers a query. The API would then determine how grounded the summary is in the input facts. An answer candidate can have a maximum length of 1024 characters.

• Facts: A set of text segments to be used as references for grounding. A set of metadata attributes (key-value pairs) can be supplied with each text segment. For example, "Author" and "Title" are typical attribute keys.

  The service supports up to 200 facts, each with a maximum of 10k characters.

  Google recommends against supplying one very large fact that contains all of the information. Instead, you can get better results by breaking large facts into smaller facts and supplying appropriate attributes for the smaller facts. For example, you can break up a large fact by title, author, or URL, and supply this information in attributes.

• Citation threshold: A float value from 0 to 1 that controls the confidence for the citations that support the answer candidate. A higher threshold imposes stricter confidence. Therefore, a higher threshold yields fewer but stronger citations.

Output data

The check grounding API returns the following for an answer candidate:

• Support score: The support score is a number from 0 to 1 that indicates how grounded an answer candidate is in the provided set of facts. It loosely approximates the fraction of claims in the answer candidate that were found to be grounded in one or more of the given facts.

• Cited chunks: Cited chunks are portions of the input facts that support the answer candidate.

• Claims and citations: The claims and citations connect a claim (typically a sentence) of the answer candidate to one or more of the cited chunks that corroborate the claim.

Obtain a support score for an answer candidate

To find out how grounded an answer candidate is in a set of facts, follow these steps:

1. Prepare your set of facts. For more information and examples, see Terms defined and explained.

2. Call the check method using the following curl command:

   curl -X POST \
   -H "Authorization: Bearer $(gcloud auth print-access-token)" \
   -H "Content-Type: application/json" \
   "https://discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/global/groundingConfigs/default_grounding_config:check" \
   -d '{
   "answerCandidate": "CANDIDATE",
   "facts": [
   {
   "factText": "TEXT_0",
   "attributes": {"ATTRIBUTE_A": "VALUE_A0","ATTRIBUTE_B": "VALUE_B0"}
   },
   {
   "factText": "TEXT_1",
   "attributes": {"ATTRIBUTE_A": "VALUE_A1","ATTRIBUTE_B": "VALUE_B1"}
   },
   {
   "factText": "TEXT_2",
   "attributes": {"ATTRIBUTE_A": "VALUE_A2","ATTRIBUTE_B": "VALUE_B2"}
   },
   ],
   "groundingSpec": {
   "citationThreshold": "CITATION_THRESHOLD"
   }
   }'
   

   Replace the following:

   • PROJECT_ID: the project number or ID of your Google Cloud project.

   • CANDIDATE: the answer candidate string for which you want to get a support score—for example, Titanic was directed by James Cameron. It was released in 1997. An answer candidate can have a maximum length of 1024 characters.

   • TEXT: the text segment to be used for grounding—for example, Titanic is a 1997 American epic... Academy Awards. (See the full text in Examples of facts.)

   • ATTRIBUTE: the name of a metadata attribute associated with the fact—for example, author or title. This is a user-defined label to add more information to the fact text. For example, if the fact text Toronto is the capital of Ontario has an author attribute with its value as Wikipedia, then the following claims are considered grounded in the fact:

     • Wikipedia cites that Toronto is the capital of Ontario
     • Toronto is the capital of Ontario

     However, the claim that Government of Ontario claims that Toronto is the capital of Ontario is not as grounded as the first two claims.

   • VALUE: the value for the attribute—for example, Simple Wikipedia or Titanic (1997 film).

   • CITATION_THRESHOLD: a float value from 0 through 1 that determines whether a fact must be cited for a claim in the answer candidate. A higher threshold leads to fewer but strong citations and a lower threshold leads to more but weak citations. If unset, the default threshold value is 0.6.

Examples of facts

The following are a couple of examples of facts and their attributes. These examples are to help you understand the grounding response and the format of the curl command.

• Fact 0

  • Text: "Titanic is a 1997 American epic romantic disaster movie. It was directed, written, and co-produced by James Cameron. The movie is about the 1912 sinking of the RMS Titanic. It stars Kate Winslet and Leonardo DiCaprio. The movie was released on December 19, 1997. It received positive critical reviews. The movie won 11 Academy Awards, and was nominated for fourteen total Academy Awards."

  • Attributes: {"Author": "Simple Wikipedia"}

• Fact 1

  • Text: "James Cameron's "Titanic" is an epic, action-packed romance set against the ill-fated maiden voyage of the R.M.S. Titanic; the pride and joy of the White Star Line and, at the time, the largest moving object ever built. She was the most luxurious liner of her era -- the "ship of dreams" -- which ultimately carried over 1,500 people to their death in the ice cold waters of the North Atlantic in the early hours of April 15, 1912."

  • Attributes: {"Author": "Rotten Tomatoes"}

Example request

After preparing the facts, you can send the following request, replacing the CANDIDATE field with different strings whose grounding you want to check.

    curl -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json" \
    "https://discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/global/groundingConfigs/default_grounding_config:check" \
    -d '{
    "answerCandidate": "CANDIDATE",
    "facts": [
     {
      "factText": "Titanic is a 1997 American epic romantic disaster movie. It was directed, written, and co-produced by James Cameron. The movie is about the 1912 sinking of the RMS Titanic. It stars Kate Winslet and Leonardo DiCaprio. The movie was released on December 19, 1997. It received positive critical reviews. The movie won 11 Academy Awards, and was nominated for fourteen total Academy Awards.",
      "attributes": {"author":"Simple Wikipedia"}
     },
     {
      "factText": "James Cameron's \"Titanic\" is an epic, action-packed romance set against the ill-fated maiden voyage of the R.M.S. Titanic; the pride and joy of the White Star Line and, at the time, the largest moving object ever built. She was the most luxurious liner of her era -- the \"ship of dreams\" -- which ultimately carried over 1,500 people to their death in the ice cold waters of the North Atlantic in the early hours of April 15, 1912.",
      "attributes": {"author":"Simple Wikipedia"}
     }
    ],
    "groundingSpec": {
      "citationThreshold": "0.6"
    }
    }'

Note: The double quotes are escaped with the backslash character, and the apostrophe is replaced by its HTML character entity.

Examples of answer candidates and grounding responses

The following table shows examples of different answer candidates and responses when you send the example request, based on the example facts.

Answer candidate	Check grounding response
Titanic was directed by James Cameron. It was released in 1997.	Support score: 0.99 Cited chunks: [From FACT 0]....Titanic is a 1997 American epic romantic disaster movie. It was directed, written, and co-produced by James Cameron. It stars Kate Winslet and Leonardo DiCaprio. The movie was released on December 19, 1997. It received positive critical reviews. The movie won 11 Academy Awards, and was nominated for fourteen total Academy Awards..... [From FACT 1]...Titanic is an epic, action-packed romance set against the ill-fated maiden voyage of the R.M.S. Titanic; the pride and joy of the White Star Line and, at the time, the largest moving object ever built. She was the most luxurious liner of her era which ultimately carried over 1,500 people to their death in the ice cold waters of the North Atlantic in the early hours of April 15, 1912.... Claims and citations: 0. Titanic was directed by James Cameron. Citations: [0] 1. It was released in 1997. Citations: [1]
Titanic was directed by James Cameron. It was based on the sinking of the RMS Titanic that led to the death of 1500 people.	Support score: 0.95 Cited chunks: [From FACT 0]....Titanic is a 1997 American epic romantic disaster movie. It was directed, written, and co-produced by James Cameron. It stars Kate Winslet and Leonardo DiCaprio. The movie was released on December 19, 1997. It received positive critical reviews. The movie won 11 Academy Awards, and was nominated for fourteen total Academy Awards..... [From FACT 1]...Titanic is an epic, action-packed romance set against the ill-fated maiden voyage of the R.M.S. Titanic; the pride and joy of the White Star Line and, at the time, the largest moving object ever built. She was the most luxurious liner of her era which ultimately carried over 1,500 people to their death in the ice cold waters of the North Atlantic in the early hours of April 15, 1912.... Claims and citations: 0. Cited chunks: Titanic was directed by James Cameron. Citations: [0] 1. Cited chunks: It was based on the sinking of the RMS Titanic that led to the death of 1500 people. Citations: [1, 2]
Titanic was directed by James Cameron. It starred Brad Pitt and Kate Winslet	Support score: 0.54 Cited chunks: [From FACT 0]....Titanic is a 1997 American epic romantic disaster movie. It was directed, written, and co-produced by James Cameron. It stars Kate Winslet and Leonardo DiCaprio. The movie was released on December 19, 1997. It received positive critical reviews. The movie won 11 Academy Awards, and was nominated for fourteen total Academy Awards..... [From FACT 1]...Titanic is an epic, action-packed romance set against the ill-fated maiden voyage of the R.M.S. Titanic; the pride and joy of the White Star Line and, at the time, the largest moving object ever built. She was the most luxurious liner of her era which ultimately carried over 1,500 people to their death in the ice cold waters of the North Atlantic in the early hours of April 15, 1912.... Claims and citations: 0. Titanic was directed by James Cameron. Citations: [0] 1. It starred Brad Pitt and Kate Winslet Citations: [] Note: Even though Kate Winslet starred in the movie, because the claim "It starred Brad Pitt and Kate Winslet" is not wholly true, it gets no citations. In this case, you can call the method with anti-citations enabled to give you a contradiction score. For more information, see Obtain a contradiction score for an answer candidate.

Note: The raw input and output from the API calls don't look exactly like this. For example, in addition to the claim text the method also returns the start and end positions of where the claim text appears in the answer candidate. For more information, see Obtain a support score for an answer candidate.

Obtain a contradiction score for an answer candidate

Experimental

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

Along with the support score, you can also obtain a contradiction score. The contradiction score loosely approximates the fraction of claims that contradict the provided facts. This is an Experimental feature.

To obtain the contradiction score, follow these steps:

1. Prepare your set of facts. For more information and examples, see Terms defined and explained.

2. Call the check method, using the following curl command:

   curl -X POST \
   -H "Authorization: Bearer $(gcloud auth print-access-token)" \
   -H "Content-Type: application/json" \
   "https://discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/global/groundingConfigs/default_grounding_config:check" \
   -d '{
   "answerCandidate": "CANDIDATE",
   "facts": [
    {
     "factText": "TEXT_0",
     "attributes": {"ATTRIBUTE_A": "VALUE_A0","ATTRIBUTE_B": "VALUE_B0"}
    },
    {
     "factText": "TEXT_1",
     "attributes": {"ATTRIBUTE_A": "VALUE_A1","ATTRIBUTE_B": "VALUE_B1"}
    },
    {
     "factText": "TEXT_2",
     "attributes": {"ATTRIBUTE_A": "VALUE_A2","ATTRIBUTE_B": "VALUE_B2"}
    },
   ],
   "groundingSpec": {
     "citationThreshold": "CITATION_THRESHOLD",
     "enableAntiCitation": "ENABLE_ANTI_CITATION",
     "antiCitationThreshold": "ANTI_CITATION_THRESHOLD",
   }
   }'
   

   Replace the following:

   • PROJECT_ID: the project number or ID of your Google Cloud project.

   • CANDIDATE: the answer candidate string for which you want to get a support score—for example, Titanic was directed by James Cameron. It was released in 1997. An answer candidate can have a maximum length of 1024 characters.

   • TEXT: the text segment to be used for grounding—for example, Titanic is a 1997 American epic... Academy Awards. (See the full text in Examples of facts.)

   • ATTRIBUTE: the name of a metadata attribute associated with the fact—for example, author or title. It is a user-defined label to add more information to the fact text. For example, if the fact text Toronto is the capital of Ontario has an author attribute with its value as Wikipedia, then the following claims are well-grounded:

     • Wikipedia cites that Toronto is the capital of Ontario
     • Toronto is the capital of Ontario

     However, the claim that Government of Ontario claims that Toronto is the capital of Ontario is not as well-grounded.

   • VALUE: the value for the attribute—for example, Simple Wikipedia or Titanic (1997 film).

   • CITATION_THRESHOLD: a float value from 0 through 1 that determines whether a fact must be cited for a claim in the answer candidate. A higher threshold leads to fewer but strong citations to support the claim and a lower threshold leads to more but weak citations to support the claim. If unset, the default threshold value is 0.6.

   • ENABLE_ANTI_CITATION: a boolean value. Set this field to true to enable the experimental feature to evaluate the contradiction score. Either remove this field or set this field to false to turn off this feature.

   • ANTI_CITATION_THRESHOLD: a float value from 0 through 1 that determines whether a fact must be contradicted for a claim in the answer candidate. A higher threshold leads to fewer but strong citations to contradict the claim and a lower threshold leads to more but weak citations to contradict the claim. If unset, the default threshold value is 0.8.

Example request

Using the example facts from the previous section, you can send the following request. Replace the CANDIDATE field with different strings whose grounding and contradictions you want to check.

    curl -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json" \
    "https://discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/global/groundingConfigs/default_grounding_config:check" \
    -d '{
    "answerCandidate": "CANDIDATE",
    "facts": [
     {
      "factText": "Titanic is a 1997 American epic romantic disaster movie. It was directed, written, and co-produced by James Cameron. The movie is about the 1912 sinking of the RMS Titanic. It stars Kate Winslet and Leonardo DiCaprio. The movie was released on December 19, 1997. It received positive critical reviews. The movie won 11 Academy Awards, and was nominated for fourteen total Academy Awards.",
      "attributes": {"author":"Simple Wikipedia"}
     },
     {
      "factText": "James Cameron's \"Titanic\" is an epic, action-packed romance set against the ill-fated maiden voyage of the R.M.S. Titanic; the pride and joy of the White Star Line and, at the time, the largest moving object ever built. She was the most luxurious liner of her era -- the \"ship of dreams\" -- which ultimately carried over 1,500 people to their death in the ice cold waters of the North Atlantic in the early hours of April 15, 1912.",
      "attributes": {"author":"Simple Wikipedia"}
     }
    ],
    "groundingSpec": {
      "citationThreshold": "0.6",
      "enableAntiCitation": true,
      "antiCitationThreshold": "0.8",
    }
    }'

Note: The double quotes are escaped with the backslash character, and the apostrophe is replaced by its HTML character entity.

Example of responses with contradictions

The following table shows an example answer candidate and its response when you send the example request, based on the example facts.

Answer candidate

Check grounding response

Titanic was directed by James Cameron. It starred Brad Pitt and Kate Winslet

Support score: 0.36 Contradiction score: 0.49 Cited chunks: [From FACT 0]....Titanic is a 1997 American epic romantic disaster movie. It was directed, written, and co-produced by James Cameron. It stars Kate Winslet and Leonardo DiCaprio. The movie was released on December 19, 1997. It received positive critical reviews. The movie won 11 Academy Awards, and was nominated for fourteen total Academy Awards..... [From FACT 1]...Titanic is an epic, action-packed romance set against the ill-fated maiden voyage of the R.M.S. Titanic; the pride and joy of the White Star Line and, at the time, the largest moving object ever built. She was the most luxurious liner of her era which ultimately carried over 1,500 people to their death in the ice cold waters of the North Atlantic in the early hours of April 15, 1912.... Claims and citations: 0. Titanic was directed by James Cameron. Citations: [0] Anti-citations: [] 1. It starred Brad Pitt and Kate Winslet Citations: [] Anti-citations: [0]

Note: The raw input and output from the API calls don't look exactly like this. For example, the span is indicated using an integer range, as an index in the answer candidate. See Obtain a support score for an answer candidate.