Smart Reply follows a conversation between a human agent and an end user and surfaces suggested responses to the human agent. Suggested responses are calculated by a custom model that has been trained on your own conversation data.
This document walks you through the process of using the API to implement Smart Reply and get suggestions from this feature. If preferred, you have the option of using the Agent Assist Console to upload your data, train a model, and test your Smart Reply results during design-time. To see Smart Reply suggestions during runtime, you must call the API directly. See the Smart Reply tutorial for details about training a model and testing its performance using the Agent Assist Console.
Agent Assist also provides publicly-available conversation data as well as a pre-trained model and allowlist. You can use these resources to see how Smart Reply works or test your integration before uploading your own data. See the conversation data format documentation for more information.
Before you begin
Complete the following before starting this guide:
- Create a conversation dataset using your own transcript data.
- Train a Smart Reply model using your conversation dataset(s).
- Create a conversation profile.
- Test your Smart Reply results using the Agent Assist simulator.
Personal Identifying Information and children's data
When you send data to this API, the API attempts to redact all personally-identifiable information (PII). If you need to ensure that the model doesn't include PII, you should sanitize your data before sending it to the API. Replace the redacted words with placeholders such as `REDACTED_NUMBER` or `REDACTED_NAME` rather than simply removing them.
Also, if your data contains information collected from children, you should remove the children's data before sending it to the API.
Train and deploy a model
Agent Assist Smart Reply models are trained using conversation datasets. A conversation dataset contains your own uploaded transcript data. This section walks you through the process of creating a conversation dataset, uploading your conversation data to it, and training and deploying a model. You can also perform these actions using the Agent Assist Console if you would prefer not to call the API directly.
Create a conversation dataset
Before you can begin uploading conversation transcripts you must first create a
conversation dataset to put them in. Call the create
method on the
ConversationDataset
resource to create a conversation dataset.
The response contains a conversation dataset ID.
Import conversation transcripts to your conversation dataset
Upload your chat conversation data to your conversation dataset so that it can
be processed by Agent Assist.
Make sure that a transcript of each conversation is in
JSON
format and stored in a
Google Cloud Storage bucket.
A conversation dataset must contain at least 30,000 conversations, otherwise
model training will fail. As a general rule, the more conversations you have the
better your model quality will be. We recommend uploading at least 3 months
of conversations to sufficiently cover as many use cases as possible. The
maximum number of messages in a conversation dataset is 1,000,000.
Call the importConversationData
method on the
ConversationDataset
resource to import your conversations.
Required fields:
- The conversation dataset ID that you created above.
inputConfig
: The path to your conversation transcript data in a GCS bucket.
The response is a long-running operation, which you can poll to check for completion.
Create a conversation model
Call the create
method on the
ConversationModel
resource to create a conversation model. This action also creates the model's
allowlist.
Required fields:
datasets
: Provide a single dataset using the conversation dataset ID you created above.smartReplyModelMetadata
: Set to an empty object, or populate the field to override the default.
The response is a long-running operation, which you can poll to check for completion. Once completed, the model ID and allowlist ID will be included in the metadata for the operation.
- Model ID:
name
- Allowlist ID:
smart_reply_model_metadata.associated_allowlist_info.document
Deploy the conversation model
Call the deploy
method on the
ConversationModel
resource to deploy the conversation model.
Required field:
conversationModels
: The conversation model ID you created above.
Manage an allowlist
Each model has an allowlist associated with it, which is automatically created when you create a conversation model. The allowlist contains all responses, generated from your conversation dataset(s), that can be surfaced to a human agent at runtime. This section describes allowlist creation and management. You can also perform these actions using the Agent Assist Console if you would prefer not to call the API directly.
Export the allowlist contents to a .csv
file
Model creation automatically creates an allowlist that's associated with the new
model. The allowlist is a Document resource with a unique ID. The ID is returned
in smart_reply_model_metadata.associated_allowlist_info.document
when a model
is created. In order to review and make changes to messages on the allowlist,
you must export it to a Cloud Storage Bucket.
Call the export
method on the
Document
resource to export the document to a .csv
file in a Cloud Storage bucket.
The smart_messaging_partial_update
field is optional but affects how you will
be able to update this allowlist
in the future. If set to true
, the exported .csv
file will include a column
that contains a unique ID for each message. You can use the message ID to only
update specified messages instead of the entire document. If
smart_messaging_partial_update
is set to false
or not set, the extra column
will not appear in the file and any updates to the allowlist will require an
update to the entire document.
Required field:
gcsDestination
: Your the path to your desired Cloud Storage bucket.
The response is a
long-running operation,
which you can poll to check for completion. Once completed, the .csv
file you
provided in the request is populated with response candidates.
Review the allowlist
The generated allowlist contains responses that were automatically generated by
Smart Reply based on your conversation data. You can now review and update those
responses as needed. Download the .csv
file from your GCS bucket, edit it to
suit your needs, and upload the file back to the Cloud Storage bucket. Only
responses on the allowlist can be surfaced to human agents.
If you edit any responses we recommend that you edit only for spelling and grammar, and do not change the meaning of the message. The more the edited text deviates from the meaning in the model, the less likely that message is to be surfaced.
You also can create new messages if needed. Similarly to edited messages, created messages are less likely to be surfaced during runtime.
Update the allowlist
After you have finished updating the .csv
file, you can use it to update the
Document
resource. You can choose to update the entire allowlist or only
specified messages. To update only specified messages, you must
have set smart_messaging_partial_update
to true
when you
exported the allowlist.
If you have already done this, use the automatically-generated column in the
exported .csv
file to indicate the messages to be updated.
Call the reload
method on the
Document
resource to update the allowlist. To update only specified messages, set
smart_messaging_partial_update
to true
in the ReloadDocumentRequest
. To
update the entire allowlist, leave smart_messaging_partial_update
unset or set
to false
.
Required fields:
gcsSource
: The Cloud Storage path to the.csv
file.name
: The allowlist resource name generated when you created a conversation model.
Example request:
{ "name":"projects/project-id/knowledgeBases/knowledge-base-id/documents/allowlist-id", "gcsSource" { "uri": "gs://revised_smart_reply_allowlist_path" } }
Evaluate a trained model's performance
You can test a model's performance after you have deployed the model and created an allowlist for it. You must also provide a test dataset. The responses generated by the trained Smart Reply model and its associated allowlist will be compared to actual agent messages in the test dataset. The test dataset should be made up of real-world conversation data but must not contain any of the data in the conversation dataset that you used to train the model. For example, given 1 month's worth of conversation traffic, you could use 3 weeks of conversation data to create a conversation dataset and the remaining 1 week's data to create the test dataset. A test dataset should contain a minimum of 1000 conversations, but as a general rule the evaluation metrics will be more reliable with more conversations in the test dataset. Test dataset format is the same as conversation dataset format.
To create a new model evaluation, call the CreateConversationModelEvaluation
method on a ConversationModel
resource. This method returns a longrunning
operation. You can poll the operation to check its state, which will return
one of INITIALIZING
, RUNNING
, SUCCEEDED
, CANCELLED
, FAILED
.
Required fields:
InputDataset
: The test dataset that will be used to test the model's performance.allowlist_document
: The allowlist associated with the Smart Reply model to be tested.
A ConversationModelEvaluation
resource is returned when the longrunning
operation has completed. Two metrics are included:
allowlist_coverage
: The percentage of agent messages in the test dataset that are covered by the allowlist.recall
: The percentage of agent messages in the test dataset that are contained in the allowlist and appear in the top 3 suggestions surfaced by the Smart Reply model.
Configure a conversation profile
A conversation profile configures a set of parameters that control the
suggestions made to an agent during a conversation. The following steps create a
ConversationProfile
with a
HumanAgentAssistantConfig
object. You can also perform these actions using the
Agent Assist Console if you would
prefer not to call the API directly.
Create a conversation profile
To create a conversation profile,
call the create
method on the
ConversationProfile
resource.
Provide your knowledge base ID, document ID, project ID, and model ID.
{ "displayName":"smart_reply_assist", "humanAgentAssistantConfig":{ "humanAgentSuggestionConfig":{ "featureConfigs":[ { "suggestionFeature": { "type":"SMART_REPLY" }, "queryConfig": { "documentQuerySource":{ "documents": ["projects/PROJECT_ID/knowledgeBases/KNOWLEDGE_BASE_ID/documents/DOCUMENT_ID"] }, "maxResults": 3 }, } }, "conversationModelConfig":{ "model": "projects/PROJECT_ID/conversationModels/MODEL_ID" } } ] } } }
We recommend that you set the
SuggestionFeatureConfig.enable_inline_suggestion
value.
If this value is true, later calls to AnalyzeContent
will result in responses with a list of suggestions.
The response contains your new conversation profile ID.
Handle conversations at runtime
When a dialog begins between an end-user and a human agent, you create a conversation. In order to see suggestions, you must also create both an end-user participant and a human agent participant and add them to the conversation. The following sections walk you through this process.
Create a conversation
To create a conversation,
call the create
method on the
Conversation
resource.
REST
Before using any of the request data, make the following replacements:
- PROJECT_ID: your Cloud project ID
- LOCATION_ID: your location ID
- CONVERSATION_PROFILE_ID: the ID you received when creating the conversation profile
HTTP method and URL:
POST https://dialogflow.googleapis.com/v2/projects/PROJECT_ID/locations/LOCATION_ID/conversations
Request JSON body:
{ "conversationProfile": "projects/PROJECT_ID/locations/LOCATION_ID/conversationProfiles/CONVERSATION_PROFILE_ID", }
To send your request, expand one of these options:
You should receive a JSON response similar to the following:
{ "name": "projects/PROJECT_ID/locations/LOCATION_ID/conversations/CONVERSATION_ID", "lifecycleState": "IN_PROGRESS", "conversationProfile": "projects/PROJECT_ID/locations/LOCATION_ID/conversationProfiles/CONVERSATION_PROFILE_ID", "startTime": "2018-11-05T21:05:45.622Z" }
The path segment after conversations
contains your new conversation ID.
Python
To authenticate to Agent Assist, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Create an end-user participant
You must add both end-user and human agent participants to the conversation
in order to see suggestions. To create an end-user participant, call the
create
method on the
Participant
resource. Provide your conversation ID and END_USER
for the role
field.
REST
Before using any of the request data, make the following replacements:
- PROJECT_ID: your Cloud project ID
- LOCATION_ID: your location ID
- CONVERSATION_ID: your conversation ID
HTTP method and URL:
POST https://dialogflow.googleapis.com/v2/projects/PROJECT_ID/locations/LOCATION_ID/conversations/CONVERSATION_ID/participants
Request JSON body:
{ "role": "END_USER", }
To send your request, expand one of these options:
You should receive a JSON response similar to the following:
{ "name": "projects/PROJECT_ID/locations/LOCATION_ID/conversations/CONVERSATION_ID/participants/PARTICIPANT_ID", "role": "END_USER" }
The path segment after participants
contains your new end user participant ID.
Python
To authenticate to Agent Assist, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Create a human agent participant
To create a human agent participant, call the create
method on the
Participant
resource. Provide your conversation ID and HUMAN_AGENT
for the role
field.
REST
Before using any of the request data, make the following replacements:
- PROJECT_ID: your Cloud project ID
- LOCATION_ID: your location ID
- CONVERSATION_ID: your conversation ID
HTTP method and URL:
POST https://dialogflow.googleapis.com/v2/projects/PROJECT_ID/locations/LOCATION_ID/conversations/CONVERSATION_ID/participants
Request JSON body:
{ "role": "HUMAN_AGENT", }
To send your request, expand one of these options:
You should receive a JSON response similar to the following:
{ "name": "projects/PROJECT_ID/locations/LOCATION_ID/conversations/CONVERSATION_ID/participants/PARTICIPANT_ID", "role": "HUMAN_AGENT" }
The path segment after participants
contains your new human agent participant ID.
Python
To authenticate to Agent Assist, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Add and analyze a message from the human agent
Each time either participant types a message in the conversation, you need to
send that message to the API for processing. Agent Assist bases its
suggestions on analysis of human agent and end-user messages. To add and analyze
a human agent message for the conversation, call the analyzeContent
method on
the
Participant
resource.
Provide the conversation ID and human agent participant ID.
REST
Before using any of the request data, make the following replacements:
- PROJECT_ID: your GCP project ID
- CONVERSATION_ID: your conversation ID
- PARTICIPANT_ID: your human agent participant ID
HTTP method and URL:
POST https://dialogflow.googleapis.com/v2/projects/PROJECT_ID/conversations/CONVERSATION_ID/participants/PARTICIPANT_ID:analyzeContent
Request JSON body:
{ "textInput": { "text": "How may I help you?", "languageCode": "en-US" } }
To send your request, expand one of these options:
You should receive a JSON response similar to the following:
{ "message": { "name": "projects/PROJECT_ID/conversations/CONVERSATION_ID/messages/MESSAGE_ID", "content": "How may I help you?", "languageCode": "en-US", "participant": "PARTICIPANT_ID", "participantRole": "HUMAN_AGENT", "createTime": "2020-02-13T00:01:30.683Z" }, "humanAgentSuggestionResults": [ { "suggestSmartRepliesResponse": { "smartReplyAnswers": [ { "reply": "I am here to help you.", "confidence": 0.5, "answerRecord": "projects/PROJECT_ID/answerRecords/ANSWER_RECORD_ID_1" }, { "reply": "Sorry for the wait, we have a high volume of chats right now.", "confidence": 0.3, "answerRecord": "projects/PROJECT_ID/answerRecords/ANSWER_RECORD_ID_2" }, { "reply": "Thank you for contacting us!", "confidence": 0.1, "answerRecord": "projects/PROJECT_ID/answerRecords/ANSWER_RECORD_ID_3" } ] } } ] }
Python
To authenticate to Agent Assist, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Add and analyze a message from the end-user
To add and analyze an end-user message for the conversation, call the
analyzeContent
method on the
Participant
resource. Provide the conversation ID and end-user participant ID.
SuggestionFeatureConfig.enable_inline_suggestion
to true in your conversation
profile, the response contains a list of smart reply suggestions.
REST
Before using any of the request data, make the following replacements:
- PROJECT_ID: your GCP project ID
- CONVERSATION_ID: your conversation ID
- PARTICIPANT_ID: your end-user participant ID
HTTP method and URL:
POST https://dialogflow.googleapis.com/v2/projects/PROJECT_ID/conversations/CONVERSATION_ID/participants/PARTICIPANT_ID:analyzeContent
Request JSON body:
{ "textInput": { "text": "I want to reserve a room.", "languageCode": "en-US" } }
To send your request, expand one of these options:
You should receive a JSON response similar to the following:
{ "message": { "name": "projects/PROJECT_ID/conversations/CONVERSATION_ID/messages/MESSAGE_ID", "content": "I want to reserve a room.", "languageCode": "en-US", "participant": "PARTICIPANT_ID", "participantRole": "END_USER", "createTime": "2020-02-13T00:07:35.925Z" }, "humanAgentSuggestionResults": [ { "suggestSmartRepliesResponse": { "smartReplyAnswers": [ { "reply": "Where would you like to reserve a room?", "confidence": 0.5, "answerRecord": "projects/PROJECT_ID/answerRecords/ANSWER_RECORD_ID_1" }, { "reply": "What type of rooms would you like to reserve?", "confidence": 0.3, "answerRecord": "projects/PROJECT_ID/answerRecords/ANSWER_RECORD_ID_2" }, { "reply": "How long do you want to stay?", "confidence": 0.1, "answerRecord": "projects/PROJECT_ID/answerRecords/ANSWER_RECORD_ID_3" } ] } } ] }
Python
To authenticate to Agent Assist, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Get a suggestion (Optional)
You can get suggestions at any time. You can optionally specify the message ID for any message to receive suggestions based on that message. If this field is not set, suggestions are based on the latest message from either participant by default.
To get suggestions, call the suggestSmartReplies
method on the
Suggestion
resource. Provide the conversation ID, human agent participant ID, and the a
message ID from either participant (optional).
The response contains Smart Reply suggestions.
REST
Before using any of the request data, make the following replacements:
- PROJECT_ID: your GCP project ID
- CONVERSATION_ID: your conversation ID
- PARTICIPANT_ID: your end-user participant ID
HTTP method and URL:
POST https://dialogflow.googleapis.com/v2/projects/PROJECT_ID/conversations/CONVERSATION_ID/participants/PARTICIPANT_ID:analyzeContent
Request JSON body:
{ "textInput": { "text": "I want to reserve a room.", "languageCode": "en-US" } }
To send your request, expand one of these options:
You should receive a JSON response similar to the following:
{ "message": { "name": "projects/PROJECT_ID/conversations/CONVERSATION_ID/messages/MESSAGE_ID", "content": "I want to reserve a room.", "languageCode": "en-US", "participant": "PARTICIPANT_ID", "participantRole": "END_USER", "createTime": "2020-02-13T00:07:35.925Z" }, "humanAgentSuggestionResults": [ { "suggestSmartRepliesResponse": { "smartReplyAnswers": [ { "reply": "Where would you like to reserve a room?", "confidence": 0.5, "answerRecord": "projects/PROJECT_ID/answerRecords/ANSWER_RECORD_ID_1" }, { "reply": "What type of rooms would you like to reserve?", "confidence": 0.3, "answerRecord": "projects/PROJECT_ID/answerRecords/ANSWER_RECORD_ID_2" }, { "reply": "How long do you want to stay?", "confidence": 0.1, "answerRecord": "projects/PROJECT_ID/answerRecords/ANSWER_RECORD_ID_3" } ] } } ] }
Python
To authenticate to Agent Assist, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Complete the conversation
To complete the conversation, call the complete
method on the
conversations
resource. Provide the conversation ID.
REST
Before using any of the request data, make the following replacements:
- PROJECT_ID: your GCP project ID
- CONVERSATION_ID: the ID you received when creating the conversation
HTTP method and URL:
POST https://dialogflow.googleapis.com/v2/projects/PROJECT_ID/conversations/CONVERSATION_ID:complete
To send your request, expand one of these options:
You should receive a JSON response similar to the following:
{ "name": "projects/PROJECT_ID/conversations/CONVERSATION_ID", "lifecycleState": "COMPLETED", "conversationProfile": "projects/PROJECT_ID/conversationProfiles/CONVERSATION_PROFILE_ID", "startTime": "2018-11-05T21:05:45.622Z", "endTime": "2018-11-06T03:50:26.930Z" }
Python
To authenticate to Agent Assist, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.