Summarization V1 custom model for chat

Preview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

Agent Assist Summarization custom model for feature lets you provide conversation summaries to your agents after each conversation is completed, leveraging a model trained exclusively on customer-specific data. This differs from the customization through a large language model, as detailed in summarization with custom sections. The summaries help agents create their conversation notes and understand end-user communication history. For example, a summary output about a conversation might look similar to the following:

You are also able to import a custom Agent Assist Summarization model to use with CCAI Insights conversations.

Before you begin

1. Make sure that your data is in the required format. You also have the option of using a sample dataset or testing the feature using the demo Summarization model, no dataset required.

Train and deploy a Summarization model using the API

Create a conversation dataset and import transcripts

Call the create method on the ConversationDataset resource to create a conversation dataset. Include the path to the Cloud Storage bucket containing your transcript data to import it into the new conversation dataset.

Note: Agent Assist provides a set of annotated test data. We recommend that you use the test data if you'd like to test the Summarization feature before you provide your own data. The test data is located in the following Cloud Storage bucket: gs://summarization_integration_test_data/data/*.

Example request:

{
  "displayName": "CONVERSATION_DATASET_NAME",
  "inputConfig": {
    "gcsSource": {
      "uris": ["gs://PATH_NAME/*"]
    }
  }
}

The response contains a conversation dataset ID. Sample response:

{
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.cloud.dialogflow.v2beta1.ConversationDataset",
    "name": "projects/PROJECT_ID/locations/global/conversationDatasets/CONVERSATION_DATASET_ID",
    "displayName": "CONVERSATION_DATASET_NAME",
    "createTime": "2022-06-16T23:13:22.627380457Z"
  }
}

Create a conversation model

Call the create method on the ConversationModel resource to create a Summarization conversation model. Each project will have up to 120 node hours for training every month and can run one training job concurrently.

Required fields:

• datasets: Provide a single dataset that contains the transcript data you used to train the model.
• summarizationModelMetadata: Set to an empty object, or populate the field to override the default.

Note: Unlike other Agent Assist features, Summarization models don't have an allowlist.

Example request:

{
  "displayName": "CONVERSATION_DATASET_NAME",
  "datasets": [{
    "dataset": "projects/PROJECT_ID/locations/global/conversationDatasets/CONVERSATION_DATASET_ID"
  }],
  "summarizationModelMetadata": {
    "trainingModelType": "SUMMARIZATION_MODEL"
  },
  "languageCode": "en-US"
}

The response is a long-running operation, which you can poll using the GetOperation API to check for completion. Training can take several hours. The response returns the status and model ID.

Example response:

{
  "name": "projects/PROJECT_ID/locations/global/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.dialogflow.v2beta1.CreateConversationModelOperationMetadata",
    "conversationModel": "projects/PROJECT_ID/locations/global/conversationModels/MODEL_ID",
    "state": "TRAINING",
    "createTime": "2022-06-16T23:27:50Z"
  }
}

Evaluate a conversation model

After the model training is completed, call the ListConversationModelEvaluations API to check the model quality.

In the ListConversationModelEvaluationsRequest, specify the name of the model to check.

{
  "parent": "projects/cloud-contact-center-quality/locations/global/conversationModels/d3f2ca9f3e2c571a"
}

The ListConversationModelEvaluationsResponse includes a Rouge-L score for evaluating automatic summarization.

{
  "conversationModelEvaluations": [{
    "name": "projects/cloud-contact-center-quality/locations/global/conversationModels/d3f2ca9f3e2c571a/evaluations/c10ac25411a23fe1",
    "displayName": "Training Auto Generated Evaluation",
    "createTime": "2022-06-04T03:38:35.151096Z",
    "evaluationConfig": {
    },
    "summarizationMetrics": {
      "rougel": 0.4474459
    }
  }]
}

Deploy a conversation model

Once the model training is completed, you can deploy the model using the DeployConversationModel API.

In the DeployConversationModelRequest, specify the name of the model to deploy. The response is a long-running operation, which you can poll using the GetOperation API to check for completion.

Configure a conversation profile

Note: If using Agent Assist's generateStatelessSummary API, this step can be skipped to instead provide the fields of the conversation profile within the request.

A conversation profile configures a set of parameters that control the suggestions made to an agent during a conversation. The following steps create a ConversationProfile with a HumanAgentAssistantConfig object.

Create a conversation profile

To create a conversation profile, call the create method on the ConversationProfile resource. Provide a name for the new conversation profile, your Google Cloud project ID, and model ID. In the CreateConversationProfileRequest, specify the conversation model to use and the suggestion feature to be CONVERSATION_SUMMARIZATION.

The following is a JSON example:

{
  "displayName": "CONVERSATION_PROFILE_NAME",
  "humanAgentAssistantConfig": {
    "humanAgentSuggestionConfig": {
      "featureConfigs": [{
        "suggestionFeature": {
          "type": "CONVERSATION_SUMMARIZATION"
        },
        "conversationModelConfig": {
          "model": "projects/PROJECT_ID/locations/global/conversationModels/MODEL_ID",
        }
      }]
    }
  },
  "languageCode": "en-US"
}

The response is a ConversationProfile object containing the conversation profile name:

{
  "name": "projects/PROJECT_ID/locations/global/conversationProfiles/CONVERSATION_PROFILE_ID",
  "displayName": "CONVERSATION_PROFILE_NAME",
  "automatedAgentConfig": {
  },
  "humanAgentAssistantConfig": {
    "notificationConfig": {
    },
    "humanAgentSuggestionConfig": {
      "featureConfigs": [{
        "suggestionFeature": {
          "type": "CONVERSATION_SUMMARIZATION"
        },
        "conversationModelConfig": {
        }
      }]
    },
    "messageAnalysisConfig": {
    }
  },
  "languageCode": "en-US",
  "createTime": "2022-06-06T21:06:46.841816Z",
  "updateTime": "2022-06-06T21:06:46.841816Z",
  "projectNumber": "344549229138"
}

Handle conversations at runtime

Note: If using Agent Assist's generateStatelessSummary API, this step is skipped as the messages of the conversation are provided within the request.

When a dialog begins between an end-user and a human agent, you create a conversation. The following sections walk you through this process.

Create a conversation

To create a conversation, call the create method on the Conversation resource.

The response contains your new conversation ID.

Create an end-user participant

To create an end-user participant, call the create method on the Participant resource. Provide your conversation ID and END_USER for the role field.

The path segment after participants contains your new end user participant ID.

Create a human agent participant

To create a human agent participant, call the create method on the Participant resource. Provide your conversation ID and HUMAN_AGENT for the role field.

The path segment after participants contains your new human agent participant ID.

Analyze message

Option 1: During a conversation

To add and analyze a human agent message for the conversation, call the analyzeContent method on the Participant resource. Provide the conversation ID and human agent participant ID.

To add and analyze an end-user message for the conversation, call the analyzeContent method on the Participant resource. Provide the conversation ID and end-user participant ID.

Don't make duplicate calls to the analyzeContent method if it has been called for other Dialogflow features.

Option 2: After a conversation

You can use this option if you don't use the analyzeContent method during the conversation. Instead, you can use the batchCreate method on the messages resource to ingest historical messages of the conversation.

Get a suggestion

You can get a suggestion at any time for the latest message from either participant.

Option 1. suggestConversationSummary

Call the suggestConversationSummary method on the conversations.suggestions resource. Provide the conversation ID and the latest message ID from either participant.

Option 2. generateStatelessSummary

Call the generateStatelessSummary method. Provide the Messages of the conversation and the latest message ID from either participant.

Here is an example JSON request for generateStatelessSummary:

{
  "statelessConversation": {
    "messages": [{
      "content": "Hello, how can I help you today?",
      "languageCode": "en-US",
      "participantRole": "HUMAN_AGENT"
    }, {
      "content": "I would like to cancel my plan.",
      "languageCode": "en-US",
      "participantRole": "END_USER"
    }, {
      "content": "Okay, I have canceled your plan. Is there anything else that I can do to help you?",
      "languageCode": "en-US",
      "participantRole": "HUMAN_AGENT"
    }, {
      "content": "No, thank you.",
      "languageCode": "en-US",
      "participantRole": "END_USER"
    }, {
      "content": "Okay, have a great day!",
      "languageCode": "en-US",
      "participantRole": "HUMAN_AGENT"
    }],
    "parent": "projects/PROJECT_ID/locations/global"
  },
  "conversationProfile": {
    "humanAgentAssistantConfig": {
      "humanAgentSuggestionConfig": {
        "featureConfigs": [{
          "suggestionFeature": {
            "type": "CONVERSATION_SUMMARIZATION"
          },
          "conversationModelConfig": {
            "model": "projects/PROJECT_ID/locations/global/conversationModels/MODEL_ID"
          }
        }]
      }
    },
    "languageCode": "en-US"
  }
}

The response contains Summarization suggestions.

Complete the conversation

Note: If using Agent Assist's generateStatelessSummary API, this step is skipped as a conversation is not created.

To complete the conversation, call the complete method on the conversations resource. Provide the conversation ID.