Experiments

Experiments are used to compare the performance of multiple flow versions (variant versions) to a control version (normally a production version) while handling live traffic. You can allocate a portion of live traffic to each flow version and monitor the following metrics:

  • Contained: Count of sessions that reached END_SESSION without triggering other metrics below. Only available to agents using a telephony integration.
  • Live agent handoff rate: Count of sessions handed off to a live agent.
  • Callback rate: Count of sessions that were restarted by an end-user. Only available to agents using a telephony integration.
  • Abandoned rate: Count of sessions that were abandoned by an end-user. Only available to agents using a telephony integration.
  • Session end rate: Count of sessions that reached END_SESSION.
  • Total no-match count: Total count of occurrences of a no-match event.
  • Total turn count: Total number of conversational turns (one end-user input and one agent response is considered a turn).
  • Average turn count: Average number of turns.

Preparation

To prepare for an experiment:

  1. Decide which flow will be used for the experiment. You cannot run multiple experiments on a single flow, so ensure that you have partitioned your agent into multiple flows.
  2. Create multiple versions for your flow. The differences between each version could be small or large, depending on what you want to compare.
  3. Decide on the amount of traffic that will be allocated to your experiment. If you are testing minor changes, you might start with a higher amount of traffic. For large changes that may be disruptive, consider allocating a small amount of traffic to your experiment.

Create an experiment

To create an experiment:

  1. Open the Dialogflow CX Console.
  2. Select your GCP project to open the agent selector.
  3. Select your agent to open the agent builder.
  4. Select the Manage tab.
  5. Click Experiments to open the Experiments panel.
  6. Click Create.
  7. Enter a description.
  8. Select the environment that you want to run the experiment from.
  9. Select the flow for the experiment.
  10. Optionally enter the number of days in which the experiment will automatically stop.
  11. Enter the control flow version and the percentage of traffic that will go to the control version.
  12. Enter one to four variant flow versions, and the percentage of traffic that will go to the variant version.
  13. Click Save.

Start and stop an experiment

You can start a saved experiment or manually stop a running experiment at any time. Stopping an experiment will cancel the traffic allocation and will revert traffic to its original state.

To start or stop an experiment:

  1. Open the Experiments panel.
  2. Click Start or Stop for an experiment in the list.

Manage experiments

You can edit or delete experiments at any time:

  1. Open the Experiments panel.
  2. Click the option menu for an experiment in the list.
  3. Click Edit or Delete.

Monitor status of experiments

All experiments, regardless of their status, can be found on the experiments panel. Experiments can have four different statuses:

  • Draft: Experiment has been created, but it has never run.
  • Pending: Experiment has started recently, but results are not available yet.
  • Running: Experiment is running and interim results are available.
  • Completed: Experiment has been completed due to automatically or manually being stopped.

Viewing experiment results

To see experiment results:

  1. Open the Experiments panel.
  2. Hover your mouse over the stats icon for an experiment to see a summary of results.
  3. To see the complete results, click the View More button. You are redirected to the analytics panel, where a table with control and variant version metrics are displayed.

Green colored results suggest a favorable outcome, while red suggests a less favorable result. Notice that in some cases, higher/lower numbers are not necessarily better (high abandonment rate / low abandonment rate).

You can also view the results directly from the Analytics panel:

  1. Open the Dialogflow CX Console.
  2. Select your GCP project to open the agent selector.
  3. Select your agent to open the agent builder.
  4. Select the Manage tab.
  5. Click Analytics to open the Analytics panel.
  6. Click the Experiments tab.
  7. Select an environment.
  8. Select an experiment name.