Experiments

Experiments are used to compare the performance of multiple flow versions (variant versions) to a control version (normally a production version) while handling live traffic. You can allocate a portion of live traffic to each flow version and monitor the following metrics:

• Contained: Count of sessions that reached END_SESSION without triggering other metrics below. Only available to agents using a telephony integration.
• Live agent handoff rate: Count of sessions handed off to a live agent.
• Callback rate: Count of sessions that were restarted by an end-user. Only available to agents using a telephony integration.
• Abandoned rate: Count of sessions that were abandoned by an end-user. Only available to agents using a telephony integration.
• Session end rate: Count of sessions that reached END_SESSION.
• Total no-match count: Total count of occurrences of a no-match event.
• Total turn count: Total number of conversational turns (one end-user input and one agent response is considered a turn).
• Average turn count: Average number of turns.

Preparation

To prepare for an experiment:

1. Decide which flow will be used for the experiment. You cannot run multiple experiments on a single flow, so ensure that you have partitioned your agent into multiple flows.
2. Create multiple versions for your flow. The differences between each version could be small or large, depending on what you want to compare.
3. Decide on the amount of traffic that will be allocated to your experiment. If you are testing minor changes, you might start with a higher amount of traffic. For large changes that may be disruptive, consider allocating a small amount of traffic to your experiment.

Create an experiment

To create an experiment:

1. Open the Dialogflow CX console.
2. Select your project to open the agent selector.
3. Select your agent to open the agent builder.
4. Select the Manage tab.
5. Click Experiments to open the Experiments panel.
6. Select the Status tab.
7. Click Create.
8. Enter a description.
9. Select the environment that you want to run the experiment from.
10. Select the flow for the experiment.
11. Optionally enter the number of days in which the experiment will automatically stop.
12. Enter the control flow version and the percentage of traffic that will go to the control version.
13. Enter one to four variant flow versions, and the percentage of traffic that will go to the variant version.
14. Optionally, click Enable auto rollout and steps for a gradual rollout of traffic to the variant flow. An automated experiment is based on steps, which are time durations in which a percentage of traffic is increased to the variant flow. Auto rollout only supports one variant flow.
    1. Under Rollout rules, you can set one or more conditional rules to determine how the experiment should proceed through the steps.
       1. If you select Match at least one rule, the experiment proceeds to the next step if at least one rule and the time duration for the current step are met.
       2. If you select Match all rules, the experiment proceeds to the next step if all rules and the time duration for the current step are met.
       3. If you select Steps only, the experiment proceeds according to the time durations for each step.
    2. Under Increase steps, define a percentage of traffic to allocate to the variant flow and a time duration for each step. The default time duration for each step is 6 hours.
    3. Select Stop conditions to set one or more conditions under which to stop sending traffic to the variant flow. Note that you cannot restart a stopped experiment.
15. Click Save.

Start and stop an experiment

You can start a saved experiment or manually stop a running experiment at any time. Stopping an experiment will cancel the traffic allocation and will revert traffic to its original state.

Note: If you stop an experiment while it is pending, results will not be available. If you stop an experiment while it is running, results might be inconclusive or missing.

To start or stop an experiment:

1. Open the Experiments panel.
2. Select the Status tab.
3. Click Start or Stop for an experiment in the list.

Manage experiments

Note: You can change variant traffic allocation while an experiment is running.

You can edit or delete experiments at any time:

1. Open the Experiments panel.
2. Select the Status tab.
3. Click the option menu for an experiment in the list.
4. Click Edit or Delete.

Monitor status of experiments

All experiments, regardless of their status, can be found on the experiments panel. Experiments can have four different statuses:

• Draft: Experiment has been created, but it has never run.
• Pending: Experiment has started recently, but results are not available yet.
• Running: Experiment is running and interim results are available.
• Completed: Experiment has been completed due to automatically or manually being stopped.

Viewing experiment results

To see experiment results:

1. Open the Dialogflow CX console.
2. Select your project to open the agent selector.
3. Select your agent to open the agent builder.
4. Select the Manage tab.
5. Click Experiments to open the Experiments panel.
6. Select the Results tab.
7. Select an environment and experiment to see the results.

Green colored results suggest a favorable outcome, while red suggests a less favorable result. Notice that in some cases, higher/lower numbers are not necessarily better (high abandonment rate / low abandonment rate).

Note: You will see "no experiment result" if not enough conversations have been through each variant of the experiment.

Limitations

The following limitations apply:

• The Enable interaction logging agent setting must be enabled.