- Resource: Experiment
- State
- Definition
- VersionVariants
- Variant
- RolloutConfig
- RolloutStep
- RolloutState
- Result
- VersionMetrics
- Metric
- MetricType
- CountType
- ConfidenceInterval
- VariantsHistory
- Methods
Resource: Experiment
Represents an experiment in an environment.
JSON representation |
---|
{ "name": string, "displayName": string, "description": string, "state": enum ( |
Fields | |
---|---|
name |
The name of the experiment. Format: projects/ |
display |
Required. The human-readable name of the experiment (unique in an environment). Limit of 64 characters. |
description |
The human-readable description of the experiment. |
state |
The current state of the experiment. Transition triggered by Experiments.StartExperiment: DRAFT->RUNNING. Transition triggered by Experiments.CancelExperiment: DRAFT->DONE or RUNNING->DONE. |
definition |
The definition of the experiment. |
rollout |
The configuration for auto rollout. If set, there should be exactly two variants in the experiment (control variant being the default version of the flow), the traffic allocation for the non-control variant will gradually increase to 100% when conditions are met, and eventually replace the control variant to become the default version of the flow. |
rollout |
State of the auto rollout process. |
rollout |
The reason why rollout has failed. Should only be set when state is ROLLOUT_FAILED. |
result |
Inference result of the experiment. |
create |
Creation time of this experiment. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
start |
Start time of this experiment. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
end |
End time of this experiment. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
last |
Last update time of this experiment. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
experiment |
Maximum number of days to run the experiment/rollout. If auto-rollout is not enabled, default value and maximum will be 30 days. If auto-rollout is enabled, default value and maximum will be 6 days. A duration in seconds with up to nine fractional digits, ending with ' |
variants |
The history of updates to the experiment variants. |
State
The state of the experiment.
Enums | |
---|---|
STATE_UNSPECIFIED |
State unspecified. |
DRAFT |
The experiment is created but not started yet. |
RUNNING |
The experiment is running. |
DONE |
The experiment is done. |
ROLLOUT_FAILED |
The experiment with auto-rollout enabled has failed. |
Definition
Definition of the experiment.
JSON representation |
---|
{ "condition": string, // Union field |
Fields | |
---|---|
condition |
The condition defines which subset of sessions are selected for this experiment. If not specified, all sessions are eligible. E.g. "queryInput.language_code=en" See the conditions reference. |
Union field variants . The variants of the experiment. We currently only support single variant experiment. variants can be only one of the following: |
|
version |
The flow versions as the variants of this experiment. |
VersionVariants
A list of flow version variants.
JSON representation |
---|
{
"variants": [
{
object ( |
Fields | |
---|---|
variants[] |
A list of flow version variants. |
Variant
A single flow version with specified traffic allocation.
JSON representation |
---|
{ "version": string, "trafficAllocation": number, "isControlGroup": boolean } |
Fields | |
---|---|
version |
The name of the flow version. Format: |
traffic |
Percentage of the traffic which should be routed to this version of flow. Traffic allocation for a single flow must sum up to 1.0. |
is |
Whether the variant is for the control group. |
RolloutConfig
The configuration for auto rollout.
JSON representation |
---|
{
"rolloutSteps": [
{
object ( |
Fields | |
---|---|
rollout |
Steps to roll out a flow version. Steps should be sorted by percentage in ascending order. |
rollout |
The conditions that are used to evaluate the success of a rollout step. If not specified, all rollout steps will proceed to the next one unless failure conditions are met. E.g. "containment_rate > 60% AND callback_rate < 20%". See the conditions reference. |
failure |
The conditions that are used to evaluate the failure of a rollout step. If not specified, no rollout steps will fail. E.g. "containment_rate < 10% OR average_turn_count < 3". See the conditions reference. |
RolloutStep
A single rollout step with specified traffic allocation.
JSON representation |
---|
{ "displayName": string, "trafficPercent": integer, "minDuration": string } |
Fields | |
---|---|
display |
The name of the rollout step; |
traffic |
The percentage of traffic allocated to the flow version of this rollout step. (0%, 100%]. |
min |
The minimum time that this step should last. Should be longer than 1 hour. If not set, the default minimum duration for each step will be 1 hour. A duration in seconds with up to nine fractional digits, ending with ' |
RolloutState
State of the auto-rollout process.
JSON representation |
---|
{ "step": string, "stepIndex": integer, "startTime": string } |
Fields | |
---|---|
step |
Display name of the current auto rollout step. |
step |
Index of the current step in the auto rollout steps list. |
start |
Start time of the current step. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
Result
The inference result which includes an objective metric to optimize and the confidence interval.
JSON representation |
---|
{
"versionMetrics": [
{
object ( |
Fields | |
---|---|
version |
Version variants and metrics. |
last |
The last time the experiment's stats data was updated. Will have default value if stats have never been computed for this experiment. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
VersionMetrics
Version variant and associated metrics.
JSON representation |
---|
{
"version": string,
"metrics": [
{
object ( |
Fields | |
---|---|
version |
The name of the flow |
metrics[] |
The metrics and corresponding confidence intervals in the inference result. |
session |
Number of sessions that were allocated to this version. |
Metric
Metric and corresponding confidence intervals.
JSON representation |
---|
{ "type": enum ( |
Fields | |
---|---|
type |
Ratio-based metric type. Only one of type or countType is specified in each Metric. |
count |
Count-based metric type. Only one of type or countType is specified in each Metric. |
confidence |
The probability that the treatment is better than all other treatments in the experiment |
Union field value . The actual value of the metric. value can be only one of the following: |
|
ratio |
Ratio value of a metric. |
count |
Count value of a metric. |
MetricType
Types of ratio-based metric for Dialogflow experiment.
Enums | |
---|---|
METRIC_UNSPECIFIED |
Metric unspecified. |
CONTAINED_SESSION_NO_CALLBACK_RATE |
Percentage of contained sessions without user calling back in 24 hours. |
LIVE_AGENT_HANDOFF_RATE |
Percentage of sessions that were handed to a human agent. |
CALLBACK_SESSION_RATE |
Percentage of sessions with the same user calling back. |
ABANDONED_SESSION_RATE |
Percentage of sessions where user hung up. |
SESSION_END_RATE |
Percentage of sessions reached Dialogflow 'END_PAGE' or 'END_SESSION'. |
CountType
Types of count-based metric for Dialogflow experiment.
Enums | |
---|---|
COUNT_TYPE_UNSPECIFIED |
Count type unspecified. |
TOTAL_NO_MATCH_COUNT |
Total number of occurrences of a 'NO_MATCH'. |
TOTAL_TURN_COUNT |
Total number of turn counts. |
AVERAGE_TURN_COUNT |
Average turn count in a session. |
ConfidenceInterval
A confidence interval is a range of possible values for the experiment objective you are trying to measure.
JSON representation |
---|
{ "confidenceLevel": number, "ratio": number, "lowerBound": number, "upperBound": number } |
Fields | |
---|---|
confidence |
The confidence level used to construct the interval, i.e. there is X% chance that the true value is within this interval. |
ratio |
The percent change between an experiment metric's value and the value for its control. |
lower |
Lower bound of the interval. |
upper |
Upper bound of the interval. |
VariantsHistory
The history of variants update.
JSON representation |
---|
{ "updateTime": string, // Union field |
Fields | |
---|---|
update |
Update time of the variants. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
Union field variants . The variants updated. We currently only support single variant experiment. variants can be only one of the following: |
|
version |
The flow versions as the variants. |
Methods |
|
---|---|
|
Creates an Experiment in the specified Environment . |
|
Deletes the specified Experiment . |
|
Retrieves the specified Experiment . |
|
Returns the list of all experiments in the specified Environment . |
|
Updates the specified Experiment . |
|
Starts the specified Experiment . |
|
Stops the specified Experiment . |