Managing Service Rollouts

This page describes how to use Service Infrastructure to perform a gradual service configuration rollouts.

Updating the service configuration for a production service is risky and can potentially cause an outage. The Service Management API allows you to gradually roll out configuration changes, thus mitigating the impact caused by bad service configurations.

You can deploy multiple service configuration versions and define how these versions will be used at runtime, all through calling the services.rollouts.create method to initiate a Rollout. At most 5 service configurations can be rolled out at a time.

Before you begin

To run the examples in this guide, make sure you first follow the instructions to complete the initial setup in Getting Started with the Service Management API.

Performing rollout

Assuming you have a managed service endpointsapis.appspot.com built on the Service Management API, you can perform the following steps to roll out a service configuration change in a staged and controlled manner.

For example, endpointsapis.appspot.com is currently using service configuration old, and you want to change it to use service configuration new. Instead of having the new service configuration instantaneously used by all production traffic, you can create a rollout to test the new service configuration with 10% of the total traffic:

# Create rollout to test the new configuration with 10% traffic.
$ gcurl -d '{
  "rolloutId": "canary-rollout",
  "serviceName": "endpointsapis.appspot.com",
  "trafficPercentStrategy": {
    "percentages": {
      "new": 10,
      "old": 90
    }
  }
}' https://servicemanagement.googleapis.com/v1/services/endpointsapis.appspot.com/rollouts
{
  "name": "operations/rollouts.endpointsapis.appspot.com:canary-rollout"
  "metadata": {
    "@type": "type.googleapis.com/google.api.servicemanagement.v1.OperationMetadata",
    "resourceNames": [
      "services/endpointsapis.appspot.com/rollouts/canary-rollout"
    ],
    "startTime": ...
  },
  "response": {
    "@type": "type.googleapis.com/google.api.servicemanagement.v1.Rollout",
    "rolloutId": "canary-rollout",
    "createTime": ...
    "trafficPercentStrategy": {
      "percentages": {
        "old": 90,
        "new": 10,
      }
    },
    "serviceName": "endpointsapis.appspot.com"
  }
}

After the rollout is created, you can then get the status of the rollout by running the following command, substituting your own rollout ID:

# Get rollout status of `operations/rollouts.endpointsapis.appspot.com:canary-rollout`.
$ gcurl https://servicemanagement.googleapis.com/v1/operations/rollouts.endpointsapis.appspot.com:canary-rollout
{
  "name": "operations/rollouts.endpointsapis.appspot.com:canary-rollout",
  "metadata": {
    "@type": "type.googleapis.com/google.api.servicemanagement.v1.OperationMetadata",
    "resourceNames": [
      "services/endpointsapis.appspot.com/rollouts/canary-rollout"
    ],
    "steps": [
      {
        "description": "update Service Controller",
        "status": "DONE"
      }
    ],
    "progressPercentage": 100,
    "startTime": ...
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.api.servicemanagement.v1.Rollout",
    "rolloutId": "canary-rollout",
    "createTime": ...
    "status": "SUCCESS",
    "trafficPercentStrategy": {
      "percentages": {
        "old": 90,
        "new": 10,
      }
    },
    "serviceName": "endpointsapis.appspot.com"
  }
}

After making sure the canary rollout has completed and the new service configuration is good, you can then create a rollout to let it serve 100% of the traffic:

# Create rollout to let new configuration serve 100% traffic.
$ gcurl -d '{
  "rolloutId": "full-rollout",
  "serviceName": "endpointsapis.appspot.com",
  "trafficPercentStrategy": {
    "percentages": {
      "new": 100,
    }
  }
}' https://servicemanagement.googleapis.com/v1/services/endpointsapis.appspot.com/rollouts
{
  "name": "operations/rollouts.endpointsapis.appspot.com:full-rollout",
  "metadata": {
    "@type": "type.googleapis.com/google.api.servicemanagement.v1.OperationMetadata",
    "resourceNames": [
      "services/endpointsapis.appspot.com/rollouts/full-rollout"
    ],
    "startTime": ...
  },
  "response": {
    "@type": "type.googleapis.com/google.api.servicemanagement.v1.Rollout",
    "rolloutId": "full-rollout",
    "createTime": ...
    "trafficPercentStrategy": {
      "percentages": {
        "new": 100,
      }
    },
    "serviceName": "endpointsapis.appspot.com"
  }
}

In case problems are observed during the test stage, you can roll back to the old configuration by doing:

# Rollback to the old configuration.
$ gcurl -d '{
  "rolloutId": "rollout-to-old",
  "serviceName": "endpointsapis.appspot.com",
  "trafficPercentStrategy": {
    "percentages": {
      "old": 100,
    }
  }
}' https://servicemanagement.googleapis.com/v1/services/endpointsapis.appspot.com/rollouts
{
 "name": "operations/rollouts.endpointsapis.appspot.com:rollout-to-old",
 "metadata": {
   "@type": "type.googleapis.com/google.api.servicemanagement.v1.OperationMetadata",
   "resourceNames": [
     "services/endpointsapis.appspot.com/rollouts/rollout-to-old"
   ],
   "startTime": ...
 },
 "response": {
   "@type": "type.googleapis.com/google.api.servicemanagement.v1.Rollout",
   "rolloutId": "rollout-to-old",
   "createTime": ...
   "trafficPercentStrategy": {
     "percentages": {
       "old": 100,
     }
   },
   "serviceName": "endpointsapis.appspot.com"
 }

}

Viewing rollout history

The Service Management API keeps a history of rollouts. To view the rollout history for endpointsapis.appspot.com, you can do:

# List rollout history for `endpointsapis.appspot.com`.
$ gcurl https://servicemanagement.googleapis.com/v1/services/endpointsapis.appspot.com/rollouts
{
  "rollouts": [
    {
      "rolloutId": "canary-rollout",
      "createTime": ...
      "status": "IN_PROGRESS",
      "trafficPercentStrategy": {
        "percentages": {
          "old": 90,
          "new": 10
         }
      },
      "serviceName": "endpointsapis.appspot.com"
    },
    {
      "rolloutId": "old-rollout",
      "createTime": ...
      "status": "SUCCESS",
      "trafficPercentStrategy": {
        "percentages": {
          "old": 100
         }
      },
      "serviceName": "endpointsapis.appspot.com"
    },
    ...
  ]
}