Retry failed requests

This page describes best practices for retrying failed requests to the Identity and Access Management (IAM) API.

For requests that are safe to retry, we recommend using truncated exponential backoff with introduced jitter.

Overview of truncated exponential backoff

Each request to the IAM API can succeed or fail. If your application retries failed requests without waiting, it might send a large number of retries to IAM in a short period of time. As a result, you might exceed quotas and limits that apply to every IAM resource in your Google Cloud project.

To avoid triggering this issue, we strongly recommend that you use truncated exponential backoff with introduced jitter, which is a standard error-handling strategy for network applications. In this approach, a client periodically retries a failed request with exponentially increasing delays between retries. A small, random delay, known as jitter, is also added between retries. This random delay helps prevent a synchronized wave of retries from multiple clients, also known as the thundering herd problem.

Exponential backoff algorithm

The following algorithm implements truncated exponential backoff with jitter:

  1. Send a request to IAM.
  2. If the request fails, wait 1 + random-fraction seconds, then retry the request.
  3. If the request fails, wait 2 + random-fraction seconds, then retry the request.
  4. If the request fails, wait 4 + random-fraction seconds, then retry the request.
  5. Continue this pattern, waiting 2n + random-fraction seconds after each retry, up to a maximum-backoff time.
  6. After deadline seconds, stop retrying the request.

Use the following values as you implement the algorithm:

  • Before each retry, the wait time is min((2n + random-fraction), maximum-backoff), with n starting at 0 and incremented by 1 for each retry.
  • Replace random-fraction with a random fractional value less than or equal to 1. Use a different value for each retry. Adding this random value prevents clients from becoming synchronized and sending large numbers of retries at the same time.
  • Replace maximum-backoff with the maximum amount of time, in seconds, to wait between retries. Typical values are 32 or 64 (25 or 26) seconds. Choose the value that works best for your use case.
  • Replace deadline with the maximum number of seconds to keep sending retries. Choose a value that reflects your use case. For example, in a continuous integration/continuous deployment (CI/CD) pipeline that is not highly time-sensitive, you might set deadline to 300 seconds (5 minutes).

Types of errors to retry

Use this retry strategy for all requests to the IAM API that return the error codes 500, 502, 503, or 504.

Optionally, you can use this retry strategy for requests to the IAM API that return the error code 404. IAM reads are eventually consistent; as a result, resources might not be visible immediately after you create them, which can lead to 404 errors.

In addition, use a modified version of this retry strategy for all requests to the IAM API that return the error code 409 and the status ABORTED. This type of error indicates a concurrency issue; for example, you might be trying to update an allow policy that another client has already overwritten. For this type of error, you should always retry the entire read-modify-write series of requests, using truncated exponential backoff with introduced jitter. If you retry only the write operation, the request will continue to fail.

What's next