Retrying Background Functions

This document describes how to enable retrying for background functions.

Semantics of Retry

By default, without retries enabled, the semantics of executing a background function are "best effort." This means that while the goal is to execute the function exactly once, this is not guaranteed.

When you enable retries on a background function, however, the semantics change to "at-least-once" delivery (with the caveat that a function that keeps failing for days may eventually time out).

Why Background Functions Fail to Complete

On rare occasions, a function might exit prematurely due to an internal error, and by default the function might or might not be automatically retried.

More typically, a background function may fail to successfully complete due to errors thrown in the function code itself. Some of the reasons this might happen are as follows:

  • The function contains a bug and the runtime throws an exception.
  • The function cannot reach a service endpoint, or times out while trying to reach the endpoint.
  • The function intentionally throws an exception (for example, when a parameter fails validation).
  • When using promise syntax, the function returns a rejected promise.
  • When using callback syntax, the function detects an error and sends a callback other than null.

In any of the above cases, the function stops executing by default and the event is discarded. If you want to retry the function when an error occurs, you can change the default retry policy by setting the "retry on failure" property. This causes the event to be retried repeatedly for up to multiple days until the function successfully completes.

How to Enable Retries

To enable retries, you can either use the gcloud command-line tool or the Cloud Platform Console.

Using the gcloud Command-Line Tool

To enable retries via the gcloud command-line tool, set the --retry flag when deploying your function:

gcloud beta functions deploy <NAME> --stage-bucket <BUCKET_NAME> <TRIGGER> --retry

Using the Cloud Platform Console

You can enable retries in the Cloud Platform Console as follows:

  1. Go to Cloud Functions Overview page in the Cloud Platform Console.

  2. Click Create function. Alternatively, you can enable retries in an existing function by editing that function.

  3. Name your function.

  4. In the Trigger field, select a background function trigger type, such as Pub/Sub or Cloud Storage.

  5. Fill in the other required fields for your function.

  6. Expand the advanced settings by clicking the "configure region, timeout and retry policy" link.

  7. Check the box labeled retry on failure.

Best Practices

This section describes best practices for using retries.

Use Retry to Handle Transient Errors

Because your function is retried continuously until successful execution, permanent errors like bugs should be eliminated from your code thorough testing before enabling retries. Retries are best used to handle intermittent/transient failures that have a high likelihood of resolution upon retrying, such as a flaky service endpoint or timeout.

Set an End Condition to Avoid Infinite Retry Loops

It is best practice to protect your function against continuous looping when using retries. You can do this by including a well-defined end condition, before the function begins processing. A simple yet effective approach is to discard events with timestamps older than a certain time. This helps to avoid excessive executions when failures are either persistent or longer-lived than expected.

For example, this code snippet discards all events older than 10 seconds:

const eventAgeMs = Date.now() - Date.parse(event.timestamp);
const eventMaxAgeMs = 10000;
if (eventAgeMs > eventMaxAgeMs) {
  console.log("Dropping event ", event, " that with age[ms]: ", eventAgeMs);
  callback();
  return;
} else {
  // Do what the function is supposed to do.
}

Use catch with Promises and Callbacks

If you turn on retries, you should consider adding catch to your promises and callbacks. Cloud Functions does not understand a fatal error that shouldn’t be retried, so you should modify your code accordingly if you support fatal errors.

Here is an example of what you should do if you're using promise syntax:

return doFooAsync().catch(function (err) {
  if (!isFatal(err)) {
    throw err;
  }
  console.error(“Fatal error”, err);
  return null;
});

Here is an example of what you should do if you're using callback syntax:

try {
   doFooAsync(callback);
} catch (err) {
  if (!isFatal(err)) {
    callback(err);
  } else {
    console.error(“Fatal error”, err);
    callback();
  }
}

Make Retryable Background Functions Idempotent

Background functions that can retried must be idempotent, meaning that the function always returns the same result no matter how many times it is executed.

Here are some general guidelines for making a background function idempotent:

  • Many external APIs (such as Stripe) let you supply an idempotency key as a parameter. If you are using such an API, you should use the event ID as the idempotency key.
  • Idempotency works well with at-least-once delivery, because it makes it safe to retry. So a general best practice for writing reliable code is to combine idempotency with retries.

Next steps

Monitor your resources on the go

Get the Google Cloud Console app to help you manage your projects.

Send feedback about...

Cloud Functions Documentation