Configure scheduled snapshots

Stay organized with collections Save and categorize content based on your preferences.

Cloud Composer 1 | Cloud Composer 2

This page explains how to set up a Cloud Scheduler job that saves snapshots of your environment on a schedule.

For more information about how environment snapshots work and how to save snapshots manually, see Save and load snapshots.

How scheduled snapshots work

Scheduled snapshots use Cloud Scheduler and Cloud Composer API. A Cloud Scheduler Cron Job periodically saves snapshots of your environment to a Cloud Storage bucket.

Cloud Composer saves snapshots in the environment's bucket by default. This guide uses a different approach. For scheduled snapshots, we recommend to save snapshots in a separate bucket. In this way, you can set up permissions and a lifecycle configuration for this bucket. One bucket can hold snapshots from several environments, and you can also create buckets that are geo-redundant.

Before you begin

  • Cloud Composer supports environment snapshots in 2.0.9 and later versions.

  • Cloud Scheduler is not supported in the VPC Service Controls configuration.

  • Creating and storing snapshots produces additional costs, which are related to Cloud Storage. For more information, see Pricing.

  • Enable the Cloud Scheduler API:

    Console

    Enable the Cloud Scheduler API.

    Enable the API

    gcloud

    Enable the Cloud Scheduler API:

    gcloud services enable cloudscheduler.googleapis.com

Step 1. Create a bucket for scheduled snapshots

Create a bucket in the same region where your environment is located. By doing so you reduce the costs of transferring data between regions.

For example, if your environment is located in us-central1, select us-central1 as the region for your bucket.

Step 2. Configure permissions for the bucket

Configure the following permissions:

  • The service account of your environment must have read and write permissions for this bucket. For example, the Storage Object Admin role has such permissions.

  • User accounts do not need any additional permissions for the bucket to load snapshots from it, unless you want to view the contents of the bucket from Google Cloud console. In this case a user account must have read permission on the bucket.

Step 3. Set a lifecycle configuration for the bucket

To save storage costs, you can configure a rule that deletes environment snapshots after a certain period of time.

For example, to automatically delete snapshots older than 30 days:

  1. Set a lifecycle configuration for the bucket.
  2. Specify the Delete object action.
  3. Select Age and 30 days as the condition.

Step 4. Choose a service account for the Cloud Scheduler job

A Cloud Scheduler job, which you create in the next step, requires a service account to make calls to Cloud Composer API.

You can use one of the options:

Step 5. Create the Cloud Scheduler job

Console

Create a new job with the following parameters:

  1. In Google Cloud console, go to the Cloud Scheduler page.

    Go to Cloud Scheduler

  2. (Recommended) In the Region drop-down list, select the same region where your environment and the bucket for snapshots are located.

  3. In the Frequency field, in the unix-cron format, specify how often environment snapshots must be taken. For example, to run this job every two hours, specify 0 */2 * * *.

  4. Expand the Configure the execution section.

  5. In the Target type drop-down list, select HTTP.

  6. In the URL field, enter the URL for the saveSnapshot REST API method:

    https://composer.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME:saveSnapshot
    

    Replace:

    • ENVIRONMENT_NAME with the name of the environment.
    • LOCATION with the region where the environment is located.
    • PROJECT_ID with the Project ID.

    Example:

    https://composer.googleapis.com/v1beta1/projects/example-project/locations/us-central1/environments/example-environment:saveSnapshot
    
  7. In the HTTP method drop-down list, select POST.

  8. In the Body field, in the snapshotLocation field, specify the URI of the bucket folder where you want to save snapshots.

    {
      "snapshotLocation": "SNAPSHOTS_FOLDER"
    }
    

    Replace:

    • SNAPSHOTS_FOLDER with the URI of a bucket folder where to save the snapshot.

    Example:

      {
        "snapshotLocation": "gs://example-snapshots-bucket/snapshots"
      }
    
  9. In the Auth header drop-down list, select Add OAuth token.

  10. In the Service account drop-down list, select a service account ( see the previous step).

  11. In the Scope field, specify https://www.googleapis.com/auth/cloud-platform.

gcloud

Create a new job with the following parameters:

  • (Recommended) In the --location argument, specify the region where your environment and the bucket for snapshots are located. In this way, the job is created in the same region.

  • In the --schedule argument, in the unix-cron format, specify how often environment snapshots must be taken. For example, to run this job every two hours, specify 0 */2 * * *.

  • In the --uri argument, specify the URL for the saveSnapshot REST API method:

    https://composer.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME:saveSnapshot
    

    Replace:

    • ENVIRONMENT_NAME with the name of the environment.
    • LOCATION with the region where the environment is located.
    • PROJECT_ID with the Project ID.

    Example:

    https://composer.googleapis.com/v1beta1/projects/example-project/locations/us-central1/environments/example-environment:saveSnapshot
    
  • In the --http-method argument, specify post.

  • In the --message-body argument, in the snapshotLocation field, specify the URI of the bucket folder where you want to save the snapshots. As an alternative, you can use the --message-body-from-file argument.

  • In the --oauth-service-account-email argument, specify a service account (see the previous step).

  • In the --oauth-token-scope, specify https://www.googleapis.com/auth/cloud-platform.

gcloud scheduler jobs create http JOB_NAME \
--location=JOB_LOCATION \
--schedule="0 */2 * * *" \
--uri="https://composer.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME:saveSnapshot" \
--http-method="post" \
--message-body="{\"snapshotLocation\": \"SNAPSHOTS_FOLDER\"}" \
--oauth-service-account-email="SERVICE_ACCOUNT_EMAIL" \
--oauth-token-scope="https://www.googleapis.com/auth/cloud-platform"

Replace:

  • JOB_NAME with the name of the job.
  • JOB_LOCATION with the region where the job is located.
  • PROJECT_ID with the Project ID.
  • LOCATION with the region where the environment is located.
  • ENVIRONMENT_NAME with the name of the environment.
  • SNAPSHOTS_FOLDER with the URI of a bucket folder where to save the snapshot.
  • SERVICE_ACCOUNT_EMAIL with the email of a service account from the previous step.

Example:

gcloud scheduler jobs create http example-snapshot-job \
--location=us-central1 \
--schedule="0 */2 * * *" \
--uri="https://composer.googleapis.com/v1beta1/projects/example-project/locations/us-central1/environments/example-environment:saveSnapshot" \
--http-method="post" \
--message-body="{\"snapshotLocation\": \"gs://example-snapshots-bucket/snapshots\"}" \
--oauth-service-account-email="000000000000-compute@developer.gserviceaccount.com" \
--oauth-token-scope="https://www.googleapis.com/auth/cloud-platform"

Terraform

Create a new job using the google_cloud_scheduler_job resource.

  • (Recommended) In the region field, specify the region where your environment and the bucket for snapshots are located. In this way, the job is created in the same region.

  • In the schedule field, in the unix-cron format, specify how often environment snapshots must be taken. For example, to run this job every two hours, specify 0 */2 * * *.

  • In the http_target block:

  • In the http_method field, specify POST.

  • In the uri field, specify the URL for the saveSnapshot REST API method:

    https://composer.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME:saveSnapshot
    

    Replace:

    • ENVIRONMENT_NAME with the name of the environment.
    • LOCATION with the region where the environment is located.
    • PROJECT_ID with the Project ID.

    Example:

    https://composer.googleapis.com/v1beta1/projects/example-project/locations/us-central1/environments/example-environment:saveSnapshot
    
  • In the body field, specify the message body for the API call. In the snapshotLocation field, specify the URI of the bucket folder where you want to save snapshots.

  • In the oauth_token block:

  • In the service_account_email field, specify a service account (see the previous step).

  • In the scope field, specify https://www.googleapis.com/auth/cloud-platform.

resource "google_cloud_scheduler_job" "example" {
name             = "JOB_NAME"
project          = "PROJECT_ID"
region           = "JOB_LOCATION"
description      = "job that creates environment snapshots"
schedule         = "0 */2 * * *"

http_target {
  http_method = "POST"
  uri = "https://composer.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME:saveSnapshot"
  body = base64encode("{\"snapshotLocation\":\"SNAPSHOTS_FOLDER\"}")

  oauth_token {
    service_account_email = "SERVICE_ACCOUNT_EMAIL"
    scope = "https://www.googleapis.com/auth/cloud-platform"
  }
}
}

Replace:

  • JOB_NAME with the name of the job.
  • JOB_LOCATION with the region where the job is located.
  • PROJECT_ID with the Project ID.
  • LOCATION with the region where the environment is located.
  • ENVIRONMENT_NAME with the name of the environment.
  • SNAPSHOTS_FOLDER with the URI of a bucket folder where to save the snapshot.
  • SERVICE_ACCOUNT_EMAIL with the email of a service account from the previous step.

Example:

resource "google_cloud_scheduler_job" "example" {
name             = "example-terraform-scheduled-snapshots-job"
project          = "example-project"
region           = "us-central1"
description      = "job that creates environment snapshots"
schedule         = "0 */2 * * *"

http_target {
  http_method = "POST"
  uri = "https://composer.googleapis.com/v1beta1/projects/example-project/locations/us-central1/environments/example-environment:saveSnapshot"
  body = base64encode("{\"snapshotLocation\":\"gs://example-snapshots-bucket/snapshots\"}")

  oauth_token {
    service_account_email = "000000000000-compute@developer.gserviceaccount.com"
    scope = "https://www.googleapis.com/auth/cloud-platform"
  }
}
}

Step 6. Run the job manually

Check that automated snapshots are configured correctly by running the job.

It takes some time for Cloud Composer to create a snapshot. After the job reports success, your environment's state changes, showing that a snapshot is being created.

Console

On the Cloud Scheduler page, in the Run column, click Run Now.

gcloud

Run the following command:

gcloud scheduler jobs run JOB_NAME \
  --location=JOB_LOCATION

Replace:

  • JOB_NAME with the name of the job.
  • JOB_LOCATION with the region where the job is located.

Example:

gcloud scheduler jobs run example-snapshot-job \
  --location=us-central1

What's next