Cloud Composer 1 | Cloud Composer 2
This page explains how to set up a Cloud Scheduler job that saves snapshots of your environment on a schedule.
For more information about how environment snapshots work and how to save snapshots manually, see Save and load snapshots.
How scheduled snapshots work
Scheduled snapshots use Cloud Scheduler and Cloud Composer API. A Cloud Scheduler Cron Job periodically saves snapshots of your environment to a Cloud Storage bucket.
Cloud Composer saves snapshots in the environment's bucket by default. This guide uses a different approach. For scheduled snapshots, we recommend to save snapshots in a separate bucket. In this way, you can set up permissions and a lifecycle configuration for this bucket. One bucket can hold snapshots from several environments, and you can also create buckets that are geo-redundant.
Before you begin
Cloud Composer supports environment snapshots in 2.0.9 and later versions.
Cloud Scheduler is not supported in the VPC Service Controls configuration.
Enable the Cloud Scheduler API:
Console
Enable the Cloud Scheduler API.
gcloud
Enable the Cloud Scheduler API.
gcloud services enable cloudscheduler.googleapis.com
Step 1. Create a bucket for scheduled snapshots
Create a bucket in the same region where your environment is located. By doing so you reduce the costs of transferring data between regions.
For example, if your environment is located in
us-central1
, select us-central1
as the region for your bucket.
Step 2. Configure permissions for the bucket
Configure the following permissions:
The service account of your environment must have read and write permissions for this bucket. For example, the Storage Object Admin role has such permissions.
User accounts do not need any additional permissions for the bucket to load snapshots from it, unless you want to view the contents of the bucket from Google Cloud console. In this case a user account must have read permission on the bucket.
Step 3. Set a lifecycle configuration for the bucket
To save storage costs, you can configure a rule that deletes environment snapshots after a certain period of time.
For example, to automatically delete snapshots older than 30 days:
- Set a lifecycle configuration for the bucket.
- Specify the Delete object action.
- Select Age and 30 days as the condition.
Step 4. Choose a service account for the Cloud Scheduler job
A Cloud Scheduler job, which you create in the next step, requires a service account to make calls to Cloud Composer API.
You can use one of the options:
Select the default compute service account.
Use a different service account that has the
composer.environments.update
permission.
Step 5. Create the Cloud Scheduler job
Console
Create a new job with the following parameters:
In Google Cloud console, go to the Cloud Scheduler page.
(Recommended) In the Region drop-down list, select the same region where your environment and the bucket for snapshots are located.
In the Frequency field, in the unix-cron format, specify how often environment snapshots must be taken. For example, to run this job every two hours, specify
0 */2 * * *
.Expand the Configure the execution section.
In the Target type drop-down list, select HTTP.
In the URL field, enter the URL for the
saveSnapshot
REST API method:https://composer.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME:saveSnapshot
Replace:
ENVIRONMENT_NAME
with the name of the environment.LOCATION
with the region where the environment is located.PROJECT_ID
with the Project ID.
Example:
https://composer.googleapis.com/v1beta1/projects/example-project/locations/us-central1/environments/example-environment:saveSnapshot
In the HTTP method drop-down list, select POST.
In the Body field, in the
snapshotLocation
field, specify the URI of the bucket folder where you want to save snapshots.{ "snapshotLocation": "SNAPSHOTS_FOLDER" }
Replace:
SNAPSHOTS_FOLDER
with the URI of a bucket folder where to save the snapshot.
Example:
{ "snapshotLocation": "gs://example-snapshots-bucket/snapshots" }
In the Auth header drop-down list, select Add OAuth token.
In the Service account drop-down list, select a service account ( see the previous step).
In the Scope field, specify
https://www.googleapis.com/auth/cloud-platform
.
gcloud
Create a new job with the following parameters:
(Recommended) In the
--location
argument, specify the region where your environment and the bucket for snapshots are located. In this way, the job is created in the same region.In the
--schedule
argument, in the unix-cron format, specify how often environment snapshots must be taken. For example, to run this job every two hours, specify0 */2 * * *
.In the
--uri
argument, specify the URL for thesaveSnapshot
REST API method:https://composer.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME:saveSnapshot
Replace:
ENVIRONMENT_NAME
with the name of the environment.LOCATION
with the region where the environment is located.PROJECT_ID
with the Project ID.
Example:
https://composer.googleapis.com/v1beta1/projects/example-project/locations/us-central1/environments/example-environment:saveSnapshot
In the
--http-method
argument, specifypost
.In the
--message-body
argument, in thesnapshotLocation
field, specify the URI of the bucket folder where you want to save the snapshots. As an alternative, you can use the--message-body-from-file
argument.In the
--oauth-service-account-email
argument, specify a service account (see the previous step).In the
--oauth-token-scope
, specifyhttps://www.googleapis.com/auth/cloud-platform
.
gcloud scheduler jobs create http JOB_NAME \
--location=JOB_LOCATION \
--schedule="0 */2 * * *" \
--uri="https://composer.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME:saveSnapshot" \
--http-method="post" \
--message-body="{\"snapshotLocation\": \"SNAPSHOTS_FOLDER\"}" \
--oauth-service-account-email="SERVICE_ACCOUNT_EMAIL" \
--oauth-token-scope="https://www.googleapis.com/auth/cloud-platform"
Replace:
JOB_NAME
with the name of the job.JOB_LOCATION
with the region where the job is located.PROJECT_ID
with the Project ID.LOCATION
with the region where the environment is located.ENVIRONMENT_NAME
with the name of the environment.SNAPSHOTS_FOLDER
with the URI of a bucket folder where to save the snapshot.SERVICE_ACCOUNT_EMAIL
with the email of a service account from the previous step.
Example:
gcloud scheduler jobs create http example-snapshot-job \
--location=us-central1 \
--schedule="0 */2 * * *" \
--uri="https://composer.googleapis.com/v1beta1/projects/example-project/locations/us-central1/environments/example-environment:saveSnapshot" \
--http-method="post" \
--message-body="{\"snapshotLocation\": \"gs://example-snapshots-bucket/snapshots\"}" \
--oauth-service-account-email="000000000000-compute@developer.gserviceaccount.com" \
--oauth-token-scope="https://www.googleapis.com/auth/cloud-platform"
Terraform
Create a new job using the google_cloud_scheduler_job
resource.
(Recommended) In the
region
field, specify the region where your environment and the bucket for snapshots are located. In this way, the job is created in the same region.In the
schedule
field, in the unix-cron format, specify how often environment snapshots must be taken. For example, to run this job every two hours, specify0 */2 * * *
.
In the
http_target
block:In the
http_method
field, specifyPOST
.In the
uri
field, specify the URL for thesaveSnapshot
REST API method:https://composer.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME:saveSnapshot
Replace:
ENVIRONMENT_NAME
with the name of the environment.LOCATION
with the region where the environment is located.PROJECT_ID
with the Project ID.
Example:
https://composer.googleapis.com/v1beta1/projects/example-project/locations/us-central1/environments/example-environment:saveSnapshot
In the
body
field, specify the message body for the API call. In thesnapshotLocation
field, specify the URI of the bucket folder where you want to save snapshots.In the
oauth_token
block:In the
service_account_email
field, specify a service account (see the previous step).In the
scope
field, specifyhttps://www.googleapis.com/auth/cloud-platform
.
resource "google_cloud_scheduler_job" "example" {
name = "JOB_NAME"
project = "PROJECT_ID"
region = "JOB_LOCATION"
description = "job that creates environment snapshots"
schedule = "0 */2 * * *"
http_target {
http_method = "POST"
uri = "https://composer.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME:saveSnapshot"
body = base64encode("{\"snapshotLocation\":\"SNAPSHOTS_FOLDER\"}")
oauth_token {
service_account_email = "SERVICE_ACCOUNT_EMAIL"
scope = "https://www.googleapis.com/auth/cloud-platform"
}
}
}
Replace:
JOB_NAME
with the name of the job.JOB_LOCATION
with the region where the job is located.PROJECT_ID
with the Project ID.LOCATION
with the region where the environment is located.ENVIRONMENT_NAME
with the name of the environment.SNAPSHOTS_FOLDER
with the URI of a bucket folder where to save the snapshot.SERVICE_ACCOUNT_EMAIL
with the email of a service account from the previous step.
Example:
resource "google_cloud_scheduler_job" "example" {
name = "example-terraform-scheduled-snapshots-job"
project = "example-project"
region = "us-central1"
description = "job that creates environment snapshots"
schedule = "0 */2 * * *"
http_target {
http_method = "POST"
uri = "https://composer.googleapis.com/v1beta1/projects/example-project/locations/us-central1/environments/example-environment:saveSnapshot"
body = base64encode("{\"snapshotLocation\":\"gs://example-snapshots-bucket/snapshots\"}")
oauth_token {
service_account_email = "000000000000-compute@developer.gserviceaccount.com"
scope = "https://www.googleapis.com/auth/cloud-platform"
}
}
}
Step 6. Run the job manually
Check that automated snapshots are configured correctly by running the job.
It takes some time for Cloud Composer to create a snapshot. After the job reports success, your environment's state changes, showing that a snapshot is being created.
Console
On the Cloud Scheduler page, in the Run column, click Run Now.
gcloud
Run the following command:
gcloud scheduler jobs run JOB_NAME \
--location=JOB_LOCATION
Replace:
JOB_NAME
with the name of the job.JOB_LOCATION
with the region where the job is located.
Example:
gcloud scheduler jobs run example-snapshot-job \
--location=us-central1