Scheduling Memorystore for Redis database exports using Cloud Scheduler

Last reviewed 2023-03-18 UTC

---

This tutorial shows how to use Cloud Scheduler and Cloud Functions to automatically export a Memorystore for Redis database to Cloud Storage. Having database exports on Cloud Storage lets you create a robust, diverse disaster recovery plan. For example, you can export to a different region, and import to other Memorystore for Redis instances.

Architecture

This tutorial includes the following Google Cloud components:

• Cloud Scheduler jobs: Jobs to make calls on a set schedule to start the database export.
• Cloud Functions: Functions to export the data from Memorystore to Cloud Storage.
• Pub/Sub messages: Messages sent and received for each data export event.
• Cloud Storage buckets: Bucket to store the exported data.
• Memorystore for Redis: Source database to export the data from.

A Cloud Scheduler job posts a message on a Pub/Sub topic with information about the Memorystore instance ID, the project ID, the region where it's located, and the Cloud Storage location at which to store the backup. This event triggers a Cloud Function that gets this payload and starts a database export on Memorystore for Redis through its API. The database generates the export and saves it to Cloud Storage. The following diagram shows this workflow.

Objectives

• Create a Cloud Storage bucket and a Memorystore instance.
• Create a Pub/Sub topic, a Cloud Function, and a Cloud Scheduler job.
• Trigger the export of database data by running the Cloud Scheduler job manually.

Costs

In this document, you use the following billable components of Google Cloud:

• Cloud Scheduler
• Cloud Functions
• Pub/Sub
• Cloud Storage
• Memorystore for Redis

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.

Before you begin

1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

2. Make sure that billing is enabled for your Google Cloud project. Learn how to check if billing is enabled on a project.

3. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

4. Enable the Memorystore for Redis, Cloud Functions, Cloud Scheduler, and Cloud Build APIs.

   Enable the APIs

Throughout this tutorial, you run all commands from Cloud Shell.

Set up your environment

To get started, you first configure your environment and then create custom roles that have the permissions needed for this tutorial.

1. In Cloud Shell, configure the following environment variables:

   export PROJECT_ID=`gcloud config get-value project`
   export DEMO="mem-exporter"
   export BUCKET_NAME=${USER}-mem-$(date +%s)
   export MEM_INSTANCE="${DEMO}-instance"
   export FUNCTION_NAME="${DEMO}-gcf"
   export PUBSUB_TOPIC="${DEMO}-topic"
   export SCHEDULER_JOB="${DEMO}-job"
   export MEM_EXPORT_ROLE="memExporter"
   export STORAGE_ROLE="simpleStorageRole"
   export REGION="us-central1"
   

2. Create two custom roles that have only the permissions needed for this tutorial:

   gcloud iam roles create ${STORAGE_ROLE} --project=${PROJECT_ID} \
       --title="Simple Storage Role" \
       --description="Grant permissions to view and create objects in Cloud Storage" \
       --permissions="storage.objects.create,storage.buckets.get"
   
   gcloud iam roles create ${MEM_EXPORT_ROLE} --project=${PROJECT_ID} \
       --title="Memorystore Exporter Role" \
       --description="Grant permissions to export data from a Memorystore instance to a Cloud Storage bucket" \
       --permissions="redis.instances.export"
   

   These roles reduce the scope of access of Cloud Functions and Memorystore service accounts, following the principle of least privilege.

Create a Cloud Storage bucket and a Memorystore instance

In this section, you first create a Cloud Storage bucket and a Memorystore for Redis instance. Then you populate the Memorystore with sample data.

Create a Cloud Storage bucket

You use the gsutil command-line tool to create a Cloud Storage bucket.

• Create a Cloud Storage bucket where you want to save the data exports:

  gsutil mb -l ${REGION} gs://${BUCKET_NAME}
  

Create a Memorystore instance and grant permissions to its service account

Next, you create a Memorystore instance and grant its service account the permissions to export data to Cloud Storage.

1. Create a Memorystore for Redis 4 instance:

   gcloud redis instances create ${MEM_INSTANCE} --size=1 --region=${REGION}
   

   This operation takes a few minutes to complete.

2. Verify that the Memorystore instance is READY:

   gcloud redis instances list --region=${REGION}
   

   The output looks similar to the following:

   INSTANCE_NAME   VERSION    REGION       TIER   SIZE_GB  HOST          PORT  NETWORK  RESERVED_IP      STATUS  CREATE_TIME
   redis-instance  REDIS_4_0  us-central1  BASIC  1        10.61.20.131  6379  default  10.61.20.128/29  READY   2020-04-23T18:38:54
   

3. Grant your Memorystore service account the permissions to export data to Cloud Storage with the custom Simple Storage role that you created earlier:

   export MEM_SA=$(gcloud redis instances describe ${MEM_INSTANCE} --region ${REGION} \
       --project ${PROJECT_ID} \
       --format "value(persistenceIamIdentity)")
   
   gsutil iam ch ${MEM_SA}:projects/${PROJECT_ID}/roles/${STORAGE_ROLE} gs://${BUCKET_NAME}
   

Create the scheduled data export task

In this section, you create a custom service account and bind it to the custom Redis role that you create. You then create a Pub/Sub topic that's used to trigger the execution of a Cloud Function. You also create a Cloud Scheduler job to periodically execute the data export function.

Create a service account for the Cloud Function

The first step is to create a service account and bind it to the roles.

1. Create an IAM service account for the Cloud Function to use and save it to the variable:

   gcloud iam service-accounts create ${FUNCTION_NAME} \
       --display-name="Service Account for GCF and Memorystore"
   
   export GCF_SA=$(gcloud iam service-accounts list --filter="${FUNCTION_NAME}" --format="value(email)")
   

2. Grant the service account access to the custom Memorystore Exporter role allowing it to request Memorystore exports:

   gcloud projects add-iam-policy-binding ${PROJECT_ID} \
       --member="serviceAccount:${GCF_SA}" \
       --role="projects/${PROJECT_ID}/roles/${MEM_EXPORT_ROLE}"
   

3. Grant the service-account access to the custom Simple Storage role

   gsutil iam ch \
       serviceAccount:${GCF_SA}:projects/${PROJECT_ID}/roles/${STORAGE_ROLE} \
       gs://${BUCKET_NAME}
   

Create a Pub/Sub topic

The next step is to create a Pub/Sub topic that's used to trigger the Cloud Function that interacts with the Memorystore database.

• Create the Pub/Sub topic:

  gcloud pubsub topics create ${PUBSUB_TOPIC}
  

Create a Cloud Function

Next, you create the Cloud Function.

1. Create a folder for the Cloud Function code:

   mkdir scheduler_gcf_code && cd scheduler_gcf_code
   

2. Create a main.py file by pasting the following into Cloud Shell:

   cat <<EOF  > main.py
   
   import base64
   import logging
   import json
   
   from datetime import datetime
   from httplib2 import Http
   
   from googleapiclient import discovery
   from googleapiclient.errors import HttpError
   from oauth2client.client import GoogleCredentials
   
   def main(event, context):
       pubsub_message = json.loads(base64.b64decode(event['data']).decode('utf-8'))
       credentials = GoogleCredentials.get_application_default()
   
       service = discovery.build('redis', 'v1beta1', http=credentials.authorize(Http()), cache_discovery=False)
   
       datestamp = datetime.now().strftime("%Y%m%d%H%M") # format timestamp: YearMonthDayHourMinute
       instance_name=pubsub_message['name'].split("/")[-1]
       uri = f"{pubsub_message['gs']}/backup-{instance_name}-{datestamp}.rdb"
   
       request_body = {
           "outputConfig": {
               "gcsDestination" : {
                   "uri": uri
               }
           }
       }
   
       try:
           request = service.projects().locations().instances().export(
               name=pubsub_message['name'],
               body=request_body
           )
   
           response = request.execute()
       except HttpError as err:
           logging.error(f"Could NOT run backup. Reason: {err}")
       else:
           logging.info(f"Backup task status: {response}")
   EOF
   

3. Create a requirements.txt file by pasting the following into Cloud Shell:

   cat <<EOF > requirements.txt
   
   google-api-python-client
   Oauth2client
   EOF
   

4. Deploy the code.

   gcloud functions deploy ${FUNCTION_NAME} \
       --trigger-topic=${PUBSUB_TOPIC} \
       --runtime=python37 \
       --entry-point=main \
       --service-account=${FUNCTION_NAME}@${PROJECT_ID}.iam.gserviceaccount.com \
       --ingress-settings=internal-and-gclb
   

Create a Cloud Scheduler job

Finally, you create a Cloud Scheduler job to periodically execute the data export function.

1. Save the Memorystore complete name into a variable:

   export MEM_NAME=$(gcloud redis instances describe ${MEM_INSTANCE} --region ${REGION} --format "value(name)")
   

2. Create a Cloud Scheduler job to periodically execute the data export function:

   gcloud scheduler jobs create pubsub ${SCHEDULER_JOB} \
       --schedule='0 23 * * *' --topic=${PUBSUB_TOPIC} \
       --message-body='{"name":'\"${MEM_NAME}\"',"gs":'\"gs://${BUCKET_NAME}\"'}' \
       --time-zone='America/Los_Angeles' --location=${REGION}
   

   This job is scheduled to run at 11 PM Pacific time every day.

   The message body contains the name of the Memorystore instance to be exported, and the destination Cloud Storage bucket.

Test your solution

The final step is to test your solution. You start by running the Cloud Scheduler job.

1. Run the Cloud Scheduler job manually to trigger a Memorystore export of your database.

   gcloud scheduler jobs run ${SCHEDULER_JOB} --location=${REGION}
   

2. List the operations performed on the Memorystore instance, and verify that there's an operation of type EXPORT:

   gcloud redis operations list --region=${REGION} --filter="${MEM_INSTANCE}"
   

   The following output example shows an export job with a DONE status of True to indicate that it has completed. If the DONE status shows False, this indicates that the job is still processing; wait a minute and then re-run the previous command.

   OPERATION_NAME                                           REGION       TYPE    TARGET                 DONE  CREATE_TIME          DURATION
   operation-1592329364987-5a837122a600c-b22c2703-5077c6b7  us-central1  export  mem-exporter-instance  True  2020-06-16T17:42:45  16S
   

3. Check the Cloud Storage bucket to see if the export file file was created:

   gsutil ls -l gs://${BUCKET_NAME}/*.rdb
   

   You see a file named backup-INSTANCE_NAME-TIMESTAMP.rdb.

Clean up

You can avoid incurring charges to your Google Cloud account for the resources used in this tutorial by following these steps. The easiest way to eliminate billing is to delete the project you created for the tutorial.

Caution: Deleting a project has the following effects:
• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.
1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

• Learn how to schedule compute instances with Cloud Scheduler.
• Learn about scheduling Cloud SQL database exports using Cloud Scheduler.
• Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.