Scheduling Memorystore for Redis database exports using Cloud Scheduler

This tutorial shows how to use Cloud Scheduler and Cloud Functions to automatically export a Memorystore for Redis database to Cloud Storage. Having database exports on Cloud Storage lets you create a robust, diverse disaster recovery plan. For example, you can export to a different region, and import to other Memorystore for Redis instances.

Architecture

This tutorial includes the following Google Cloud components:

A Cloud Scheduler job posts a message on a Pub/Sub topic with information about the Memorystore instance ID, the project ID, the region where it's located, and the Cloud Storage location at which to store the backup. This event triggers a Cloud Function that gets this payload and starts a database export on Memorystore for Redis through its API. The database generates the export and saves it to Cloud Storage. The following diagram shows this workflow.

Workflow from Cloud Scheduler to Pub/Sub, which triggers a Cloud Function that starts the export.

Objectives

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Before you begin

  1. In the Cloud Console, on the project selector page, select or create a Cloud project.

    Go to the project selector page

  2. Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

  3. In the Cloud Console, activate Cloud Shell.

    Activate Cloud Shell

  4. Enable the Memorystore for Redis, Cloud Functions, Cloud Scheduler, and App Engine APIs.

    Enable the APIs

Throughout this tutorial, you run all commands from Cloud Shell.

Setting up your environment

To get started, you first configure your environment and then create custom roles that have the permissions needed for this tutorial.

  1. In Cloud Shell, configure the following environment variables:

    export PROJECT_ID=`gcloud config get-value project`
    export DEMO="mem-exporter"
    export BUCKET_NAME=${USER}-mem-$(date +%s)
    export MEM_INSTANCE="${DEMO}-instance"
    export GCF_NAME="${DEMO}-gcf"
    export PUBSUB_TOPIC="${DEMO}-topic"
    export SCHEDULER_JOB="${DEMO}-job"
    export MEM_ROLE="memExporter"
    export STORAGE_ROLE="simpleStorageRole"
    export REGION="us-central1"
    
  2. Create two custom roles that have only the permissions needed for this tutorial:

    gcloud iam roles create ${STORAGE_ROLE} --project=${PROJECT_ID} \
        --title="Simple Storage Role" \
        --description="Grant permissions to view and create objects in Cloud Storage" \
        --permissions="storage.objects.create,storage.buckets.get"
    
    gcloud iam roles create ${MEM_ROLE} --project=${PROJECT_ID} \
        --title="Memorystore Exporter Role" \
        --description="Grant permissions to export data from a Memorystore instance to a Cloud Storage bucket" \
        --permissions="redis.instances.export"
    

    These roles reduce the scope of access of Cloud Functions and Memorystore service accounts, following the principle of least privilege.

Creating a Cloud Storage bucket and a Memorystore instance

In this section, you first create a Cloud Storage bucket and a Memorystore for Redis instance. Then you populate the Memorystore with sample data.

Create a Cloud Storage bucket

You use the gsutil command-line tool to create a Cloud Storage bucket.

  • Create a Cloud Storage bucket where you want to save the data exports:

    gsutil mb -l ${REGION} gs://${BUCKET_NAME}
    

Create a Memorystore instance and grant permissions to its service account

Next, you create a Memorystore instance and grant its service account the permissions to export data to Cloud Storage.

  1. Create a Memorystore for Redis 4 instance:

    gcloud redis instances create ${MEM_INSTANCE} --size=1 --region=${REGION}
    

    This operation takes a few minutes to complete.

  2. Verify that the Memorystore instance is READY:

    gcloud redis instances list --region=${REGION}
    

    The output looks similar to the following:

    INSTANCE_NAME   VERSION    REGION       TIER   SIZE_GB  HOST          PORT  NETWORK  RESERVED_IP      STATUS  CREATE_TIME
    redis-instance  REDIS_4_0  us-central1  BASIC  1        10.61.20.131  6379  default  10.61.20.128/29  READY   2020-04-23T18:38:54
    
  3. Grant your Memorystore service account the permissions to export data to Cloud Storage with the Simple Storage Role:

    export MEM_SA=$(gcloud redis instances describe ${MEM_INSTANCE} --region ${REGION} \
        --project ${PROJECT_ID} \
        --format "value(persistenceIamIdentity)")
    
    gsutil iam ch ${MEM_SA}:projects/${PROJECT_ID}/roles/${STORAGE_ROLE} gs://${BUCKET_NAME}
    

Creating a Pub/Sub topic, a Cloud Function, and a Cloud Scheduler job

In this section, you create a custom service account and bind it to the custom Redis role that you create. You then create a Pub/Sub topic that's used to trigger the execution of a Cloud Function. You also create a Cloud Scheduler job to periodically execute the data export function.

Create a service account for the Cloud Function

The first step is to create a custom service account and bind it to the custom Redis role that you create.

  1. Create an IAM service account for the Cloud Function to use and save it to the variable:

    gcloud iam service-accounts create ${GCF_NAME} \
        --display-name="Service Account for GCF and Memorystore"
    
    export GCF_SA=$(gcloud iam service-accounts list --filter="${GCF_NAME}" --format="value(email)")
    
  2. Grant the Cloud Function service account access to the custom Redis role:

    gcloud projects add-iam-policy-binding ${PROJECT_ID} \
        --member="serviceAccount:${GCF_SA}" \
        --role="projects/${PROJECT_ID}/roles/${MEM_ROLE}"
    
  3. Grant the Cloud Function service-account access to the custom Storage role:

    gsutil iam ch \
        serviceAccount:${GCF_SA}:projects/${PROJECT_ID}/roles/${STORAGE_ROLE} \
        gs://${BUCKET_NAME}
    

Create a Pub/Sub topic

The next step is to create a Pub/Sub topic that's used to trigger the Cloud Function that interacts with the Memorystore database.

  • Create the Pub/Sub topic:

    gcloud pubsub topics create ${PUBSUB_TOPIC}
    

Create a Cloud Function

Next, you create the Cloud Function.

  1. Create a folder for the Cloud Function code:

    mkdir scheduler_gcf_code && cd scheduler_gcf_code
    
  2. Create a main.py file by pasting the following into Cloud Shell:

    cat <<EOF  > main.py
    
    import base64
    import logging
    import json
    
    from datetime import datetime
    from httplib2 import Http
    
    from googleapiclient import discovery
    from googleapiclient.errors import HttpError
    from oauth2client.client import GoogleCredentials
    
    def main(event, context):
        pubsub_message = json.loads(base64.b64decode(event['data']).decode('utf-8'))
        credentials = GoogleCredentials.get_application_default()
    
        service = discovery.build('redis', 'v1beta1', http=credentials.authorize(Http()), cache_discovery=False)
    
        datestamp = datetime.now().strftime("%Y%m%d%H%M") # format timestamp: YearMonthDayHourMinute
        uri = f"{pubsub_message['gs']}/backup-{datestamp}.rdb"
    
        request_body = {
            "outputConfig": {
                "gcsDestination" : {
                    "uri": uri
                }
            }
        }
    
        try:
            request = service.projects().locations().instances().export(
                name=pubsub_message['name'],
                body=request_body
            )
    
            response = request.execute()
        except HttpError as err:
            logging.error(f"Could NOT run backup. Reason: {err}")
        else:
            logging.info(f"Backup task status: {response}")
    EOF
    
  3. Create a requirements.txt file by pasting the following into Cloud Shell:

    cat <<EOF > requirements.txt
    
    google-api-python-client
    Oauth2client
    EOF
    
  4. Deploy the code. When you are asked if you want to allow unauthenticated invocations of the new function, answer no.

    gcloud functions deploy ${GCF_NAME} \
        --trigger-topic=${PUBSUB_TOPIC} \
        --runtime=python37 \
        --entry-point=main \
        --service-account=${GCF_NAME}@${PROJECT_ID}.iam.gserviceaccount.com
    

Create a Cloud Scheduler job

Finally, you create a Cloud Scheduler job to periodically execute the data export function.

  1. Create an App Engine instance for the Cloud Scheduler job:

    gcloud app create --region=${REGION::-1}
    
  2. Save the Memorystore complete name into a variable:

    export MEM_NAME=$(gcloud redis instances describe ${MEM_INSTANCE} --region ${REGION} --format "value(name)")
    
  3. Create a Cloud Scheduler job to periodically execute the data export function:

    gcloud scheduler jobs create pubsub ${SCHEDULER_JOB} \
        --schedule='0 23 * * *' --topic=${PUBSUB_TOPIC} \
        --message-body='{"name":'\"${MEM_NAME}\"',"gs":'\"gs://${BUCKET_NAME}\"'}' \
        --time-zone='America/Los_Angeles'
    

    This job is scheduled to run at 11 PM every day.

Testing your solution

The final step is to test your solution. You start by running the Cloud Scheduler job.

  1. Run the Cloud Scheduler job manually to trigger a Memorystore export of your database.

    gcloud scheduler jobs run ${SCHEDULER_JOB}
    
  2. List the operations performed on the Memorystore instance, and verify that there's an operation of type EXPORT:

    gcloud redis operations list --region=${REGION} --filter="${MEM_INSTANCE}"
    

    The output shows a completed export job—for example:

    OPERATION_NAME                                           REGION       TYPE    TARGET                 DONE  CREATE_TIME          DURATION
    operation-1592329364987-5a837122a600c-b22c2703-5077c6b7  us-central1  export  mem-exporter-instance  True  2020-06-16T17:42:45  16S
    
  3. Check the Cloud Storage bucket to see if the export file file was created:

    gsutil ls gs://${BUCKET_NAME} | grep rdb
    

    You see a file named backup-mem-timestamp.rdb after the STATUS operation from the previous step returns DONE.

Cleaning up

You can avoid incurring charges to your Google Cloud account for the resources used in this tutorial by following these steps. The easiest way to eliminate billing is to delete the project you created for the tutorial.

  1. In the Cloud Console, go to the Manage resources page.

    Go to the Manage resources page

  2. In the project list, select the project that you want to delete and then click Delete .
  3. In the dialog, type the project ID and then click Shut down to delete the project.

What's next