Scheduling Memorystore for Redis database exports using Cloud Scheduler

Last reviewed 2023-03-18 UTC

This tutorial shows how to use Cloud Scheduler and Cloud Functions to automatically export a Memorystore for Redis database to Cloud Storage. Having database exports on Cloud Storage lets you create a robust, diverse disaster recovery plan. For example, you can export to a different region, and import to other Memorystore for Redis instances.

Architecture

This tutorial includes the following Google Cloud components:

A Cloud Scheduler job posts a message on a Pub/Sub topic with information about the Memorystore instance ID, the project ID, the region where it's located, and the Cloud Storage location at which to store the backup. This event triggers a Cloud Function that gets this payload and starts a database export on Memorystore for Redis through its API. The database generates the export and saves it to Cloud Storage. The following diagram shows this workflow.

Workflow from Cloud Scheduler to Pub/Sub, which triggers a Cloud Function that starts the export.

Objectives

Costs

In this document, you use the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

  1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  2. Make sure that billing is enabled for your Google Cloud project.

  3. In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

  4. Enable the Memorystore for Redis, Cloud Functions, Cloud Scheduler, and Cloud Build APIs.

    Enable the APIs

Throughout this tutorial, you run all commands from Cloud Shell.

Set up your environment

To get started, you first configure your environment and then create custom roles that have the permissions needed for this tutorial.

  1. In Cloud Shell, configure the following environment variables:

    export PROJECT_ID=`gcloud config get-value project`
    export DEMO="mem-exporter"
    export BUCKET_NAME=${USER}-mem-$(date +%s)
    export MEM_INSTANCE="${DEMO}-instance"
    export FUNCTION_NAME="${DEMO}-gcf"
    export PUBSUB_TOPIC="${DEMO}-topic"
    export SCHEDULER_JOB="${DEMO}-job"
    export MEM_EXPORT_ROLE="memExporter"
    export STORAGE_ROLE="simpleStorageRole"
    export REGION="us-central1"
    
  2. Create two custom roles that have only the permissions needed for this tutorial:

    gcloud iam roles create ${STORAGE_ROLE} --project=${PROJECT_ID} \
        --title="Simple Storage Role" \
        --description="Grant permissions to view and create objects in Cloud Storage" \
        --permissions="storage.objects.create,storage.buckets.get"
    
    gcloud iam roles create ${MEM_EXPORT_ROLE} --project=${PROJECT_ID} \
        --title="Memorystore Exporter Role" \
        --description="Grant permissions to export data from a Memorystore instance to a Cloud Storage bucket" \
        --permissions="redis.instances.export"
    

    These roles reduce the scope of access of Cloud Functions and Memorystore service accounts, following the principle of least privilege.

Create a Cloud Storage bucket and a Memorystore instance

In this section, you first create a Cloud Storage bucket and a Memorystore for Redis instance. Then you populate the Memorystore with sample data.

Create a Cloud Storage bucket

You use the gsutil command-line tool to create a Cloud Storage bucket.

  • Create a Cloud Storage bucket where you want to save the data exports:

    gsutil mb -l ${REGION} gs://${BUCKET_NAME}
    

Create a Memorystore instance and grant permissions to its service account

Next, you create a Memorystore instance and grant its service account the permissions to export data to Cloud Storage.

  1. Create a Memorystore for Redis 4 instance:

    gcloud redis instances create ${MEM_INSTANCE} --size=1 --region=${REGION}
    

    This operation takes a few minutes to complete.

  2. Verify that the Memorystore instance is READY:

    gcloud redis instances list --region=${REGION}
    

    The output looks similar to the following:

    INSTANCE_NAME   VERSION    REGION       TIER   SIZE_GB  HOST          PORT  NETWORK  RESERVED_IP      STATUS  CREATE_TIME
    redis-instance  REDIS_4_0  us-central1  BASIC  1        10.61.20.131  6379  default  10.61.20.128/29  READY   2020-04-23T18:38:54
    
  3. Grant your Memorystore service account the permissions to export data to Cloud Storage with the custom Simple Storage role that you created earlier:

    export MEM_SA=$(gcloud redis instances describe ${MEM_INSTANCE} --region ${REGION} \
        --project ${PROJECT_ID} \
        --format "value(persistenceIamIdentity)")
    
    gsutil iam ch ${MEM_SA}:projects/${PROJECT_ID}/roles/${STORAGE_ROLE} gs://${BUCKET_NAME}
    

Create the scheduled data export task

In this section, you create a custom service account and bind it to the custom Redis role that you create. You then create a Pub/Sub topic that's used to trigger the execution of a Cloud Function. You also create a Cloud Scheduler job to periodically execute the data export function.

Create a service account for the Cloud Function

The first step is to create a service account and bind it to the roles.

  1. Create an IAM service account for the Cloud Function to use and save it to the variable:

    gcloud iam service-accounts create ${FUNCTION_NAME} \
        --display-name="Service Account for GCF and Memorystore"
    
    export GCF_SA=$(gcloud iam service-accounts list --filter="${FUNCTION_NAME}" --format="value(email)")
    
  2. Grant the service account access to the custom Memorystore Exporter role allowing it to request Memorystore exports:

    gcloud projects add-iam-policy-binding ${PROJECT_ID} \
        --member="serviceAccount:${GCF_SA}" \
        --role="projects/${PROJECT_ID}/roles/${MEM_EXPORT_ROLE}"
    
  3. Grant the service-account access to the custom Simple Storage role

    gsutil iam ch \
        serviceAccount:${GCF_SA}:projects/${PROJECT_ID}/roles/${STORAGE_ROLE} \
        gs://${BUCKET_NAME}
    

Create a Pub/Sub topic

The next step is to create a Pub/Sub topic that's used to trigger the Cloud Function that interacts with the Memorystore database.

  • Create the Pub/Sub topic:

    gcloud pubsub topics create ${PUBSUB_TOPIC}
    

Create a Cloud Function

Next, you create the Cloud Function.

  1. Create a folder for the Cloud Function code:

    mkdir scheduler_gcf_code && cd scheduler_gcf_code
    
  2. Create a main.py file by pasting the following into Cloud Shell:

    cat <<EOF  > main.py
    
    import base64
    import logging
    import json
    
    from datetime import datetime
    from httplib2 import Http
    
    from googleapiclient import discovery
    from googleapiclient.errors import HttpError
    from oauth2client.client import GoogleCredentials
    
    def main(event, context):
        pubsub_message = json.loads(base64.b64decode(event['data']).decode('utf-8'))
        credentials = GoogleCredentials.get_application_default()
    
        service = discovery.build('redis', 'v1beta1', http=credentials.authorize(Http()), cache_discovery=False)
    
        datestamp = datetime.now().strftime("%Y%m%d%H%M") # format timestamp: YearMonthDayHourMinute
        instance_name=pubsub_message['name'].split("/")[-1]
        uri = f"{pubsub_message['gs']}/backup-{instance_name}-{datestamp}.rdb"
    
        request_body = {
            "outputConfig": {
                "gcsDestination" : {
                    "uri": uri
                }
            }
        }
    
        try:
            request = service.projects().locations().instances().export(
                name=pubsub_message['name'],
                body=request_body
            )
    
            response = request.execute()
        except HttpError as err:
            logging.error(f"Could NOT run backup. Reason: {err}")
        else:
            logging.info(f"Backup task status: {response}")
    EOF
    
  3. Create a requirements.txt file by pasting the following into Cloud Shell:

    cat <<EOF > requirements.txt
    
    google-api-python-client
    Oauth2client
    EOF
    
  4. Deploy the code.

    gcloud functions deploy ${FUNCTION_NAME} \
        --trigger-topic=${PUBSUB_TOPIC} \
        --runtime=python37 \
        --entry-point=main \
        --service-account=${FUNCTION_NAME}@${PROJECT_ID}.iam.gserviceaccount.com \
        --ingress-settings=internal-and-gclb
    

Create a Cloud Scheduler job

Finally, you create a Cloud Scheduler job to periodically execute the data export function.

  1. Save the Memorystore complete name into a variable:

    export MEM_NAME=$(gcloud redis instances describe ${MEM_INSTANCE} --region ${REGION} --format "value(name)")
    
  2. Create a Cloud Scheduler job to periodically execute the data export function:

    gcloud scheduler jobs create pubsub ${SCHEDULER_JOB} \
        --schedule='0 23 * * *' --topic=${PUBSUB_TOPIC} \
        --message-body='{"name":'\"${MEM_NAME}\"',"gs":'\"gs://${BUCKET_NAME}\"'}' \
        --time-zone='America/Los_Angeles' --location=${REGION}
    

    This job is scheduled to run at 11 PM Pacific time every day.

    The message body contains the name of the Memorystore instance to be exported, and the destination Cloud Storage bucket.

Test your solution

The final step is to test your solution. You start by running the Cloud Scheduler job.

  1. Run the Cloud Scheduler job manually to trigger a Memorystore export of your database.

    gcloud scheduler jobs run ${SCHEDULER_JOB} --location=${REGION}
    
  2. List the operations performed on the Memorystore instance, and verify that there's an operation of type EXPORT:

    gcloud redis operations list --region=${REGION} --filter="${MEM_INSTANCE}"
    

    The following output example shows an export job with a DONE status of True to indicate that it has completed. If the DONE status shows False, this indicates that the job is still processing; wait a minute and then re-run the previous command.

    OPERATION_NAME                                           REGION       TYPE    TARGET                 DONE  CREATE_TIME          DURATION
    operation-1592329364987-5a837122a600c-b22c2703-5077c6b7  us-central1  export  mem-exporter-instance  True  2020-06-16T17:42:45  16S
    
  3. Check the Cloud Storage bucket to see if the export file file was created:

    gsutil ls -l gs://${BUCKET_NAME}/*.rdb
    

    You see a file named backup-INSTANCE_NAME-TIMESTAMP.rdb.

Clean up

You can avoid incurring charges to your Google Cloud account for the resources used in this tutorial by following these steps. The easiest way to eliminate billing is to delete the project you created for the tutorial.

  1. In the Google Cloud console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next