The import and export feature for Cloud Memorystore for Redis is in beta. This page explains how imports and exports work in Cloud Memorystore for Redis.
The import and export feature uses the native RDB snapshot feature of Redis to import data into or export data out of a Cloud Memorystore for Redis instance. The use of the native RDB format prevents lock-in and makes it very easy to move data within Google Cloud Platform or outside of Google Cloud Platform. Import and export uses Cloud Storage buckets to store RDB files.
The behavior of a Cloud Memorystore for Redis instance during export is very similar
BGSAVE command in open source Redis. Cloud Memorystore for Redis imports by
copying your RDB file into your instance's data directory.
Import feature behavior
You can only import RDB files that are stored in Cloud Storage buckets in the same region as the Cloud Memorystore instance.
Your Cloud Storage bucket's default storage class needs to be Regional Storage.
Your Redis instance is unavailable during an import operation. Your instance becomes available after the import operation finishes.
If an import is successful, the existing data in the instance is overwritten by the data in the RDB file.
If an import fails for any reason, the instance is brought online, but the data may be fully flushed. You can retry the import using the same RDB file, or you may use another RDB file to restore data.
Your Cloud Memorystore for Redis instance can import RDB files from the same Redis version and from a previous Redis version, but it cannot import RDB files from a newer version.
An import operation can be cancelled, but you should note that the cancellation results in your instance recovering with a fully flushed cache. This is the case for both Basic Tier and Standard Tier instances.
Export feature behavior
You can read and write data to your Redis instance during an export operation; however, you cannot perform any admin operations like scaling, updating, or configuring your instance.
Your instance may experience increased latency during the export.
During an export, the RDB file is saved to a Cloud Storage bucket of your choosing.
You can cancel an export operation at any time. Cancelling does not impact the data or availability of an instance.
- Cancelling an export deletes the RDB file being written to the
Cloud Storage bucket, and frees up memory used by the
- Cancelling an export deletes the RDB file being written to the Cloud Storage bucket, and frees up memory used by the
An export can overwrite an existing file with the same name with no extra permissions when using the GCP Console. To export over an existing file with
gcloud, the user account needs the
Exporting under high write load
If you run an export during a period of high write-load your instance memory usage can increase up to 2X, causing the export operation to fail. You should perform export operations during periods of low writes. To monitor how much Redis data is stored in your instance you can create a dashboard to monitor your memory usage and monitor the instance memory usage metrics with Stackdriver.
Cloud Memorystore for Redis export uses the
BGSAVE feature of Redis to take a
snapshot of the data in an instance. When executing
BGSAVE, Redis forks a new
child process to take the snapshot. Redis utilizes copy-on-write
during this process.
This means that the initial fork allocates minimal extra memory, but any pages changed by Redis server write traffic are duplicated. Therefore, under periods of high writes, in the worst case, the memory footprint of Redis doubles, causing the export operation to fail.
Exporting during a BGSAVE operation
You cannot export while a
BGSAVE operation is running. Standard Tier
Cloud Memorystore for Redis instances sometimes run full resynchronization
operations, or 'full syncs', to keep the master and the replica synchronized. A
full resynchronization uses the
BGSAVE command, which prevents
Cloud Memorystore from performing an export.
To verify if a
BGSAVE operation is ongoing, view the
redis.googleapis.com/persistence/rdb/bgsave_in_progress API endpoint with
Stackdriver or your own
command line API tools.
Required permissions for importing and exporting
A user needs to configure sufficient Cloud IAM permissions to import or export RDB files. The simplest permissions setup is to apply the Cloud Memorystore Redis Admin and Storage Admin IAM roles to the user account, however this setup may be inappropriate for some user accounts because it gives them permissions to create, edit, or delete resources they shouldn't have access to. For instructions on setting minimally restricted permissions see Granting restricted permissions for import and export.
Duration of imports and exports
The duration of the import or export depends on the instance size and the size of the data set in the instance. Depending on the size, the imports and exports can take anywhere from a few minutes to a few hours.
For example, if you import or export a 10 GB instance with 8 GB of used memory, the operation can take 5 minutes. If you import or export a 120 GB instance with 100 GB of used memory, the operation can take an hour.
Stopping an import or export
To stop an import or export, refer to the following pages:
Monitoring progress of the operation
The time estimate for completion is not available while doing import and export. You can check whether the operation is in progress by checking the instance details page. To view the instance details page, click your instance ID on the Cloud Memorystore for Redis instances list page.
You can also monitor the progress of an import operation by running the following command:
gcloud redis instances describe [INSTANCE_ID] --region=[REGION]
[INSTANCE_ID]is the ID for your Cloud Memorystore for Redis instance.
[REGION]is the region your instance is in.
The command outputs
state: IMPORTING or
state: READY as one of the
Selecting appropriate Cloud Storage buckets
To use import and export, your Cloud Storage buckets must:
- Be in the same region as your Cloud Memorystore for Redis instance.
- Have its default storage class set to Regional Storage.
- Not use the Requester Pays feature.
If you need to export to a bucket in another region you can export to a Regional Storage bucket first, and then use Cloud Storage tools to copy the file over to the Multi-Regional Storage bucket.
Securing your Cloud Storage buckets
You should ensure that your Cloud Storage bucket has the right restrictions to prevent unauthorized access to the Redis RDB backup it contains. Please refer to Cloud Storage Access Control Options on how to secure your Cloud Storage buckets.
Ensuring sufficient memory to complete an export
To prevent export from causing memory starvation, you should have 50% free memory in the instance during the export process. This may not be required all the time, depending on your write load, but having 50% free memory ensures that there is enough free space to accommodate copy-on-write overhead, if many pages are touched during the export process.
To understand the current memory usage of your instance, set up a Stackdriver alert to notify you when your memory usage reaches a customizable threshold. Additionally you should set up a memory monitoring dashboard with Stackdriver. These measures allow you to understand your instances memory usage patterns. Understanding memory usage patterns enables you to size your instance correctly when exporting at regular intervals.