Migrating Memcache to Memorystore

High performance scalable Python web applications often use a distributed in-memory data cache instead of robust persistent storage for some tasks.

The App Engine solution for this is Memcache, a distributed in-memory datastore that is used as a cache for specific tasks.

When migrating off legacy bundled services, the recommended replacement for App Engine Memcache is Memorystore, a fully-managed cloud-based caching service that supports open source caching engines, Redis and Memcached. This guide covers using Memorystore for Redis, which can build application caches that provide sub-millisecond data access.

If your Python app uses Memcache to only reduce latency for ndb or Cloud NDB requests, you can use Cloud NDB's built-in support for Redis, instead of Memcache or Memorystore for Redis.

Before getting started, make sure your app will stay within the Memorystore for Redis quotas.

When to use a memory cache for Python apps

In your Python apps, session data, user preferences, and other data returned by queries for web pages are good candidates for caching. In general, if a frequently run query returns a set of results that do not need to appear in your app immediately, you can cache the results. Subsequent requests can check the cache and only query the database if the results are absent or have expired.

If you store a value only in Memorystore without backing it up in persistent storage, be sure that your application behaves acceptably if the value expires and is removed from the cache. For example, if the sudden absence of a user's session data would cause the session to malfunction, that data should probably be stored in the database in addition to Memorystore.

Before you begin

If you have not done so already, set up your Python development environment to use a Python version that is compatible with Google Cloud, and install testing tools for creating isolated Python environments.

Understanding Memorystore permissions

Every interaction with a Google Cloud service needs to be authorized. For example, to interact with a Redis database hosted by Memorystore, your app needs to supply the credentials of an account that is authorized to access Memorystore.

By default, your app supplies the credentials of the App Engine default service account, which is authorized to access databases in the same project as your app.

If any of the following conditions are true, you need to use an alternative authentication technique that explicitly provides credentials:

Your app and the Memorystore database are in different Google Cloud projects.
You have changed the roles assigned to the default App Engine service account.

For information about alternative authentication techniques, see Setting up Authentication for Server to Server Production Applications.

Overview of the migration process

To use Memorystore instead of Memcache in your Python app:

Set up Memorystore for Redis, which requires you to create a Redis instance on Memorystore and create a Serverless VPC Access that your app uses to communicate with the Redis instance. The order of creating these two independent entities is not strict and can be set up in any order. The instructions in this guide show setting up Serverless VPC Access first.
Install a client library for Redis and use Redis commands to cache data.

Memorystore for Redis is compatible with any client library for Redis.

This guide describes using the redis-py client library to send Redis commands from your app.
Test your updates.
Deploy your app to App Engine.

Setting up Memorystore for Redis

To set up Memorystore for Redis:

Connect your App Engine to a VPC network. Your app can only communicate with Memorystore through a VPC connector.

Be sure to add the VPC connection information to your app.yaml file as described in Configuring your app use the connector.
Note the IP address and port number of the Redis instance you create. You will use this information when you create a Redis client in your code.
Create a Redis instance in Memorystore.

When prompted to select a region for your Redis instance, select the same region in which your App Engine app is located.

Installing dependencies

To use the redis-py client library:

Update the app.yaml file. Follow the instructions for your version of Python:
Python 2
For Python 2 apps, add the latest versions of grpcio and setuptools libraries.

The following is an example app.yaml file:
```
runtime: python27
threadsafe: yes
api_version: 1

libraries:
- name: grpcio
  version: latest
- name: setuptools
  version: latest
```
Python 3
For Python 3 apps, specify the runtime element in your app.yaml file with a supported Python 3 version. For example:
```
runtime: python310 # or another support version
```
The Python 3 runtime installs libraries automatically, so you do not need to specify built-in libraries from the previous Python 2 runtime. If your Python 3 app is using other legacy bundled services when migrating, you can continue to specify the necessary built-in libraries. Otherwise, you can delete the unnecessary lines in your app.yaml file.
Update the requirements.txt file. Follow the instructions for your version of Python:
Python 2
Add the Cloud Client Libraries for Memorystore for Redis to your list of dependencies in the requirements.txt file.
```
redis
```
Run pip install -t lib -r requirements.txt to update the list of available libraries for your app.
Python 3
Add the Cloud Client Libraries for Memorystore for Redis to your list of dependencies in the requirements.txt file.
```
redis
```
App Engine automatically installs these dependencies during app deployment in the Python 3 runtime, so delete the lib folder if one exists.

For Python 2 apps, if your app is using built-in or copied libraries specified in the lib directory, you must specify those paths in the appengine_config.py file, located in the same folder as your app.yaml file:

import pkg_resources
from google.appengine.ext import vendor

# Set PATH to your libraries folder.
PATH = 'lib'
# Add libraries installed in the PATH folder.
vendor.add(PATH)
# Add libraries to pkg_resources working set to find the distribution.
pkg_resources.working_set.add_entry(PATH)

Creating a Redis client

To interact with a Redis database, your code needs to create a Redis client to manage the connection to your Redis database. The following sections describe creating a Redis client using the redis-py client library.

Specifying environment variables

The redis-py client library uses two environment variables to assemble the URL for your Redis database:

A variable to identify the IP address of the Redis database you created in Memorystore.
A variable to identify the port number of the Redis database you created in Memorystore.

We recommend you define these variables in your app's app.yaml file instead of defining them directly in your code. This makes it easier to run your app in different environments, such as a local environment and App Engine.

For example, add the following lines to your app.yaml file:

 env_variables:
      REDISHOST: '10.112.12.112'
      REDISPORT: '6379'

Importing redis-py and creating the client

After you define the REDISHOST and REDISPORT environment variables, use the following lines to import the redis-py library and create a client:

  import redis

  redis_host = os.environ.get('REDISHOST', 'localhost')
  redis_port = int(os.environ.get('REDISPORT', 6379))
  redis_client = redis.Redis(host=redis_host, port=redis_port)

If you've used an older version of redis-py for other apps, you might have used the StrictClient class instead of Client. However, redis-py now recommends Client instead of StrictClient.

Using Redis commands to store and retrieve data in the cache

While the Memorystore Redis database supports most Redis commands, you only need to use a few commands to store and retrieve data from the cache. The following table suggests Redis commands you can use to cache data. To see how to call these commands from your app, view your client library's documentation.

Note that for Python 2 apps, while Memcache provides asynchronous alternatives for many of its commands, the redis-py client library doesn't always provide equivalent asynchronous methods. If you require all interactions with the cache to be asynchronous, other Redis client libraries for Python are available.

Task	Redis command
Create an entry in the data cache and set an expiration time for the entry	SETNX MSETNX
Retrieve data from the cache	GET MGET
Replace existing cache values	SET MSET
Increment or decrement numeric cache values	INCR INCRBY DECR DECRBY
Delete entries from the cache	DEL UNLINK
Support concurrent interactions with the cache (compare and set)	See details about Redis transactions. Note that the `redis-py` client library requires all transactions to occur in a pipeline.

Testing your updates

When you test your app locally, consider running a local instance of Redis to avoid interacting with production data (Memorystore doesn't provide an emulator). To install and run Redis locally, follow the directions in the Redis documentation. Note that it currently isn't possible to run Redis locally on Windows.

For more information about testing Python apps, see Using the local development server.

Deploying your app

Once your app is running in the local development server without errors:

Test the app on App Engine.
If the app runs without errors, use traffic splitting to slowly ramp up traffic for your updated app. Monitor the app closely for any database issues before routing more traffic to the updated app.

What's next

For a hands-on tutorial, see the Migrate from App Engine Memcache to Memorystore for Redis codelab.
See Memorystore for Redis documentation for more details.