Memcache is a distributed in-memory data store that is bundled into the Python 2 runtime. Many App Engine apps use Memcache as a cache for specific tasks instead of relying solely on persistent storage. Bundled services such as Memcache are not available in newer App Engine runtimes, including the Python 3 runtime.
If your Python 2 app explicitly uses Memcache, you need to migrate to another caching solution before you can run your app in a Python 3 environment. We recommend using Memorystore for Redis as your caching service. Note that unlike Memcache, Memorystore for Redis doesn't provide a free tier. See Memorystore Pricing for details.
Before getting started, make sure your app will stay within the Memorystore for Redis quotas.
If you only cache data to speed up Cloud NDB, follow the instructions in the Cloud NDB migration guide instead of these instructions.
Understanding Memorystore permissions
Every interaction with a Google Cloud service needs to be authorized. For example, to interact with a Redis database hosted by Memorystore, your app needs to supply the credentials of an account that is authorized to access Memorystore.
By default your app supplies the credentials of the App Engine default service account, which is authorized to access databases in the same project as your app.
If any of the following conditions are true, you need to use an alternative authentication technique that explicitly provides credentials:
Your app and the Memorystore database are in different Google Cloud projects.
You have changed the roles assigned to the default App Engine service account.
For information about alternative authentication techniques, see Setting up Authentication for Server to Server Production Applications.
Overview of the migration process
To migrate your Python app to use Memorystore instead of Memcache:
Set up Memorystore for Redis, which requires you to create a Redis instance on Memorystore and create a Serverless VPC Access that your app uses to communicate with the Redis instance.
Install a client library for Redis and use Redis commands to cache data.
Memorystore for Redis is compatible with any client library for Redis. This guide describes using the redis-py 3.0 client library to send Redis commands from your app.
Setting up Memorystore for Redis
To set up Memorystore for Redis:
Create a Redis instance in Memorystore.
When prompted to select a region for your Redis instance, select the same region in which your App Engine app is located.
Note the IP address and port number of the Redis instance you create. You will use this information when you create a Redis client in your code.
Connect your App Engine to a VPC network. Your app can only communicate with Memorystore through a VPC connector.
Be sure to add the VPC connection information to your
app.yaml
file as described in Configuring your app use the connector.
Installing dependencies
To make the redis-py client library available to your app when it runs in App Engine:Add the following line to your app's
requirements.txt
file:redis == 3.3.11
We recommend you use the 3.3.11 version since it supports Python 2.7 apps.
If you still run your app in the Python 2 runtime:
Use pip (version 6 or later) with the
-t <directory>
flag to install the libraries into a local folder namedlib
. For example:pip install -t lib
-r requirements.txt Make sure the app's
appengine_config.py
file specifies thelib
directory.
App Engine will upload all of the libraries in the directory you specified in the
appengine_config.py
file. For details, see Using Python 2 libraries.
The App Engine Python 3 runtime will automatically upload all libraries
your app's requirements.txt
file when you
deploy the app.
For local development, we recommend that you install dependencies in a virtual environment such as virtualenv for Python 2 or venv for Python 3.
Creating a Redis client
To interact with a Redis database, your code needs to create a Redis client to manage the connection to your Redis database. The following sections describe creating a Redis client using the redis-py client library.
Specifying environment variables
The redis-py client library uses two environment variables to assemble the URL for your Redis database:
- A variable to identify the IP address of the Redis database you created in Memorystore.
- A variable to identify the port number of the Redis database you created in Memorystore.
We recommend you define these variables in your app's app.yaml
file instead of
defining them directly in your code. This makes it easier to run your app in
different environments, such as a local environment and App Engine.
app.yaml
file:
env_variables:
REDISHOST: '10.112.12.112'
REDISPORT: '6379'
Importing redis-py and creating the client
After you define theREDISHOST
and REDISPORT
environment variables,
use the following lines to import the redis-py library and
create a client:
import redis
redis_host = os.environ.get('REDISHOST', 'localhost')
redis_port = int(os.environ.get('REDISPORT', 6379))
redis_client = redis.Redis(host=redis_host, port=redis_port)
If you've used an older version of redis-py for other apps, you might have used
the StrictClient
class instead of Client
. However, redis-py now
recommends Client
instead of StrictClient
.
Using Redis commands to store and retrieve data in the cache
While the Memorystore Redis database supports most Redis commands, you only need to use a few commands to store and retrieve data from the cache. The following table suggests Redis commands you can use to cache data. To see how to call these commands from your app, view your client library's documentation.
Note that while Memcache provides asynchronous alternatives for many of its commands, the redis-py client library doesn't always provide equivalent asynchronous methods. If you require all interactions with the cache to be asynchronous, other Redis client libraries for Python are available.Task | Redis command |
---|---|
Create an entry in the data cache and set an expiration time for the entry |
SETNX MSETNX |
Retrieve data from the cache | GET MGET |
Replace existing cache values | SET MSET |
Increment or decrement numeric cache values | INCR INCRBY DECR DECRBY |
Delete entries from the cache | DEL UNLINK |
Support concurrent interactions with the cache (compare and set) | See details about Redis transactions. Note that the redis-py client library requires all transactions to occur in a pipeline. |
Testing your updates
When you test your app locally, consider running a local instance of Redis to avoid interacting with production data (Memorystore doesn't provide an emulator). To install and run Redis locally, follow the directions in the Redis documentation. Note that it currently isn't possible to run Redis locally on Windows.
For more information about testing Python 2 apps, see Using the local development server.Deploying your app
Once your app is running in the local development server without errors:
If the app runs without errors, use traffic splitting to slowly ramp up traffic for your updated app. Monitor the app closely for any database issues before routing more traffic to the updated app.