Tutorial: Application-level encryption on Memorystore for Redis

This tutorial shows you how to create an application that communicates with the Cloud Key Management Service (Cloud KMS) API in order to encrypt content stored in a Memorystore for Redis store on Google Cloud. To learn more about the concepts used in this tutorial, see the document associated with this tutorial, Application-level encryption: Memorystore for Redis.

This tutorial is intended for application developers and security professionals who are familiar with Google Cloud, Linux, key management services, Redis, Git, Maven, and Java.

The tutorial is structured incrementally—similarly to a test-driven development methodology. First, you run a test. Then, you introduce a change, causing the test to fail. Finally, you take steps to remove the test failure.

Objectives

• Create and test a pair of Memorystore for Redis instances.
• Convert data and load it into Redis.
• Create your own key management system (KMS).
• Learn about the encryption key management process.

Costs

This tutorial uses the following billable components of Google Cloud:

• Memorystore for Redis
• Compute Engine

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Before you begin

1. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to the project selector page

2. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

3. Enable the Compute Engine, Cloud KMS, and Memorystore for Redis APIs.

   Enable the APIs

4. In the Cloud Console, activate Cloud Shell.

   Activate Cloud Shell

Creating a project VM instance

1. In Cloud Shell, set up your gcloud command-line tool configuration:

   gcloud config set project PROJECT_NAME
   gcloud config set compute/zone us-central1-f
   gcloud config set compute/region us-central1
   

   Replace PROJECT_NAME with the name of your Google Cloud project for this tutorial.

2. Create a virtual machine (VM) instance:

   gcloud compute instances create ale-instance \
       --machine-type=n1-standard-1 \
       --image-family=debian-10 \
       --image-project=debian-cloud \
       --boot-disk-size=200GB
   

3. In the user console on the VM instances page, click SSH to log in to your VM instance.

   

   A new window opens, which this tutorial calls the project window (versus the Cloud Shell window). To complete some of the tutorial steps, you need to switch between the project window and the Cloud Shell window.

Installing Git and Maven

• In the project window, install Git and Maven:

  sudo apt-get install git wget maven
  

  You might get the following warning:

  E: Could not get lock /var/lib/dpkg/lock
  

  This error can occur if the VM startup scripts are still running. If you get this error, wait one or two minutes, and then re-run the previous command.

Installing the Java Development Kit (JDK)

1. In the project window, change the directory:

   cd /tmp
   

2. Download the Java installation package:

   wget https://download.java.net/java/GA/jdk14.0.1/664493ef4a6946b186ff29eb326336a2/7/GPL/openjdk-14.0.1_linux-x64_bin.tar.gz
   

   The output is similar to the following:

   --2020-05-17 19:21:39--  https://download.java.net/java/GA/jdk14.0.1/664493ef4a6946b186ff29eb326336a2/7/GPL/openjdk-14.0.1_linux-x64_bin.tar.gz
   Resolving download.java.net (download.java.net)... 23.213.168.108
   Connecting to download.java.net (download.java.net)|23.213.168.108|:443... connected.
   HTTP request sent, awaiting response... 200 OK
   Length: 198665889 (189M) [application/x-gzip]
   Saving to: 'openjdk-14.0.1_linux-x64_bin.tar.gz'
   oenjdk-14.0.1_linux-x64_  80%[======================>      ] 151.72M  4.86MB/s    eta 8s
   

3. Display the contents of the Java Virtual Machine (JVM) directory:

   ls /usr/lib/jvm/
   

   You might get output similar to the following:

   ls: /usr/lib/jvm/: No such file or directory
   

   This output indicates that the folder doesn't exist. If that's the case, create the folder:

   sudo mkdir /usr/lib/jvm
   

4. Install the JDK:

   cd /usr/lib/jvm
   sudo tar xzf /tmp/openjdk-14.0.1_linux-x64_bin.tar.gz
   export JAVA_HOME=/usr/lib/jvm/jdk-14.0.1/
   

Cloning the tinkCryptoHelper repository

In the following steps, you clone the code repository for tinkCryptoHelper, the Java application that's used in this tutorial.

1. In the project window, go to your home directory:

   cd ~
   

2. Clone the tinkCryptoHelper code base:

   git clone https://github.com/google/tinkCryptoHelper.git
   cd tinkCryptoHelper
   

Creating a build of the tinkCryptoHelper application

In the following steps, you use the mvn tool to perform your first build of tinkCryptoHelper from the Java source code to an executable Java package.

1. In the project window, build the package:

   mvn package
   

   The output ends with the following:

   [INFO] --------------------------------------------------------------------
   [INFO] BUILD SUCCESS
   [INFO] --------------------------------------------------------------------
   [INFO] Total time: 13.880 s
   [INFO] Finished at: 2020-05-17T19:29:13+00:00
   [INFO] Final Memory: 19M/116M
   [INFO] --------------------------------------------------------------------
   

Testing the system

In the following sections, you run tests to evaluate how the system behaves after you make configuration changes. There are 14 integration tests in the code in the repository. Each tests different behavior to ensure that the system is working properly. Importantly, the tests are configuration dependent—for example, if you don't have a Redis database attached to the system, the tests don't try to test Redis connectivity.

Basic functionality tests

After you create a build of tinkCryptoHelper, you're ready to test its basic functionality. Here you don't have a Redis database online, so the connectivity tests don't fail.

1. In the project window, run a test:

   mvn test
   

   The output is similar to the following:

   [INFO] -------------------------------------------------------
   [INFO]  T E S T S
   [INFO] -------------------------------------------------------
   [INFO] Running com.google.samples.kms.ale.AppTest
   AES256_GCM encryption cipherlength: 64
   Envelope encryption cipherlength: 224
   [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.057 s - in com.google.samples.kms.ale.AppTest
   [INFO]
   [INFO] Results:
   [INFO]
   [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0
   [INFO]
   [INFO] --------------------------------------------------------------------
   [INFO] BUILD SUCCESS
   [INFO] --------------------------------------------------------------------
   [INFO] Total time: 7.087 s
   [INFO] Finished at: 2020-05-17T19:34:21+00:00
   [INFO] Final Memory: 12M/56M
   

   The key result is Tests run: 14, Failures: 0, Errors: 0, Skipped: 0. This output indicates that all the tests ran successfully.

Test the Redis connection without instances

The next step in full-integration testing requires the application (tinkCryptoHelper) to communicate with Redis. The application gets the Redis connection-string information from a prefs.xml file. As part of the first test that you ran in the preceding section, the tinkCryptoHelper application created several prefs.xml files at various locations, following the class structure of the Java source code.

As an exercise, you set redisIsOnline to true in the prefs.xml file. The Redis instance isn't online because you still need to create it. This test causes the connectivity tests to fail.

1. In the project window, list the XML files in the source code structure to ensure that you find and edit the correct prefs.xml file:

   find ~/.java -name '*.xml'
   

   The output is similar to the following:

   /home/USERNAME/.java/.userPrefs/com/google/samples/kms/ale/prefs.xml
   /home/USERNAME/.java/.userPrefs/com/google/samples/kms/prefs.xml
   /home/USERNAME/.java/.userPrefs/com/google/samples/prefs.xml
   /home/USERNAME/.java/.userPrefs/com/google/prefs.xml
   /home/USERNAME/.java/.userPrefs/com/prefs.xml
   

   In this output, USERNAME is the username that you logged in with.

2. To prepare to test Redis with Maven, open the following file with nano (or another UNIX editor such as vi):

   nano ~/.java/.userPrefs/com/google/samples/kms/ale/prefs.xml
   

   The content of this configuration file is similar to the following:

   <?xml version="1.0" encoding="UTF-8" standalone="no"?>
   <!DOCTYPE map SYSTEM "http://java.sun.com/dtd/preferences.dtd">
   <map MAP_XML_VERSION="1.0">
     <entry key="com.google.samples.kms.ale.AppTest" value="true"/>
   </map>
   

3. In the map section of the configuration file, add a second entry:

   <entry key="redisIsOnline" value="true"/>
   

   The map section now looks similar to the following:

   <map MAP_XML_VERSION="1.0">
     <entry key="com.google.samples.kms.ale.AppTest" value="true"/>
     <entry key="redisIsOnline" value="true"/>
   </map>
   

4. Save and exit the file.

5. Run a test:

   mvn test
   

   Because Redis isn't online, the output shows errors similar to the following:

   [ERROR]   AppTest.testRedis3_2:86 » JedisConnection Failed connecting to host 127.0.0.2:...
   [ERROR]   AppTest.testRedis4_0:79 » JedisConnection Failed connecting to host 127.0.0.1:...
   [ERROR]   AppTest.testRedisRoundtripRedis3_2:93->testRedisRoundtripRedisSeries:105->testRedisRoundtripClear:149 » JedisConnection
   [ERROR]   AppTest.testRedisRoundtripRedis4_0:99->testRedisRoundtripRedisSeries:105->testRedisRoundtripClear:149 » JedisConnection
   [INFO]
   

   You caused a failure to occur. In the next section, you fix the failure.

Creating Redis instances

In the previous step, the test failed because there was no Redis instance to connect to. To correct this, you need to create Redis instances. To perform tests with the tinkCryptoHelper application, you need two Redis instances of different release versions. The tests run on both instances.

Create the Redis instances

1. In Cloud Shell, create two Redis instances of different versions:

   gcloud redis instances create redis32 --redis-version redis_3_2 --region us-central1
   gcloud redis instances create redis40 --redis-version redis_4_0 --region us-central1
   

   It takes a few minutes to create the Redis 3.2 and Redis 4.0 instances.

2. View information about the instances:

   gcloud redis instances list --region us-central1
   

   The output is similar to the following:

   INSTANCE_NAME  VERSION    REGION       TIER   SIZE_GB  HOST            PORT  NETWORK  RESERVED_IP        STATUS  CREATE_TIME
   redis32        REDIS_3_2  us-central1  BASIC  1        10.126.141.179  6379  default  10.126.141.176/29  READY   2020-05-17T19:47:15
   redis40        REDIS_4_0  us-central1  BASIC  1        10.55.16.163    6379  default  10.55.16.160/29    READY   2020-05-17T19:50:44
   

3. Copy the IP addresses from the output. You need the host IP addresses in the next steps.

Configure the IP addresses for the Redis instances

The next step is to tell the tinkCryptoHelper application where to find the Redis instances.

1. In the project window, open the prefs.xml file that you opened earlier:

   nano ~/.java/.userPrefs/com/google/samples/kms/ale/prefs.xml
   

2. In the map section of the configuration file, add two lines after the entry that you added earlier:

   <entry key="redisHost3_2" value="IP_ADDRESS3.2" />
   <entry key="redisHost4_0" value="IP_ADDRESS4.0" />
   

   Replace the following:

   • IP_ADDRESS3.2: the value of the redis32 instance that you copied earlier
   • IP_ADDRESS4.0: the value of the redis40 instance that you copied earlier

3. Save and exit the file.

4. Run a test:

   mvn test
   

   The output is similar to the following:

   Set of 5000 cleartext values took on average 2µs on host 10.126.141.179
   Get of 5000 cleartext values took on average 1µs on host 10.126.141.179
   Set of 5000 encrypted values took on average 40µs on host 10.126.141.179
   ...
   Envelope encryption cipherlength: 224
   [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.312 s - in com.google.samples.kms.ale.AppTest
   [INFO]
   [INFO] Results:
   [INFO]
   [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0
   [INFO]
   [INFO] ------------------------------------------------------------------------
   [INFO] BUILD SUCCESS
   [INFO] ------------------------------------------------------------------------
   [INFO] Total time: 12.470 s
   [INFO] Finished at: 2020-05-17T20:55:46+00:00
   [INFO] Final Memory: 12M/56M
   [INFO] ------------------------------------------------------------------------
   

You successfully completed the integration test with Redis.

Convert data and load it into Redis

In the previous section, the tests had several side effects, one of which was to insert data into the Redis instances. This process was necessary to complete the integration tests. If you want to test the system with your own data, the following steps demonstrate how to use tinkCryptoHelper to encrypt the data, then update the Redis database with your own data. A sample data.csv file is provided in the Git repository, but you can create your own.

1. In the project window, create a JAR file:

   mvn assembly:single
   

2. Convert your data.csv file into a file that's compatible with the Redis import command:

   $JAVA_HOME/bin/java -jar target/cryptoHelper-1.0-jar-with-dependencies.jar data.csv redis_import.txt
   

   The redis_import.txt file contains a series of commands that you can use to insert your data into Redis.

3. Install the Redis tools, which include a command-line interface (CLI):

   sudo apt-get install redis-tools
   

4. If you don't want to keep the existing data in the Redis database, you can do the following:

   1. Using the Redis CLI, connect to Redis:

      redis-cli -h IP_ADDRESS
      

      Replace IP_ADDRESS with the one of the IP addresses that you copied earlier.

   2. Delete the data in the database:

      FLUSHALL
      exit
      

5. Load the data:

   cat redis_import.txt | redis-cli -h IPADDRESS --pipe
   

6. To test if the data is there, connect to Redis again and get the key:

   1. Connect to Redis:

      redis-cli -h IP_ADDRESS
      

   2. Get the key:

      get 1
      

      The output is the following:

      Hello World
      

      If you used your own data, the key and value are likely different.

Creating your own KMS

The KMS encryption key in this tutorial belongs to a Google-owned Cloud project named tink-test-infrastructure—not to the project that you created. This sample project has a KMS that's used to make the encryption key available for public use.

To create your own KMS instance and keys, you create a service account and associate the account with the Cloud KMS CryptoKey Encrypter/Decrypter IAM role (roles/cloudkms.cryptoKeyEncrypterDecrypter). With this role, the service account can create a KMS key ring and add keys.

Set up a service account

1. In Cloud Shell, create a new service account:

   gcloud iam service-accounts create tink-509 \
       --description="Account for KMS" \
       --display-name="tinkAccount"
   

2. In the Cloud Console, go to the IAM & Admin page.

   Go to the IAM & Admin page

3. Click Add, and then add the service account tink-509 to the Cloud KMS CryptoKey Encrypter/Decrypter role.

   

4. Save the change and exit the Cloud Console.

5. In the project window, save a JSON credentials key file for the new service account:

   1. Back up the old credentials key file:

      mv kmsServiceAccountCredentials.json kmsServiceAccountCredentials.json.old
      

   2. Ensure that you're logged in. Run the command, and then follow the OAuth instructions:

      gcloud config set project PROJECT_ID
      gcloud auth login
      

      Replace PROJECT_ID with your Cloud project ID.

   3. Create the new credentials key file:

      gcloud iam service-accounts keys create kmsServiceAccountCredentials.json \
          --iam-account [e-mail]
      

      The output is similar to the following:

      created key [c09dfea3892d6c309333f1998caf35845dc50608] of type [json] as [kmsServiceAccountCredentials.json] for [[e-mail]]
      

Create a Cloud KMS key ring and key

1. In Cloud Shell, create a Cloud KMS key ring:

   gcloud kms keyrings create "unit-and-integration-testing" --location "global"
   gcloud kms keys create aead-key --purpose=encryption --location "global" \
       --keyring unit-and-integration-testing
   

2. Add the roles/cloudkms.cryptoKeyEncrypterDecrypter IAM role to an IAM policy:

   gcloud kms keys add-iam-policy-binding \
       aead-key --location global --keyring unit-and-integration-testing \
       --member serviceAccount:[e-mail] \
       --role roles/cloudkms.cryptoKeyEncrypterDecrypter
   

   This role lets your service account access the keys.

Clear out previous Redis data

In the previous tests, the Redis database was populated with sample data, but the encryption was based on an older key. You need to clear the data and delete the older key.

1. In Cloud Shell, list all your Redis instances:

   gcloud redis instances list --region us-central1
   

   Note the IP address of the instances that you created.

2. Connect to the two Redis instances:

   redis-cli -h IP_ADDRESS
   

   Run the preceding command twice, replacing IP_ADDRESS with the IP address for each instance.

3. Delete the data in your database:

   FLUSHALL
   exit
   

   When you run the final test later in this tutorial, the data in the Redis tables is encrypted with the new keys.

Specify where tinkCryptoHelper finds the new KMS

1. In the project window, open the KMS pref.xml file using a UNIX editor:

   nano ~/.java/.userPrefs/com/google/samples/kms/prefs.xml
   

2. Add the following line as the second entry:

   <entry key="keyResourceIdUri" value="gcp-kms://projects/PROJECT_ID/locations/global/keyRings/unit-and-integration-testing/cryptoKeys/aead-key"/>
   

   Replace PROJECT_ID with your Cloud project ID.

Run a final test

• In the project window, run tests:

  mvn test
  

  The output is similar to the following:

  ...
  Envelope encryption cipherlength: 224
  [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.312 s - in com.google.samples.kms.ale.AppTest
  ...
  

  You successfully created the KMS and keys and then used them to test tinkCryptoHelper.

Cleaning up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, you can delete the project.

Delete the project

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple tutorials and quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Cloud Console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

• Try out other Google Cloud features for yourself. Have a look at our tutorials.