Client-side encryption with Tink and Cloud KMS

This topic describes how to encrypt data locally and upload it to Cloud Storage with Google's open source cryptographic SDK, Tink, and Cloud Key Management Service (KMS).

Overview

Client-side encryption is any encryption performed prior to sending your data to the cloud. When using client-side encryption, you're responsible for creating and managing your own encryption keys and using your own tools to encrypt data prior to sending it to the cloud.

Data that you encrypt on the client side is sent to the cloud in an encrypted state. To use client-side encryption you have the following options:

  • Using an encryption key stored in Cloud Key Management Service.
  • Using an encryption key stored locally in your application.

In this topic, you will create an encryption key in Cloud Key Management Service and implement envelope encryption using Tink, Google's open-source cryptographic SDK.

Before you begin

  1. Create a symmetric Cloud Key Management Service encryption key for encryption. Take note of the key's URI which is used later.
  2. Install Tink for use with Cloud Key Management Service.
  3. Create a bucket in Cloud Storage to upload your encrypted data.

Envelope encryption with Tink

In envelope encryption, the Cloud KMS key acts as a key encryption key (KEK). That is, it is used to encrypt data encryption keys (DEK) which in turn are used to encrypt actual data.

After creating a KEK in Cloud KMS, to encrypt each message you need to:

  1. Generate a data encryption key (DEK) locally.
  2. Use the DEK locally to encrypt the message.
  3. Use Cloud KMS to encrypt (wrap) the DEK with the KEK.
  4. Store the encrypted data and the wrapped DEK.

Instead of implementing the above steps for envelope encryption from scratch, you'll use Tink.

To encrypt data with Tink's envelope encryption, you provide Tink with a key URI pointing to your KEK in Cloud KMS, and credentials that allow Tink to use the KEK. Tink generates the DEK, encrypts the data, wraps the DEK and returns a single ciphertext with the encrypted data and wrapped DEK.

Tink supports envelope encryption in Python, Java, C++, and Go using the Authenticated Encryption with Associated Data (AEAD) primitive.

Connect Tink and Cloud KMS

To encrypt the DEKs generated by Tink with your KEK in Cloud KMS, you need to get your KEK's URI. In Cloud KMS, the KEK URI has the format:

gcp-kms://projects/<PROJECT>/locations/<LOCATION>/keyRings/ <KEY RING>/cryptoKeys/<KEY NAME>/cryptoKeyVersions/<VERSION>

See Getting a Cloud KMS resource ID for details on how to get the path to your key.

To provide access to the key in Cloud KMS, you need to grant the service account that your application uses with the cloudkms.cryptoKeyEncrypterDecrypter role. For more information on roles, see Permission and roles. In gcloud, use the following command:

gcloud kms keys add-iam-policy-binding key \
    --keyring key-ring \
    --location location \
    --member serviceAccount:service-account-name@example.domain.com \
    --role roles/cloudkms.cryptoKeyEncrypterDecrypter

Initialize Tink and encrypt data

Tink uses primitives, cryptographic building blocks that manage the details of their underlying algorithms so users can perform tasks safely. Each primitive offers an API that handles a specific task. Here, we're using AEAD, so we'll use the AEAD Tink primitive.

Python

Python

To learn how to install and use the client library for Cloud KMS, see the Cloud KMS Client Libraries.

"""A command-line utility for performing file encryption using GCS.

It is inteded for use with small files, utilizes envelope encryption and
facilitates ciphertexts stored in GCS.
"""

from __future__ import absolute_import
from __future__ import division
# Placeholder for import for type annotations
from __future__ import print_function

from absl import app
from absl import flags
from absl import logging
from google.cloud import storage

import tink
from tink import aead
from tink.integration import gcpkms


FLAGS = flags.FLAGS

flags.DEFINE_enum('mode', None, ['encrypt', 'decrypt'],
                  'The operation to perform.')
flags.DEFINE_string('kek_uri', None,
                    'The Cloud KMS URI of the key encryption key.')
flags.DEFINE_string('gcp_credential_path', None,
                    'Path to the GCP credentials JSON file.')
flags.DEFINE_string('gcp_project_id', None,
                    'The ID of the GCP project hosting the GCS blobs.')
flags.DEFINE_string('local_path', None, 'Path to the local file.')
flags.DEFINE_string('gcs_blob_path', None, 'Path to the GCS blob.')


_GCS_PATH_PREFIX = 'gs://'


def main(argv):
  del argv  # Unused.

  # Initialise Tink
  try:
    aead.register()
  except tink.TinkError as e:
    logging.exception('Error initialising Tink: %s', e)
    return 1

  # Read the GCP credentials and setup client
  try:
    gcpkms.GcpKmsClient.register_client(
        FLAGS.kek_uri, FLAGS.gcp_credential_path)
  except tink.TinkError as e:
    logging.exception('Error initializing GCP client: %s', e)
    return 1

  # Create envelope AEAD primitive using AES256 GCM for encrypting the data
  try:
    template = aead.aead_key_templates.create_kms_envelope_aead_key_template(
        kek_uri=FLAGS.kek_uri,
        dek_template=aead.aead_key_templates.AES256_GCM)
    handle = tink.KeysetHandle.generate_new(template)
    env_aead = handle.primitive(aead.Aead)
  except tink.TinkError as e:
    logging.exception('Error creating primitive: %s', e)
    return 1

  storage_client = storage.Client.from_service_account_json(
      FLAGS.gcp_credential_path)

  try:
    bucket_name, object_name = _get_bucket_and_object(FLAGS.gcs_blob_path)
  except ValueError as e:
    logging.exception('Error parsing GCS blob path: %s', e)
    return 1
  bucket = storage_client.bucket(bucket_name)
  blob = bucket.blob(object_name)
  associated_data = FLAGS.gcs_blob_path.encode('utf-8')

  if FLAGS.mode == 'encrypt':
    with open(FLAGS.local_path, 'rb') as input_file:
      output_data = env_aead.encrypt(input_file.read(), associated_data)
    blob.upload_from_string(output_data)

  elif FLAGS.mode == 'decrypt':
    ciphertext = blob.download_as_string()
    with open(FLAGS.local_path, 'wb') as output_file:
      output_file.write(env_aead.decrypt(ciphertext, associated_data))

  else:
    logging.error(
        'Error mode not supported. Please choose "encrypt" or "decrypt".')
    return 1


def _get_bucket_and_object(gcs_blob_path):
  """Extract bucket and object name from a GCS blob path.

  Args:
    gcs_blob_path: path to a GCS blob

  Returns:
    The bucket and object name of the GCS blob

  Raises:
    ValueError: If gcs_blob_path parsing fails.
  """
  if not gcs_blob_path.startswith(_GCS_PATH_PREFIX):
    raise ValueError(
        f'GCS blob paths must start with gs://, got {gcs_blob_path}')
  path = gcs_blob_path[len(_GCS_PATH_PREFIX):]
  parts = path.split('/', 1)
  if len(parts) < 2:
    raise ValueError(
        'GCS blob paths must be in format gs://bucket-name/object-name, '
        f'got {gcs_blob_path}')
  return parts[0], parts[1]

if __name__ == '__main__':
  flags.mark_flags_as_required([
      'mode', 'kek_uri', 'gcp_credential_path', 'gcp_project_id', 'local_path',
      'gcs_blob_path'])
  app.run(main)

Java

Java

To learn how to install and use the client library for Cloud KMS, see the Cloud KMS Client Libraries.

package gcs;

import static java.nio.charset.StandardCharsets.UTF_8;

import com.google.auth.oauth2.GoogleCredentials;
import com.google.cloud.storage.BlobId;
import com.google.cloud.storage.BlobInfo;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import com.google.crypto.tink.Aead;
import com.google.crypto.tink.KeyTemplates;
import com.google.crypto.tink.KeysetHandle;
import com.google.crypto.tink.aead.AeadConfig;
import com.google.crypto.tink.aead.KmsEnvelopeAeadKeyManager;
import com.google.crypto.tink.integration.gcpkms.GcpKmsClient;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.security.GeneralSecurityException;
import java.util.Arrays;
import java.util.Optional;

/**
 * A command-line utility for encrypting small files with envelope encryption and uploading the
 * results to GCS.
 *
 * <p>The CLI takes the following required arguments:
 *
 * <ul>
 *   <li>mode: "encrypt" or "decrypt" to indicate if you want to encrypt or decrypt.
 *   <li>kek-uri: The URI for the Cloud KMS key to be used for envelope encryption.
 *   <li>gcp-credential-file: Name of the file with the GCP credentials (in JSON format) that can
 *       access the Cloud KMS key and the GCS input/output blobs.
 *   <li>gcp-project-id: The ID of the GCP project hosting the GCS blobs that you want to encrypt or
 *       decrypt.
 * </ul>
 *
 * <p>When mode is "encrypt", it takes the following additional arguments:
 *
 * <ul>
 *   <li>local-input-file: Read the plaintext from this local file.
 *   <li>gcs-output-blob: Write the encryption result to this blob in GCS. The encryption result is
 *       bound to the location of this blob. That is, if you rename or move it to a different
 *       bucket, decryption will fail.
 * </ul>
 *
 * <p>When mode is "decrypt", it takes the following additional arguments:
 *
 * <ul>
 *   <li>gcs-input-blob: Read the ciphertext from this blob in GCS.
 *   <li>local-output-file: Write the decryption result to this local file.
 */
public final class GcsEnvelopeAeadExample {
  private static final String MODE_ENCRYPT = "encrypt";
  private static final String MODE_DECRYPT = "decrypt";
  private static final String GCS_PATH_PREFIX = "gs://";

  public static void main(String[] args) throws Exception {
    if (args.length != 6) {
      System.err.printf("Expected 6 parameters, got %d\n", args.length);
      System.err.println(
          "Usage: java GcsEnvelopeAeadExample encrypt/decrypt kek-uri gcp-credential-file"
              + " gcp-project-id input-file output-file");
      System.exit(1);
    }
    String mode = args[0];
    String kekUri = args[1];
    String gcpCredentialFilename = args[2];
    String gcpProjectId = args[3];

    // Initialise Tink: register all AEAD key types with the Tink runtime
    AeadConfig.register();

    // Read the GCP credentials and set up client
    try {
      GcpKmsClient.register(Optional.of(kekUri), Optional.of(gcpCredentialFilename));
    } catch (GeneralSecurityException ex) {
      System.err.println("Error initializing GCP client: " + ex);
      System.exit(1);
    }

    // Create envelope AEAD primitive using AES256 GCM for encrypting the data
    Aead aead = null;
    try {
      KeysetHandle handle =
          KeysetHandle.generateNew(
              KmsEnvelopeAeadKeyManager.createKeyTemplate(kekUri, KeyTemplates.get("AES256_GCM")));
      aead = handle.getPrimitive(Aead.class);
    } catch (GeneralSecurityException ex) {
      System.err.println("Error creating primitive: %s " + ex);
      System.exit(1);
    }

    GoogleCredentials credentials =
        GoogleCredentials.fromStream(new FileInputStream(gcpCredentialFilename))
            .createScoped(Arrays.asList("https://www.googleapis.com/auth/cloud-platform"));
    Storage storage =
        StorageOptions.newBuilder()
            .setProjectId(gcpProjectId)
            .setCredentials(credentials)
            .build()
            .getService();

    // Use the primitive to encrypt/decrypt files.
    if (MODE_ENCRYPT.equals(mode)) {
      // Encrypt the local file
      byte[] input = Files.readAllBytes(Paths.get(args[4]));
      String gcsBlobPath = args[5];
      // This will bind the encryption to the location of the GCS blob. That if, if you rename or
      // move the blob to a different bucket, decryption will fail.
      // See https://developers.google.com/tink/AEAD#associated_data.
      byte[] associatedData = gcsBlobPath.getBytes(UTF_8);
      byte[] ciphertext = aead.encrypt(input, associatedData);

      // Upload to GCS
      String bucketName = getBucketName(gcsBlobPath);
      String objectName = getObjectName(gcsBlobPath);
      BlobId blobId = BlobId.of(bucketName, objectName);
      BlobInfo blobInfo = BlobInfo.newBuilder(blobId).build();
      storage.create(blobInfo, ciphertext);
    } else if (MODE_DECRYPT.equals(mode)) {
      // Download the GCS blob
      String gcsBlobPath = args[4];
      String bucketName = getBucketName(gcsBlobPath);
      String objectName = getObjectName(gcsBlobPath);
      byte[] input = storage.readAllBytes(bucketName, objectName);

      // Decrypt to a local file
      byte[] associatedData = gcsBlobPath.getBytes(UTF_8);
      byte[] plaintext = aead.decrypt(input, associatedData);
      File outputFile = new File(args[5]);
      try (FileOutputStream stream = new FileOutputStream(outputFile)) {
        stream.write(plaintext);
      }
    } else {
      System.err.println("The first argument must be either encrypt or decrypt, got: " + mode);
      System.exit(1);
    }

    System.exit(0);
  }

  private static String getBucketName(String gcsBlobPath) {
    if (!gcsBlobPath.startsWith(GCS_PATH_PREFIX)) {
      throw new IllegalArgumentException(
          "GCS blob paths must start with gs://, got " + gcsBlobPath);
    }

    String bucketAndObjectName = gcsBlobPath.substring(GCS_PATH_PREFIX.length());
    int firstSlash = bucketAndObjectName.indexOf("/");
    if (firstSlash == -1) {
      throw new IllegalArgumentException(
          "GCS blob paths must have format gs://my-bucket-name/my-object-name, got " + gcsBlobPath);
    }
    return bucketAndObjectName.substring(0, firstSlash);
  }

  private static String getObjectName(String gcsBlobPath) {
    if (!gcsBlobPath.startsWith(GCS_PATH_PREFIX)) {
      throw new IllegalArgumentException(
          "GCS blob paths must start with gs://, got " + gcsBlobPath);
    }

    String bucketAndObjectName = gcsBlobPath.substring(GCS_PATH_PREFIX.length());
    int firstSlash = bucketAndObjectName.indexOf("/");
    if (firstSlash == -1) {
      throw new IllegalArgumentException(
          "GCS blob paths must have format gs://my-bucket-name/my-object-name, got " + gcsBlobPath);
    }
    return bucketAndObjectName.substring(firstSlash + 1);
  }

  private GcsEnvelopeAeadExample() {}
}

See How Tink works for more information on the supported primitives and their interfaces.

What's next?