Improving the Google Cloud Storage backend for HashiCorp Vault

HashiCorp Vault is a powerful open source tool for secrets management, popular with many Google Cloud Platform (GCP) customers today. HashiCorp Vault provides "secret management as a service," acting as a static secret store for encrypted key-value pairs; a secret generation tool to dynamically generate on-the-fly credentials; and pass-through encryption service so that applications don’t need to roll their own encryption. Today, we're announcing exciting improvements to the existing Google Cloud Storage backend for HashiCorp Vault, including high availability.

As mentioned in our blog post announcing Google Cloud Spanner as a supported HashiCorp Vault storage backend, we strive to make Google Cloud an excellent platform on which to operationalize Vault for all users and use cases. Your feedback from the Cloud Spanner integration was overwhelmingly positive, but many of you are already leveraging the community-supported Cloud Storage backend and don’t want to migrate your existing data to a different storage system. GCP’s wealth of offerings let you choose the best storage options to meet your needs, and now you can choose from both Cloud Spanner and Cloud Storage for HashiCorp Vault storage backends.

The improved Cloud Storage HashiCorp Vault storage backend is completely backwards compatible with the existing solution, but includes a number of new features and benefits:

• High availability - In addition to Cloud Storage's built-in multi-region architecture, the improved HashiCorp Vault storage backend also supports running Vault in "high availability" mode. By default, HashiCorp Vault runs as a single tenant, relying on the storage backend to provide distributed locking and leader election. By leveraging object metadata for read-modify-write conditions in Cloud Storage, the improved storage backend allows for a highly available Vault cluster with just a single line of configuration. You can read more about HashiCorp Vault's High Availability model in the documentation.

• Support for default application credentials - Previously the Cloud Storage Vault storage backend required you to create a dedicated service account and credentials file. While you can still specify a credentials file, the storage backend now supports pulling default application credentials, such as those from your local gcloud installation or Application Default Credentials if you're running Vault on GCP.

• Enterprise-grade security - Cloud Storage follows the same security best practices as other Google products. Objects stored in Cloud Storage are encrypted by default, and it uses IAM to provide granular permission management on buckets and folders. Google’s infrastructure has many security differentiators, including secure boot using Google’s custom-designed security chip Titan, and Google’s private network backbone.

Getting started

Next, create a Cloud Storage bucket using the gsutil CLI tool (part of the gcloud CLI) to store the Vault data . You can also create the bucket using the web interface or API directly:

In this example, we created a bucket named "company-vault-data." Note that Cloud Storage bucket names must be globally unique.

Next, create a Vault configuration file configured to use Cloud Storage as the storage backend:

Start Vault with the configuration file. Note that this example uses Vault's built-in development mode, which does not represent best practices or a production installation, but it's the fastest way to try the improved Cloud Storage storage backend for HashiCorp Vault. For more details on a production-grade Vault installation, please read the Vault production hardening guide.

During this process, Vault authenticates and connects to Cloud Storage to populate and manage objects in the provided bucket. After a few seconds, you can view the objects in the web interface and see that data has been populated.

You can now create, read, update and delete secrets:

To learn more about the backend configuration options, read the HashiCorp Vault Cloud Storage backend documentation. To learn more about Cloud Storage, check out the documentation.

Toward a seamless Vault experience on GCP