How to get started with automatic password rotation on Google Cloud

Introduction

Password rotation is a broadly-accepted best practice, but implementing it can be a cumbersome and disruptive process. Automation can help ease that burden, and in this guide we offer some best practices to automate password rotation on Google Cloud. 

As an example, we share a reference architecture to automate the process of rotating passwords for a Cloud SQL instance on Google Cloud. This method can be extended to other tools and types of secrets.

Storing passwords in Google Cloud

While there are many solutions you can use to store secrets such as passwords in Google Cloud, we suggest using Secret Manager, our fully-managed product for securely storing secrets. Regardless of the tool you choose, stored passwords should be protected using additional measures. Here are some of the ways you can secure your secrets when using Secret Manager:

1. Limiting access: Secrets should be readable/writable only through the Service Accounts via IAM roles. The principle of least privilege should be followed while granting roles to the service accounts.

2. Encryption: Secret Manager encrypts secrets at rest using AES-256 by default. You can also use your own customer-managed encryption keys (CMEK) to encrypt your secrets at rest. For details, see enable customer-managed encryption keys for Secret Manager.

3. Password rotation: Passwords stored in Secret Manager should be rotated on a regular basis to reduce the risk of a security incident.

The why and how of password rotation

Regularly changing passwords mitigates risk in the event passwords are compromised. Forrester Research estimates that 80% of data breaches have a connection to compromised privileged credentials, such as passwords, tokens, keys, or certificates. 

We don’t recommend manually rotating passwords, since human handling of the passwords can introduce additional risk, such as misuse of the password. Manual rotation processes also introduce the risk that the rotation isn't actually performed due to human error. 

The more secure method is to automate password rotation as part of your workflow. The password could be for an application, a database, a third-party service, or a SaaS vendor.

Automatic password rotation

Typically, rotating a password requires these steps:

• Change the password in the underlying software or system (such as applications, databases, SaaS.)

• Update Secret Manager to store the new password.

• Restart the applications that use that password. This will make the application source the latest passwords.

Generic architecture for automatic password rotation

The following architecture represents a general design for a system in Google Cloud that can rotate passwords for any underlying software/system based on the best practices we’ve just outlined.

Here’s how the workflow should operate:

• A pipeline or a Cloud Scheduler sends a message to a pub/sub topic. The message contains the information about the password that is to be rotated. For example, this information may include a Secret ID in Secret Manager, or the database instance and username if it is a database password. 

• The message arriving at the pub/sub topic triggers a Cloud Run Function that reads the message and gathers information as supplied in the message.

• The function changes the password in the corresponding system. For example, if the message contained a database instance, database name and user, the function changes the password for that user in the given database.

• The function updates the password in the secret manager to reflect the new password. It knows what Secret ID to update since it was provided in the pub/sub message.

• The function publishes a message to a different pub/sub topic indicating that the password has been rotated. This topic can be subscribed by any application or system that may want to know in the event of password rotation, whether to restart themselves or perform any other task.

This guide shows an example deployment on how to automate rotating CloudSQL passwords on Google Cloud based on this architecture.

Take the next step

To learn more about Secret Manager, consult the documentation. To learn about other best practices for securing Google Cloud applications and resources, visit our Security Best Practices Center.