Secret Manager best practices

This guide introduces some best practices when using Secret Manager. The guide is not an exhaustive list of recommendations. We recommend reviewing the platform overview in order to understand the overall Google Cloud landscape and the Secret Manager overview before you read this guide.

Access control

Access to the Secret Manager API is protected by IAM. Follow the principle of least privilege when granting permissions to secrets.

Limit organization ownership to a secured super admin account.
Segment applications and environments (staging/production) into separate projects as described in Decide a resource hierarchy for your Google Cloud landing zone. This can help isolate environments with project level IAM binding and ensures that quotas are enforced independently.
Choose a curated role with the minimal necessary permissions, or create a custom role if necessary.
When secrets for many services are in a single project, use secret level IAM bindings, or IAM Conditions to limit access to the necessary subset of secrets.
IAM Recommender can further assist in identifying over privileged IAM bindings.

Credentials are required to authenticate to the Secret Manager API. The client libraries all use a similar strategy to look for credentials referred to as "Application Default Credentials".

When developing locally use gcloud auth application-default login. This creates a file with credentials that are automatically picked up by client libraries.
On Compute Engine and other Google Cloud compute platforms like Cloud Run functions, the client libraries obtain credentials through the instance metadata server.
On GKE, workload identity provides credentials through a metadata service as well.
On other platforms like Amazon Web Services or Microsoft Azure, consider setting up workload identity federation which uses existing identity mechanisms to authenticate to Google Cloud APIs.

All of these methods are preferred to exporting a service account credential because they do not require securely storing and accessing an additional secret outside of the Secret Manager API.

Coding Practices

Passing secrets to your application through the filesystem or through the environment variables is common, but should be avoided when possible due to the following reasons:

When a secret is accessible on the filesystem, application vulnerabilities like directory traversal attacks can become higher severity as the attacker may gain the ability to read the secret material.
When a secret is consumed through environment variables, misconfigurations such as enabling debug endpoints or including dependencies that log process environment details may leak secrets.
When syncing secret material to another data store (like Kubernetes Secrets), consider the access controls of that data store, for example:
- Does the datastore expand the access of the secret?
- Is it possible to audit access of the secret?
- Does the datastore comply with your data-at-rest encryption and regionalization requirements?

We recommend using the Secret Manager API directly (using one of the provided client libraries, or by following the REST or GRPC documentation).

However, for some product integrations such as serverless integrations, you may pass secrets through the filesystem or through the environment variables. For information, see Use Secret Manager with other products.

Administration

Choose the automatic replication policy when creating secrets unless your workload has specific location requirements (enforceable using the constraints/gcp.resourceLocations constraint).

Reference secrets by their version number rather than using the latest alias. Deploy updates to version numbers using your existing release processes. Typically this means configuring your application with a specific secret version that is read on startup. While using the latest alias might be convenient, if there is a problem with the new version of the secret, your workload may be left unable to use the secret version. By pinning to a version number the configuration can be validated and rolled back using your existing release processes.

Disable secret versions before destroying them or deleting secrets. This helps prevent outages by putting the secret in a state that appears the same as destroy but is reversible. That is, you can disable and wait a week so that you can be sure there are no lingering dependencies before permanently deleting data.

Do not set expiration on production secrets unless you are certain that it should be irreversibly deleted. The expiration feature is best suited for automated cleanup of temporary environments. Consider time-based IAM conditions as an alternative to expiring secrets.

Periodically rotate your secrets to do the following:

Limit the impact in the case of a leaked secret.
Ensure individuals who no longer require access to a secret can not continue to use a previously accessed secret.
Continuously exercise the rotation flow to reduce the likelihood of an outage.

Monitor secrets across your organization using the Cloud Asset Inventory to...

Help identify secrets across your organization.
Identify non-conformance to organization requirements such as rotation, encryption configuration, and location.

Enable data access logs to obtain and analyze AccessSecretVersion request information. Enable this at the folder or organization level to enforce logging without having to configure it on every secret or project.

In addition to IAM controls you can limit access to the Secret Manager API with network-based controls by setting up a VPC Service Controls perimeter for your organization.

The constraints/iam.allowedPolicyMemberDomains organization policy can be used to limit the identities that can be added to IAM policies for secrets.

Estimate your peak secret usage (considering a "thundering herd" of requests due to concurrent application deploys or autoscaling of your service) and ensure that your project has enough quota to handle such an event. If more quota is needed, request an increase.