Set up notifications on a secret

This topic discusses support for event notifications in Secret Manager.

Overview

Event notifications sends information about changes to your secrets and secret versions to Pub/Sub. These notifications can be used to trigger arbitrary workflows, such as restarting an application when a new secret version is added, or notifying security engineers when a secret is deleted. For more information on how to use these notifications to trigger workflows, see the Pub/Sub documentation.

How event notifications work in Secret Manager

Secrets can be configured with a list of up to 10 Pub/Sub topics. Whenever an operation is performed that modifies the secret or one of its versions, Secret Manager will automatically publish a message to each of the Pub/Sub topics on that secret. Get, List, and Access calls do not result in message publications.

Pub/Sub messages have a set of "attribute" key-value pairs containing metadata about the event, as well as a "data" field containing a full JSON serialization of the Secret or SecretVersion resource which was created or modified. This JSON is a UTF-8 encoded string that represents the Secret or SecretVersion resource in exactly the form specified by the Secret Manager public API, encoded in JSON as specified in the proto3 JSON Mapping.

Event types

The following is a list of event types currently supported by Secret Manager:

Event type Description
SECRET_CREATE Sent when a new secret is successfully created.
SECRET_UPDATE Sent when a new secret is successfully updated.
SECRET_DELETE Sent when a secret is deleted, either because of a user-initiated request or secret expiration.
SECRET_VERSION_ADD Sent when a new secret version is successfully added.
SECRET_VERSION_ENABLE Sent when a secret version is enabled.
SECRET_VERSION_DISABLE Sent when a secret version is disabled.
SECRET_VERSION_DESTROY Sent when a secret version is destroyed.
SECRET_ROTATE Sent when it is time to rotate a secret. See Creating and managing rotation policies on secrets for more information.
TOPIC_CONFIGURED

This is a test message with no body or attributes other than eventType: TOPIC_CONFIGURED. This is sent when a secret is created or updated with a list of Pub/Sub topics, but does not indicate that the operation was successful.

A SECRET_CREATE or SECRET_UPDATE message will be sent immediately afterwards if the operation was successful.

Whenever topics are updated on a secret, a TOPIC_CONFIGURED message is sent to all topics on the secret, including ones that were already present.

Notification format

Notifications sent to the Pub/Sub topic consist of two parts:

  • Attributes: A set of key:value pairs describing the event.
  • Data: A string that contains the metadata of the changed object.

Attributes

Attributes are key:value pairs contained in notifications sent by Secret Manager to your Pub/Sub topic. All notifications other than TOPIC_CONFIGURED test messages always contain the following set of key:value pairs, regardless of the notification's data:

Attribute name Example Description
eventType SECRET_CREATE The type of event that has just occurred. See Event types for a list of possible values.
dataFormat JSON_API_V1 The format of the object data.
secretId projects/p/secrets/my-secret The full resource name of the secret on which the event occurred.
timestamp 2021-01-20T11:17:45.081104-08:00 The time the event occurred.

In addition, notifications sometimes contain the following set of key:value pairs:

Attribute name Example Description
versionId projects/p/secrets/my-secret/versions/456

The name of the secret version on which the event occurred.

This is only present on SECRET_VERSION_ADD, SECRET_VERSION_ENABLE, SECRET_VERSION_DISABLE, and SECRET_VERSION_DESTROY event notifications.

deleteType REQUESTED Whether the delete was requested by a user (REQUESTED) or due to secret expiration (EXPIRATION). Only present on SECRET_DELETE event notifications.

Data

The data field is a UTF-8 string that contains the metadata of the changed object. Data is either a Secret or Secret Version.

For SECRET_DELETE notifications, the metadata contained in the data field represents the object metadata as it was before the delete. For all other notifications, the metadata included in the data field represents the object metadata after the change occurs.

Limitations

Event notifications is available only in the Secret Manager v1 API and Google Cloud CLI.

Before you begin

You may choose to store all resources in the same project or to store secrets and Pub/Sub topics in separate projects. Complete the following prerequisites to set up Secret Manager and Pub/Sub:

  • Secret Manager:

    • Create or use an existing project to hold your Secret Manager resources.
    • If necessary, complete the steps mentioned in the Enable the Secret Manager API page of the Secret Manager guide.

  • Pub/Sub:

Authenticate to Google Cloud:

$ gcloud auth login --update-adc

Create a service agent identity

You need to create a service agent identity for each project that requires secrets with event notifications.

To create a service identity with Google Cloud CLI, run the following command:

$ gcloud beta services identity create \
    --service "secretmanager.googleapis.com" \
    --project "PROJECT_ID"

The previous command returns a service account name, using the following format:

service-PROJECT_NUMBER@gcp-sa-secretmanager.iam.gserviceaccount.com

You will grant this service account permission to publish on the Pub/Sub topics which will be configured on your secrets.

Save the service account name as an environment variable:

# This is from the output of the command above
$ export SM_SERVICE_ACCOUNT="service-...."

The environment variables for the Secret Manager project, Pub/Sub project, and Secret Manager service account must be set the entire time you are following this procedure.

Create Pub/Sub topics

Follow the Pub/Sub quickstart to create topics in your Pub/Sub project in the Google Cloud console. Alternatively, you can create topics with Google Cloud CLI as in this example.

$ gcloud pubsub topics create "projects/PUBSUB_PROJECT_ID/topics/PUBSUB_TOPIC_NAME"

Repeat this multiple times if you wish to create multiple Pub/Sub topics on the secret.

Grant the service account for Secret Manager permission to publish on the topics just created. This can be done through the Google Cloud console or with Google Cloud CLI. The following command grants the Pub/Sub Publisher role (roles/pubsub.publisher) on the my-topic Pub/Sub topic to the service account.

$ gcloud pubsub topics add-iam-policy-binding PUBSUB_TOPIC_NAME \
    --member "serviceAccount:${SM_SERVICE_ACCOUNT}" \
    --role "roles/pubsub.publisher"

Create Pub/Sub subscriptions

In order to view the messages published to a topic, you must also create a subscription to the topic. Follow the Pub/Sub quickstart to create subscriptions in your Pub/Sub project in the Google Cloud console. Alternatively, you can create subscriptions with Google Cloud CLI as in this example.

$ gcloud pubsub subscriptions create "projects/PUBSUB_PROJECT_ID/subscriptions/PUBSUB_SUBSCRIPTION_NAME" \
    --topic "projects/PUBSUB_PROJECT_ID/topics/PUBSUB_TOPIC_NAME"

Create a secret with topics configured

Create a secret with a list of up to 10 topics configured. All topics configured on a secret will receive event notifications when the secret or one of its versions is changed. The following command creates a secret with my-topic configured.

gcloud

To use Secret Manager on the command line, first Install or upgrade to version 378.0.0 or higher of the Google Cloud CLI. On Compute Engine or GKE, you must authenticate with the cloud-platform scope. 

$ gcloud secrets create SECRET_ID --topics TOPIC_NAME

API

These examples use curl to demonstrate using the API. You can generate access tokens with gcloud auth print-access-token. On Compute Engine or GKE, you must authenticate with the cloud-platform scope. 

$ curl "https://secretmanager.googleapis.com/v1/projects/PROJECT_ID/secrets?secretId=SECRET_ID" \
    --request "POST" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $(gcloud auth print-access-token)" \
    --data-binary @- <<EOF
{
  "replication":{
    "automatic":{}
  },
  "topics":{
    "name": "TOPIC_NAME"
  }
}
EOF

Update secret topics

Modify the Pub/Sub topics configured on a secret by updating the secret with the new Pub/Sub topic resource names. With Google Cloud CLI you can add or remove one or more topics from a secret, as well as clear all topics from the secret.

Add topics

Adds one or more topics to a secret. Adding a topic which is already present will have no effect.

$ gcloud secrets update "SECRET_ID" \
    --project "PROJECT_ID" \
    --add-topics "projects/PUBSUB_PROJECT_ID/topics/my-topic-2,projects/PUBSUB_PROJECT_ID/topics/PUBSUB_TOPIC_NAME"

Remove topics

Removes one or more topics from a secret. Removing a topic which is not present will have no effect.

$ gcloud secrets update "SECRET_ID" \
    --project "PROJECT_ID" \
    --remove-topics "projects/PUBSUB_PROJECT_ID/topics/PUBSUB_TOPIC_NAME,projects/PUBSUB_PROJECT_ID/topics/PUBSUB_OTHER_TOPIC_NAME"

Clear topics

Remove all topics from a secret.

$ gcloud secrets update SECRET_ID \
    --project "PROJECT_ID" \
    --clear-topics

Consume event notifications with Cloud Functions

Event notifications can be used to trigger arbitrary workflows by creating cloud functions to consume the Pub/Sub messages. See the Cloud Functions documentation for a full guide. The sample code below is for a cloud function that prints eventType, secretId and metadata whenever an event is published to the topic. List of all Event Types for Secret Manager can be found here.

C#

To run this code, first set up a C# development environment and install the Secret Manager C# SDK. On Compute Engine or GKE, you must authenticate with the cloud-platform scope. 

using CloudNative.CloudEvents;
using Google.Cloud.Functions.Framework;
using Google.Events.Protobuf.Cloud.PubSub.V1;
using System;
using System.Threading;
using System.Threading.Tasks;

// Triggered from a message on a Cloud Pub/Sub topic.
// The printed value will be visible in Cloud Logging
// (https://cloud.google.com/functions/docs/monitoring/logging).
namespace PubSubSample
{
    public class Function : ICloudEventFunction<MessagePublishedData>
    {
        public Task HandleAsync(CloudEvent cloudEvent, MessagePublishedData data, CancellationToken cancellationToken)
        {
          string eventType = data.Message.Attributes["eventType"];
          string secretId = data.Message.Attributes["secretId"];
          string secretMetadata = data.Message.TextData;
          Console.WriteLine($"Received {eventType} for {secretId}. New metadata: {secretMetadata}.");
          return Task.CompletedTask;
        }
    }
}

Go

To run this code, first set up a Go development environment and install the Secret Manager Go SDK. On Compute Engine or GKE, you must authenticate with the cloud-platform scope. 

import (
	"context"
	"fmt"
)

// PubSubMessage is the payload of a Pub/Sub event.
type PubSubMessage struct {
	Attributes PubSubAttributes `json:"attributes"`
	Data       []byte           `json:"data"`
}

// PubSubAttributes are attributes from the Pub/Sub event.
type PubSubAttributes struct {
	SecretId  string `json:"secretId"`
	EventType string `json:"eventType"`
}

// ConsumeEventNotification demonstrates how to consume and process the Pub/Sub
// notification from Secret Manager.
func ConsumeEventNotification(ctx context.Context, m PubSubMessage) (string, error) {
	// The printed value will be visible in Cloud Logging:
	//
	//     https://cloud.google.com/functions/docs/monitoring/logging
	//
	eventType := m.Attributes.EventType
	secretID := m.Attributes.SecretId
	data := m.Data

	return fmt.Sprintf("Received %s for %s. New metadata: %q.",
		eventType, secretID, data), nil
}

Java

To learn how to install and use the client library for Secret Manager, see Secret Manager client libraries.

To authenticate to Secret Manager, set up Application Default Credentials. For more information, see Set up authentication for a local development environment. 


import java.util.Base64;
import java.util.Map;
import java.util.logging.Logger;
import lombok.Data;

// Demonstrates how to consume and process a Pub/Sub notification from Secret Manager. Triggered
// by a message on a Cloud Pub/Sub topic.
// Ideally the class should implement a background function that accepts a Pub/Sub message.
// public class ConsumeEventNotification implements BackgroundFunction<PubSubMessage> { }
public class ConsumeEventNotification {

  // You can configure the logs to print the message in Cloud Logging.
  private static final Logger logger = Logger.getLogger(ConsumeEventNotification.class.getName());

  // Accepts a message from a Pub/Sub topic and writes it to logger.
  public static String accept(PubSubMessage message) {
    String eventType = message.attributes.get("eventType");
    String secretId = message.attributes.get("secretId");
    String data = new String(Base64.getDecoder().decode(message.data));
    String log = String.format("Received %s for %s. New metadata: %s", eventType, secretId, data);
    logger.info(log);
    return log;
  }

  // Event payload. Mock of the actual Pub/Sub message.
  @Data
  public static class PubSubMessage {

    byte[] data;
    Map<String, String> attributes;
    String messageId;
    String publishTime;
    String orderingKey;
  }
}

Node.js

To run this code, first set up a Node.js development environment and install the Secret Manager Node.js SDK. On Compute Engine or GKE, you must authenticate with the cloud-platform scope. 

/**
* Triggered from a message on a Cloud Pub/Sub topic.
* The printed value will be visible in Cloud Logging
* (https://cloud.google.com/functions/docs/monitoring/logging).
*
* @param {!Object} event Event payload.
* @param {!Object} context Metadata for the event.
*/
exports.smEventsFunction = (event, context) => {
  const eventType = event.attributes.eventType;
  const secretID = event.attributes.secretId;
  const secretMetadata = Buffer.from(event.data, 'base64').toString();
  console.log(`Received ${eventType} for ${secretID}. New metadata: ${secretMetadata}.`);
};

Python

To run this code, first set up a Python development environment and install the Secret Manager Python SDK. On Compute Engine or GKE, you must authenticate with the cloud-platform scope. 

import base64


def consume_event_notification(event: dict, unused_context: None) -> str:
    """
    consume_event_notification demonstrates how to consume and process a
    Pub/Sub notification from Secret Manager.
    Args:
          event (dict): Event payload.
          unused_context (google.cloud.functions.Context): Metadata for the event.
    """
    event_type = event["attributes"]["eventType"]
    secret_id = event["attributes"]["secretId"]
    secret_metadata = base64.b64decode(event["data"]).decode("utf-8")
    event_notification = (
        f"Received {event_type} for {secret_id}. New metadata: {secret_metadata}"
    )
    print(event_notification)
    return event_notification

Ruby

To run this code, first set up a Ruby development environment and install the Secret Manager Ruby SDK. On Compute Engine or GKE, you must authenticate with the cloud-platform scope. 

require "functions_framework"
require "base64"

# Triggered from a message on a Cloud Pub/Sub topic.
# The printed value will be visible in Cloud Logging
# (https://cloud.google.com/functions/docs/monitoring/logging).
FunctionsFramework.cloud_event "sm_events_function" do |event|
  message = event.data["message"]
  event_type = message["attributes"]["eventType"]
  secret_id = message["attributes"]["secretId"]
  message_data = Base64.decode64 message["data"]
  FunctionsFramework.logger.info "Received %s for %s. New metadata: %s." % [event_type, secret_id, message_data]
end

Misconfigured topics

If Pub/Sub topics are added to a secret in a Create or Update operation but Secret Manager cannot publish messages to the topic due to a misconfiguration, the operation will fail with an error message indicating why the publish failed. This could happen, e.g., if the topic does not exist, or if the Secret Manager service account does not have permission to publish.

If Pub/Sub topics are added to a secret and then afterwards the topic is changed so that Secret Manager can no longer publish messages (for instance, the topic is deleted, or the Secret Manager service account permissions are removed), Secret Manager will write logs to the Secret Manager Secret resource with a message indicating why the publish failed.

What's next