Create an enrollment to receive events

An enrollment identifies a subscription to a particular bus. The enrollment defines the matching criteria that determines which messages are routed to a destination. It also specifies the pipeline that matched messages should be routed through. A pipeline lets you configure a target destination and also provides the option of transforming any matched events prior to delivering them to the destination.

Note the following:

  • A pipeline and enrollment must be in the same Google Cloud project. (The bus can be in the same or a different project.)
  • One pipeline can be used for multiple enrollments.
  • Only one destination can be the target of messages routed by a pipeline.
  • Before configuring either a pipeline or an enrollment, you should have already created an Eventarc Advanced bus.

Required roles

An Identity and Access Management (IAM) role contains a set of permissions that lets you perform specific actions on Google Cloud resources. The following roles and permissions are required when creating a pipeline and an enrollment to route messages:

  • To get the permission that you need to create a pipeline, ask your administrator to grant you the Eventarc Developer (roles/eventarc.developer) IAM role on your pipeline project. This predefined role contains the eventarc.pipelines.create permission, which is required to create a pipeline.
  • To get the permission that you need to create an enrollment, ask your administrator to grant you the Eventarc Developer (roles/eventarc.developer) IAM role on your enrollment project. This predefined role contains the eventarc.enrollments.create permission, which is required to create an enrollment.
  • To get the permission that you need to use a bus, ask your administrator to grant you the Eventarc Message Bus User (roles/eventarc.messageBusUser) IAM role on your bus project. This predefined role contains the eventarc.buses.use permission, which is required to use a bus.

For more information about granting roles, see Manage access. You might also be able to get these permissions with custom roles or other predefined roles.

Enroll to receive events

You can create a pipeline and an enrollment in the following ways:

  • In the Google Cloud console
  • By using the Google Cloud CLI in either your terminal or Cloud Shell
  • By sending a request to the Eventarc API

Console

In the Google Cloud console, you can create both the pipeline and enrollment at the same time from the Pipelines page.

  1. To create an enrollment, in the Google Cloud console, go to the Eventarc > Pipelines page.

    Go to Pipelines

  2. Click Create pipeline.

  3. In the Pipeline details pane, do the following:

    1. Enter a Pipeline name. This is the ID for your pipeline.
    2. In the Region list, select a region to deploy your pipeline in. The pipeline must be created in the same region as the bus. For more information, see Eventarc Advanced locations.
    3. Optional: You can modify the default configuration to retry events.
    4. For Encryption, accept the default Google-managed encryption key or select Cloud KMS key. For more information, see Use customer-managed encryption keys (CMEK).

    5. If you select Cloud KMS key, do the following:

      1. In the Key type list, select a method to manage your keys.

        You can manage your keys manually or you can use Autokey which lets you generate key rings and keys on-demand. If the Autokey option is disabled, it isn't yet integrated with the current resource type.

      2. In the Select a customer-managed key, select a key.

        Note that you must select a region before you can view your customer-managed keys.

      3. Optional: To manually enter the resource name of the key, in the Select a customer-managed key list, click Enter key manually, and enter the key name in the specified format.

      4. If prompted, grant the cloudkms.cryptoKeyEncrypterDecrypter role to the Eventarc Service Agent.

    6. Optional: To add labels, click Add label. Labels are key-value pairs that help you organize your Google Cloud resources. For more information, see What are labels?

    7. Click Continue.

  4. In the Enrollments pane, do the following:

    1. Click Add an enrollment.
    2. Enter an Enrollment name.
    3. In the Bus list, select a bus to subscribe to.

    4. In the CEL expression field, write an evaluation expression using CEL. For example:

      message.type == "google.cloud.dataflow.job.v1beta3.statusChanged"

      Note that the default expression, true, indicates that all messages are routed, unfiltered.

    5. Click Done.

    6. You can add another enrollment or click Continue.

  5. Optional: In the Event mediation pane, do the following or click Continue:

    1. Select the Apply a transformation checkbox.

    2. In the Inbound format list, select the applicable format.

      Note that when applying a transformation, you must specify an inbound data format for a pipeline, and that all events must match that format.

    3. For Avro or Protobuf formats, you must specify an inbound schema. Optionally, you can upload an inbound schema. For more information, see Format received events.

    4. In the CEL expression field, write a transformation expression using CEL.

    5. Click Continue.

  6. In the Destination pane, do the following:

    1. In the Destination type list, select a destination type to route messages to. Depending on the destination type, do the following:

      • HTTP endpoint: Specify the destination URI. The host can be either a static IP addressable from a Virtual Private Cloud (VPC) network, or the internal Domain Name System (DNS) hostname of a service resolvable using Cloud DNS.

        Note that you can use this destination type to target a Cloud Run function.

      • Eventarc Advanced bus: Select an Eventarc Advanced bus.

      • Workflows workflow: Select a Workflows workflow.

      • Pub/Sub topic: Select or create a Pub/Sub topic.

      • Cloud Run service (via HTTP): Select a Cloud Run service that will receive events as HTTP POST requests sent to its root URL path (/); optionally, you can specify a relative URL path on the destination service to which events should be sent.

      • Cloud Run job (via HTTP): Select a Cloud Run job that will receive events as HTTP POST requests.

    2. Specify a network attachment.

      A network attachment is a resource that lets a producer VPC network initiate connections to a consumer VPC network. To publish events, Eventarc Advanced uses the network attachment to establish a connection to the endpoint hosted in a VPC network.

      You can create a network attachment that automatically accepts connections from any Private Service Connect interface that refers to the network attachment. Create the network attachment in the same network and region containing the destination endpoint.

      If you are routing messages to a Google destination using a DNS address—for example, Cloud Run, Pub/Sub, Workflows, or another Eventarc Advanced bus—make sure that Private Google Access is enabled on the subnet used in the network attachment; otherwise, the DNS address can't be resolved.

    3. If applicable, in the Outbound format list, select a format.

      Note that if an inbound data format is not specified for a pipeline, an outbound format can't be set.

    4. If applicable, apply a Message binding. For more information, see Define a message binding.

    5. To enable authentication, select the Enable authentication checkbox.

      1. In the Auth header list, select the type of token to generate and attach as an Authorization header in the HTTP request:

        • An OAuth token should generally be used only when calling Google APIs hosted on *.googleapis.com. Optionally, specify the Scope of this token; otherwise, it defaults to https://www.googleapis.com/auth/cloud-platform. For Google Cloud services, it's a best practice to use the https://www.googleapis.com/auth/cloud-platform scope, which includes all Google Cloud APIs, together with Identity and Access Management (IAM), which provides fine-grained access control.

          Note that all requests to another Eventarc Advanced bus, Pub/Sub, or Workflows must have an HTTP Authorization header containing an OAuth token signed by Google for one of the authorized service accounts.

        • An OIDC token can be used for many scenarios, including endpoints where you intend to validate the token yourself. Additionally, specify the Audience that this token is intended for. Generally, it should match the URL of the target pipeline. If not specified, the entire URL is used, including any request parameters.

          Note that Cloud Run performs an IAM check on every request and you can use the run.routes.invoke permission to configure who can access your Cloud Run service in either of the following ways:

          • Grant the permission to select service accounts or groups to allow access to the service. All requests must have an HTTP Authorization header containing an OpenID Connect token signed by Google for one of the authorized service accounts.

          • Grant the permission to allUsers to allow unauthenticated access.

          For more information, see Access control for Cloud Run.

        Learn more about token types.

      2. In the Service account list, select the service account that will invoke your destination service. Or, you can create a new service account.

        This specifies the IAM service account email associated with the pipeline and to which you previously granted specific roles required by Eventarc Advanced.

  7. Click Create.

gcloud

When using the gcloud CLI, first create the pipeline, and then create the enrollment using the appropriate commands.

Pipeline

  1. Open a terminal.

  2. Create a pipeline using the gcloud beta eventarc pipelines create command:

    gcloud beta eventarc pipelines create PIPELINE_NAME \
    --destinations=DESTINATION_KEY \
    --location=REGION

    Replace the following:

    • PIPELINE_NAME: the ID of the pipeline or a fully qualified name

    • DESTINATION_KEY: one or more key-value pairs to configure a destination for the pipeline

      You must set only one of the following keys:

      You must set the following key:

      • network_attachment—a resource that lets a producer VPC network initiate connections to a consumer VPC network. To publish events, Eventarc Advanced uses the network attachment to establish a connection to the endpoint hosted in a VPC network.

        You can create a network attachment that automatically accepts connections from any Private Service Connect interface that refers to the network attachment. Create the network attachment in the same network and region containing the destination resource.

        If you are routing messages to a Google destination using a DNS address—for example, Cloud Run, Pub/Sub, Workflows, or another Eventarc Advanced bus—make sure that Private Google Access is enabled on the subnet used in the network attachment; otherwise, the DNS address can't be resolved.

      To enable authentication, you can set one of the following keys:

      • google_oidc_authentication_service_account—the service account email used to generate an an OIDC token which can be used for many scenarios, including endpoints where you intend to validate the token yourself. Optionally, you can set google_oidc_authentication_audience to specify the audience that this token is intended for. Generally, it should match the URL of the target pipeline. If not specified, the entire URL is used, including any request parameters.

        Note that Cloud Run performs an IAM check on every request and you can use the run.routes.invoke permission to configure who can access your Cloud Run service in either of the following ways:

        • Grant the permission to select service accounts or groups to allow access to the service. All requests must have an HTTP Authorization header containing an OpenID Connect token signed by Google for one of the authorized service accounts.

        • Grant the permission to allUsers to allow unauthenticated access.

        For more information, see Access control for Cloud Run .

      • oauth_token_authentication_service_account—the service account email used to generate an OAuth token which should generally be used only when calling Google APIs hosted on *.googleapis.com. Optionally, you can set oauth_token_authentication_scope to specify the scope of this token; otherwise, it defaults to https://www.googleapis.com/auth/cloud-platform. For Google Cloud services, it's a best practice to use the https://www.googleapis.com/auth/cloud-platform scope, which includes all Google Cloud APIs, together with Identity and Access Management (IAM), which provides fine-grained access control.

        Note that all requests to another Eventarc Advanced bus, Pub/Sub, or Workflows must have an HTTP Authorization header containing an OAuth token signed by Google for one of the authorized service accounts.

        Learn more about token types.

      Optional: You can set one of the following keys:

      • output_payload_format_avro_schema_definition
      • output_payload_format_json

      • output_payload_format_protobuf_schema_definition

        Note that if you set an output format, you must also specify an input format (see the following input-payload-format-* flags).

      Optional: If http_endpoint_uri is not used as a destination key, you can set the following keys:

      • project—the Google Cloud project ID of the destination resource; by default, the project ID for the pipeline is used.
      • location—the location of the destination resource; by default, the location for the pipeline is used.

    • REGION: a supported Eventarc Advanced location

      Alternatively, you can set the gcloud CLI location property:

      gcloud config set eventarc/location REGION

    Optional: You can use the following flags:

    • --async to return from the command immediately, without waiting for the operation in progress to complete.
    • --crypto-key to specify the fully qualified name of a customer-managed encryption key; if unspecified, Google-owned and managed keys are used.
    • --logging-config to configure the logging level which must be one of the following: NONE, DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, ALERT, EMERGENCY.

    • --mediations to apply a transformation; transformation_template is the only supported template and only one mediation per pipeline is supported; for example:

      --mediations=transformation_template='message.removeFields(["id\ ","credit_card_number","age"])'

      Note that if you are applying a transformation, you must use one of the following flags to specify an input format.

    • One of the following to specify an input format:

      • --input-payload-format-avro-schema-definition
      • --input-payload-format-json
      • --input-payload-format-protobuf-schema-definition

    • --max-retry-attempts, --max-retry-delay, and --min-retry-delay to retry events

    For example:

    gcloud beta eventarc pipelines create my-pipeline \
    --destinations=http_endpoint_uri='https://example-endpoint.com',network_attachment=my-network-attachment,oauth_token_authentication_service_account=example-service-account@example-project.gserviceaccount.iam.com,oauth_token_authentication_scope='https://www.googleapis.com/auth/cloud-platform',output_payload_format_avro_schema_definition='{"type": "record","name": "my_record", "fields": [{"name": "my_field", "type":"string"}]}' \
    --input-payload-format-avro-schema-definition='{"type":"record", "name": "my_record", "fields": [{"name": "my_field","type": "string"}]}' \
    --location=us-central1 \
    --async

    For more details and examples, see the gcloud CLI documentation.

Enrollment

  1. Open a terminal.

  2. Create an enrollment using the gcloud beta eventarc enrollments create command:

    gcloud beta eventarc enrollments create ENROLLMENT_NAME \
    --cel-match=MATCH_EXPRESSION \
    --destination-pipeline=PIPELINE_NAME \
    --message-bus=BUS_NAME \
    --message-bus-project=PROJECT_ID \
    --location=REGION

    Replace the following:

    • ENROLLMENT_NAME: the ID of the enrollment or a fully qualified name
    • MATCH_EXPRESSION: the matching expression for this enrollment using CEL—for example, "message.type == 'google.cloud.dataflow.job.v1beta3.statusChanged'"
    • PIPELINE_NAME: the target pipeline ID or its fully qualified name for this enrollment
    • BUS_NAME: the ID of the Eventarc Advanced bus or its fully qualified name
    • PROJECT_ID: the Google Cloud project ID for the bus

    • REGION: a supported Eventarc Advanced location

      Alternatively, you can set the gcloud CLI location property:

      gcloud config set eventarc/location REGION

    Optional: You can use the following flag:

    • --async to return from the command immediately, without waiting for the operation in progress to complete

    For example:

    gcloud beta eventarc enrollments create my-enrollment \
    --cel-match="message.type == 'google.cloud.dataflow.job.v1beta3.statusChanged'" \
    --destination-pipeline=my-pipeline \
    --message-bus=my-message-bus \
    --message-bus-project=another-google-cloud-project \
    --location=us-central1 \
    --async

REST API

Pipeline

To create a pipeline, use the projects.locations.pipelines.create method.

Before using any of the request data, make the following replacements:

  • name: the full resource name of the pipeline in the format projects/PROJECT_ID/locations/LOCATION/pipelines/PIPELINE_NAME

    Replace the following:

    • PROJECT_ID: the Google Cloud project ID for the pipeline project
    • LOCATION: the region in which the pipeline is to be deployed—for example, us-central1
    • PIPELINE_NAME: the name of the pipeline
  • LABEL_KEY and LABEL_VALUE: optional. A map of label key and value pairs that help you organize your Google Cloud resources. For more information, see What are labels?
  • ANNOTATION_KEY and ANNOTATION_VALUE: optional. A map of annotation key and value pairs of free-form text. You can use them to attach arbitrary information associated with the resource. For more information, see Annotations.
  • LOG_SEVERITY: optional. The minimum severity of the event described in a log entry. One of: NONE, DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, ALERT, EMERGENCY. The default is NONE. For more information, see LogSeverity.
  • destinations: the destination to which messages will be forwarded. Only one destination is supported: httpEndpoint, messageBus, topic, or workflow. For more information, see the Destination target. For example, replace the following:
    • HTTP_URI: a destination URI. The host can be either a static IP addressable from a Virtual Private Cloud (VPC) network, or the internal Domain Name System (DNS) hostname of a service resolvable using Cloud DNS. Optionally, you can also set messageBindingTemplate to specify a message binding.
    • networkAttachment: the full resource name of a network attachment this format: projects/NETWORK_ATTACHMENT_PROJECT_ID/regions/NETWORK_ATTACHMENT_LOCATION /networkAttachments/NETWORK_ATTACHMENT_NAME. A network attachment is resource that lets a producer Virtual Private Cloud network initiate connections to a consumer VPC network and must be specified. To publish events, Eventarc Advanced uses the network attachment to establish a connection to the endpoint hosted in a VPC network.

      You can create a network attachment that automatically accepts connections from any Private Service Connect interface that refers to the network attachment. Create the network attachment in the same network and region containing the destination resource.

      If you are routing messages to a Google destination using a DNS address—for example, Cloud Run, Pub/Sub, Workflows, or another Eventarc Advanced bus—make sure that Private Google Access is enabled on the subnet used in the network attachment; otherwise, the DNS address can't be resolved.

    • SERVICE_ACCOUNT_EMAIL: the name of the service account email used to generate an OIDC token which can be used for many scenarios, including endpoints where you intend to validate the token yourself. Optionally, you can set audience to specify the audience that this token is intended for. Generally, it should match the URL of the target pipeline. If not specified, the entire URL is used, including any request parameters.

      Alternatively, you can use oauthToken to specify the service account email used to generate an OAuth token which should generally be used only when calling Google APIs hosted on *.googleapis.com. Optionally, you can set scope to specify the scope of this token; otherwise, it defaults to https://www.googleapis.com/auth/cloud-platform. For Google Cloud services, it's a best practice to use the https://www.googleapis.com/auth/cloud-platform scope, which includes all Google Cloud APIs, together with Identity and Access Management (IAM), which provides fine-grained access control.

      Note that all requests to another Eventarc Advanced bus, Pub/Sub, or Workflows must have an HTTP Authorization header containing an OAuth token signed by Google for one of the authorized service accounts.

      Learn more about token types.

Request JSON body: 

{
  "name": "projects/PROJECT_ID/regions/LOCATION/pipelines/PIPELINE_NAME",
  "labels": {"LABEL_KEY":"LABEL_VALUE"},
  "annotations": {"ANNOTATION_KEY":"ANNOTATION_VALUE"},
  "loggingConfig": {"logSeverity":"LOG_SEVERITY"},
  "destinations": [{"httpEndpoint": {"uri": "HTTP_URI"},"networkConfig":
  {"networkAttachment": "projects/NETWORK_ATTACHMENT_PROJECT_ID/regions/NETWORK_ATTACHMENT_LOCATION/networkAttachments/NETWORK_ATTACHMENT_NAME"},
  "authenticationConfig": {"googleOidc": {"serviceAccount": "SERVICE_ACCOUNT_EMAIL"}}}]
}

To send your request, expand one of these options:

If successful, the response body contains a newly created instance of Operation: 

{
  "name": "projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.eventarc.v1.OperationMetadata",
    "createTime": "2024-01-25T17:17:45.782370139Z",
    "target": "projects/PROJECT_ID/locations/LOCATION/pipelines/PIPELINE_NAME",
    "verb": "create",
    "requestedCancellation": false,
    "apiVersion": "v1"
  },
  "done": false
}

Enrollment

To create an enrollment, use the projects.locations.enrollments.create method.

Before using any of the request data, make the following replacements:

  • ENROLLMENT_NAME: the display name of your enrollment—for example, my_enrollment.
  • MATCH_EXPRESSION: a CEL expression identifying which messages this enrollment applies to. For example, message.type == 'hello-world-type'.
  • destination: the full resource name of the target pipeline in the format projects/PROJECT_ID/locations/LOCATION/pipelines/PIPELINE_NAME.

    Replace the following:

    • PROJECT_ID: the Google Cloud project ID for the pipeline project
    • LOCATION: the region in which the pipeline is deployed—for example, us-central1.
    • PIPELINE_NAME: the name of the pipeline
  • message_bus: the full resource name of the bus identifying the source of the messages in the format projects/BUS_PROJECT_ID/locations/BUS_LOCATION/messageBuses/BUS_NAME.

    Replace the following:

    • BUS_PROJECT_ID: the Google Cloud project ID for the bus project
    • BUS_LOCATION: the region in which the bus is deployed—for example, us-central1
    • BUS_NAME: the name of the message bus

Request JSON body: 

{
  "display_name": "ENROLLMENT_NAME",
  "cel_match": "MATCH_EXPRESSION",
  "message_bus": "projects/BUS_PROJECT_ID/locations/BUS_LOCATION/messageBuses/BUS_NAME",
  "destination": "projects/PROJECT_ID/locations/LOCATION/pipelines/PIPELINE_NAME"
}

To send your request, expand one of these options:

If successful, the response body contains a newly created instance of Operation: 

{
  "name": "projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.eventarc.v1.OperationMetadata",
    "createTime": "2024-01-25T17:17:45.782370139Z",
    "target": "projects/PROJECT_ID/locations/LOCATION/enrollments/ENROLLMENT_NAME",
    "verb": "create",
    "requestedCancellation": false,
    "apiVersion": "v1"
  },
  "done": false
}

Delete an enrollment

You can delete an enrollment in the following ways:

  • In the Google Cloud console
  • By using the gcloud CLI in either your terminal or Cloud Shell

  • By sending a request to the Eventarc API

Console

  1. To delete an enrollment, in the Google Cloud console, go to the Eventarc > Pipelines page.

    Go to Pipelines

  2. Click the name of the pipeline from which you want to delete the enrollment.

    The Pipeline details pane opens.

  3. Click Continue.

    The Enrollments pane opens.

  4. For the enrollment that you want to delete, click the delete icon.

  5. Click Save.

gcloud

  1. Open a terminal.

  2. Delete an enrollment using the gcloud beta eventarc enrollments delete command:

    gcloud beta eventarc enrollments delete ENROLLMENT_NAME \
      --location=REGION

    Replace the following:

REST API

To delete an enrollment, use the projects.locations.enrollments.delete method.

Before using any of the request data, make the following replacements:

  • ENROLLMENT_NAME: the display name of the enrollment you want to delete—for example, my_enrollment.
  • PROJECT_ID: your Google Cloud project ID.
  • LOCATION: the region in which the enrollment is deployed—for example, us-central1.

To send your request, expand one of these options:

If successful, the response body contains a newly created instance of Operation: 

{
  "name": "projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.eventarc.v1.OperationMetadata",
    "createTime": "2024-01-25T17:17:45.782370139Z",
    "target": "projects/PROJECT_ID/locations/LOCATION/enrollments/ENROLLMENT_NAME",
    "verb": "delete",
    "requestedCancellation": false,
    "apiVersion": "v1"
  },
  "done": false
}

Delete a pipeline

You can delete a pipeline in the following ways:

  • In the Google Cloud console
  • By using the gcloud CLI in either your terminal or Cloud Shell
  • By sending a request to the Eventarc API

Note that deleting a pipeline might take more than 10 minutes.

Console

  1. To delete a pipeline, in the Google Cloud console, go to the Eventarc > Pipelines page.

    Go to Pipelines

  2. In the list of pipelines, select the checkbox next to the name of the pipeline that you want to delete.

  3. Click Delete.

  4. Confirm the deletion by entering Delete.

  5. Click Delete.

gcloud

  1. Open a terminal.

  2. Delete a pipeline using the gcloud beta eventarc pipelines delete command:

    gcloud beta eventarc pipelines delete PIPELINE_NAME \
      --location=REGION

    Replace the following:

REST API

To delete a pipeline, use the projects.locations.pipelines.delete method.

Before using any of the request data, make the following replacements:

  • PIPELINE_NAME: the display name of the pipeline you want to delete—for example, my_pipeline.
  • PROJECT_ID: your Google Cloud project ID.
  • LOCATION: the region in which the pipeline is deployed—for example, us-central1.

To send your request, expand one of these options:

If successful, the response body contains a newly created instance of Operation: 

{
  "name": "projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.eventarc.v1.OperationMetadata",
    "createTime": "2024-01-25T17:17:45.782370139Z",
    "target": "projects/PROJECT_ID/locations/LOCATION/pipelines/PIPELINE_NAME",
    "verb": "delete",
    "requestedCancellation": false,
    "apiVersion": "v1"
  },
  "done": false
}

What's next