Publish and receive events by creating a bus and enrollment

Publish and receive events by creating a bus and enrollment

Advanced

Preview — Eventarc Advanced

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This quickstart shows you how to publish and receive event messages by creating an Eventarc Advanced bus and enrollment in your Google Cloud project.

A bus lets you centralize the flow of messages through your system, and acts as a router. It receives events from a message source or published by a provider, and evaluates them according to an enrollment.

An enrollment identifies a subscription to a particular bus, and defines the matching criteria for messages, causing them to be routed accordingly to one or more destinations.

In this quickstart, you:

1. Create a subnet and enable Private Google Access.

2. Create a network attachment.

3. Deploy an event receiver service to Cloud Run.

4. Create an Eventarc Advanced bus.

5. Create an Eventarc Advanced enrollment.

6. Publish an event message to the bus.

7. View the event data in the Cloud Run logs.

You can complete this quickstart using the Google Cloud CLI.

Before you begin

Security constraints defined by your organization might prevent you from completing the following steps. For troubleshooting information, see Develop applications in a constrained Google Cloud environment.

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
2. Install the Google Cloud CLI.

3. To initialize the gcloud CLI, run the following command:

   gcloud init

4. Create or select a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   • Create a Google Cloud project:

     gcloud projects create PROJECT_ID

     Replace PROJECT_ID with a name for the Google Cloud project you are creating.

   • Select the Google Cloud project that you created:

     gcloud config set project PROJECT_ID

     Replace PROJECT_ID with your Google Cloud project name.

5. Make sure that billing is enabled for your Google Cloud project.

6. Enable the Artifact Registry, Cloud Build, Cloud Run, Compute Engine, and Eventarc APIs:

   gcloud services enable artifactregistry.googleapis.com cloudbuild.googleapis.com compute.googleapis.com eventarc.googleapis.com eventarcpublishing.googleapis.com run.googleapis.com

7. Install the Google Cloud CLI.

8. To initialize the gcloud CLI, run the following command:

   gcloud init

9. Create or select a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   • Create a Google Cloud project:

     gcloud projects create PROJECT_ID

     Replace PROJECT_ID with a name for the Google Cloud project you are creating.

   • Select the Google Cloud project that you created:

     gcloud config set project PROJECT_ID

     Replace PROJECT_ID with your Google Cloud project name.

10. Make sure that billing is enabled for your Google Cloud project.

11. Enable the Artifact Registry, Cloud Build, Cloud Run, Compute Engine, and Eventarc APIs:

    gcloud services enable artifactregistry.googleapis.com cloudbuild.googleapis.com compute.googleapis.com eventarc.googleapis.com eventarcpublishing.googleapis.com run.googleapis.com

12. Update gcloud components:

    gcloud components update

13. Sign in using your account:

    gcloud auth login

14. Set the configuration variable used in this quickstart:

    REGION=REGION

    Replace REGION with a supported location for the bus.

15. If you are the project creator, you are granted the basic Owner role (roles/owner). By default, this Identity and Access Management (IAM) role includes the permissions necessary for full access to most Google Cloud resources and you can skip this step.

    If you are not the project creator, required permissions must be granted on the project to the appropriate principal. For example, a principal can be a Google Account (for end users) or a service account (for applications and compute workloads).

    Note that by default, Cloud Build permissions include permissions to upload and download Artifact Registry artifacts.

    Required permissions

    To get the permissions that you need to complete this quickstart, ask your administrator to grant you the following IAM roles on your project:

    • Cloud Build Editor (roles/cloudbuild.builds.editor)
    • Cloud Run Admin (roles/run.admin)
    • Eventarc Developer (roles/eventarc.developer)
    • Eventarc Message Bus Admin (roles/eventarc.messageBusAdmin)
    • Logs View Accessor (roles/logging.viewAccessor)
    • Project IAM Admin (roles/resourcemanager.projectIamAdmin)
    • Service Account Admin (roles/iam.serviceAccountAdmin)
    • Service Account User (roles/iam.serviceAccountUser)
    • Service Usage Admin (roles/serviceusage.serviceUsageAdmin)

    For more information about granting roles, see Manage access to projects, folders, and organizations.

    You might also be able to get the required permissions through custom roles or other predefined roles.

16. Grant the following roles on the project to the Compute Engine default service account. These roles are needed when building and deploying your container image:
    • Artifact Registry Writer: to upload Artifact Registry artifacts
    • Logs Writer: to write logs to Cloud Logging
    • Storage Object User: to access Cloud Storage objects

    gcloud projects add-iam-policy-binding PROJECT_ID \
        --member=serviceAccount:[e-mail] \
        --role=roles/artifactregistry.writer
    gcloud projects add-iam-policy-binding PROJECT_ID \
        --member=serviceAccount:[e-mail] \
        --role=roles/logging.logWriter
    gcloud projects add-iam-policy-binding PROJECT_ID \
        --member=serviceAccount:[e-mail] \
        --role=roles/storage.objectUser

    Replace PROJECT_NUMBER with your Google Cloud project number. You can find your project number on the Welcome page of the Google Cloud console or by running the following command:

    gcloud projects describe PROJECT_ID --format='value(projectNumber)'

17. By default, Cloud Run services are only callable by Project Owners, Project Editors, and Cloud Run Admins and Invokers. To set up authentication, grant the Cloud Run Invoker role (run.invoker) on your Google Cloud project to a service account:
    1. Create a service account. For testing purposes, you will attach this service account to an Eventarc Advanced pipeline to represent the identity of the pipeline.

       gcloud iam service-accounts create SERVICE_ACCOUNT_NAME

       Replace SERVICE_ACCOUNT_NAME with a name for your service account.
    2. Grant the roles/run.invoker IAM role to the service account:

       gcloud projects add-iam-policy-binding PROJECT_ID \
           --member="serviceAccount:SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com" \
           --role=roles/run.invoker

    Note that you can configure who can access your Cloud Run service in either of the following ways:

    • Grant permission to select service accounts or groups to allow access to the service. All requests must have an HTTP Authorization header containing an OpenID Connect token signed by Google for one of the authorized service accounts. This is the way that access is configured in this quickstart.
    • Grant permission to allUsers to allow unauthenticated access.

    For more information, see Access control for Cloud Run.

Create a subnet and enable Private Google Access

Unless you create an organizational policy that prohibits it, new Google Cloud projects start with a default Virtual Private Cloud (VPC) network (an auto mode VPC network) that has one subnetwork (subnet) in each region. Subnets have IP address ranges associated with them.

Because you are routing messages to a Cloud Run destination using a DNS address, you must enable Private Google Access on the subnet used in the network attachment; otherwise, the DNS address can't be resolved. For more information about private networking and Cloud Run, see Receive requests from VPC networks.

Create a subnet in your project's default network and use the --enable-private-ip-google-access flag to enable Private Google Access:

gcloud compute networks subnets create SUBNET_NAME \
    --network=default \
    --range=10.8.0.0/24 \
    --region=$REGION \
    --enable-private-ip-google-access

Replace SUBNET_NAME with the name of your subnet—for example, my-subnet.

Subnet IP ranges must be unique and non-overlapping within a VPC network and peered VPC network. For more information about subnet types and valid subnet ranges, see Subnets.

Create a network attachment

A network attachment is a resource that lets a producer VPC network initiate connections to a consumer VPC network. To publish events, Eventarc Advanced uses the network attachment to establish a connection to the endpoint hosted in a VPC network.

Create a network attachment in the same network and region containing the event destination endpoint, and that automatically accepts connections from any Private Service Connect interface that refers to the network attachment:

gcloud compute network-attachments create ATTACHMENT_NAME \
   --region=$REGION \
   --connection-preference=ACCEPT_AUTOMATIC \
   --subnets=SUBNET_NAME

Replace ATTACHMENT_NAME with the name of your network attachment—for example, my-network-attachment.

Create an Artifact Registry standard repository

Create an Artifact Registry standard repository to store your container image.

gcloud artifacts repositories create REPOSITORY \
    --repository-format=docker \
    --location=$REGION

Replace REPOSITORY with a unique name for the Artifact Registry repository—for example, my-repo.

Deploy an event receiver service to Cloud Run

Deploy a Cloud Run service that logs the contents of an event. This service is accessible only from VPC networks within the same project, and the service URL is not directly accessible because the service only allows authenticated invocations.

1. Clone the GitHub repository:

   git clone https://github.com/GoogleCloudPlatform/eventarc-samples.git

2. Change to the directory that contains the Cloud Run sample code:

   cd eventarc-samples/eventarc-advanced-quickstart/

3. Build a Docker container image and push the image to your repository:

   gcloud builds submit \
       --tag $REGION-docker.pkg.dev/PROJECT_ID/REPOSITORY/log-events:v1

4. Deploy the container image to Cloud Run:

   gcloud run deploy SERVICE_NAME \
       --image $REGION-docker.pkg.dev/PROJECT_ID/REPOSITORY/log-events:v1 \
       --platform managed \
       --ingress internal \
       --no-allow-unauthenticated \
       --region=$REGION

   Replace SERVICE_NAME with the name of your service—for example, my-service.

When you see the Cloud Run service URL, the deployment is complete. Note this URL so that you can use it in a subsequent step.

Create an Eventarc Advanced bus

Create an Eventarc Advanced bus in your project by using the gcloud beta eventarc message-buses create command:

gcloud beta eventarc message-buses create BUS_NAME \
    --location=$REGION

Replace BUS_NAME with the ID or fully qualified identifier of your bus—for example, my-bus.

Note: You can create only one bus per Google Cloud project.

For more information, see Create a bus to route messages.

Create an Eventarc Advanced enrollment

An enrollment determines which messages are routed to a destination. It also specifies the pipeline that messages should be routed through. The pipeline is used to configure a destination for the event messages.

For more information, see Create an enrollment to receive events.

When using the gcloud CLI, you first create a pipeline, and then create an enrollment.

Note: Creating a pipeline might take more than 10 minutes.

1. Create a pipeline by using the gcloud beta eventarc pipelines create command:

   gcloud beta eventarc pipelines create PIPELINE_NAME \
       --destinations=http_endpoint_uri='CLOUD_RUN_SERVICE_URL',network_attachment=ATTACHMENT_NAME,google_oidc_authentication_service_account=SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com \
       --location=$REGION

   Replace the following:

   • PIPELINE_NAME: the ID of the pipeline or a fully qualified name.
   • CLOUD_RUN_SERVICE_URL: the fully qualified URL of your Cloud Run service—for example, https://SERVICE_NAME-abcdef-uc.a.run.app. This is the destination for your event messages.

   Note that the google_oidc_authentication_service_account key specifies a service account email which is used to generate an OIDC token.

2. Create an enrollment by using the gcloud beta eventarc enrollments create command:

   gcloud beta eventarc enrollments create ENROLLMENT_NAME \
       --cel-match=MATCH_EXPRESSION \
       --destination-pipeline=PIPELINE_NAME \
       --message-bus=BUS_NAME \
       --message-bus-project=PROJECT_ID \
       --location=$REGION

   Replace the following:

   • ENROLLMENT_NAME: the ID of the enrollment or a fully qualified name.
   • MATCH_EXPRESSION: the matching expression for this enrollment using CEL—for example, "message.type == 'hello-world-type'".

Publish an event message to the bus

To directly publish a message to your bus, you can use the gcloud beta eventarc message-buses publish command or send a request to the Eventarc Publishing REST API. For more information, see Publish events directly.

The message must be in a CloudEvents format which is a specification for describing event data in a common way. The data element is the payload of your event. Any well-formed JSON can go in this field. For more information about CloudEvents context attributes, see Event format.

The following are examples of directly publishing an event to an Eventarc Advanced bus:

Example 1

You can publish an event to a bus using the gcloud CLI and an --event-data and other event attribute flags:

gcloud beta eventarc message-buses publish BUS_NAME \
    --event-data='{"key": "hello-world-data"}' \
    --event-id=hello-world-id-1234 \
    --event-source=hello-world-source \
    --event-type=hello-world-type \
    --event-attributes="datacontenttype=application/json" \
    --location=$REGION

Example 2

You can publish an event to a bus as a JSON message using the gcloud CLI and a --json-message flag:

gcloud beta eventarc message-buses publish BUS_NAME \
    --location=$REGION \
    --json-message='{"id": "hello-world-id-1234", "type":
 "hello-world-type", "source":
 "hello-world-source", "specversion": "1.0", "data":
 {"key": "hello-world-data"}}'

View the event data in the Cloud Run logs

After publishing an event to your Eventarc Advanced bus, you can check the logs of your Cloud Run service to verify that the event was received as expected.

1. Filter the log entries and return the output by using the gcloud logging read command:

   gcloud logging read 'textPayload: "hello-world-data"'
   

2. Look for a log entry similar to the following:

   insertId: 670808e70002b5c6477709ae
   labels:
   instanceId: 007989f2a10a4a33c21024f2c8e06a9de65d9b4fdc2ee27697a50379b3fab2f975b9233dc357d50b06270829b9b479d5a1ee54a10fa2cb2d98c5f77a0895e2be0f9e6e4b20
   logName: projects/PROJECT_ID/logs/run.googleapis.com%2Fstderr
   receiveTimestamp: '2024-10-10T17:03:35.424659450Z'
   resource:
   labels:
   ...
   type: cloud_run_revision
   textPayload: "[2024-10-21 15:33:19,581] INFO in server: Body: b'{\"value\":\"hello-world-data\"\
     }'"
   timestamp: '2024-10-10T17:03:35.177606Z'
   

   Note: If you don't see a log entry, wait a few minutes and retry the gcloud logging read command.

You have successfully created an Eventarc Advanced bus and enrollment, published an event message to the bus, and verified the expected outcome in the logs of the event receiver service.

Clean up

When you finish the tasks that are described in this quickstart, you can avoid continued billing by deleting the resources that you created:

1. Delete a VPC subnet.

2. Delete a VPC network attachment.

3. Delete an Artifact Registry repository.

4. Delete a Cloud Run service.

5. Delete Eventarc Advanced resources:

   1. Delete an enrollment.

   2. Delete a pipeline.

   3. Delete a bus.

Alternatively, you can delete your Google Cloud project to avoid incurring charges. Deleting your Google Cloud project stops billing for all the resources used within that project.

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

Delete a Google Cloud project:

gcloud projects delete PROJECT_ID

What's next

• Publish events from Google sources
• Publish events directly
• Transform received events
• Format received events