Receive Pub/Sub events at a private HTTP endpoint in a private GKE cluster

---

Preview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This tutorial shows you how to create a private HTTP endpoint in a private Google Kubernetes Engine (GKE) cluster that receives Pub/Sub message events using Eventarc. To learn more about this event destination, see Route events to an internal HTTP endpoint in a VPC network.

Private GKE clusters are a type of Virtual Private Cloud (VPC)-native cluster where nodes only have internal IP addresses, which means that nodes and Pods are isolated from the internet by default. You can choose to have no client access, limited access, or unrestricted access to the control plane. You can't convert an existing, non-private cluster to a private cluster. For more information, see About private clusters.

You can run the following commands using the Google Cloud CLI in either your terminal or Cloud Shell.

Objectives

In this tutorial, you will:

1. Create a proxy-only subnet in the default VPC network, and create a VPC firewall rule.
2. Create a private GKE Autopilot cluster with no client access to the public endpoint.
3. Create a Compute Engine virtual machine (VM) instance in a specified subnet of the VPC network.
4. Establish an SSH connection to the VM instance and deploy an event receiver service on the VM instance.
5. Deploy a Gateway in your cluster and an HTTPRoute manifest to configure the routing of traffic in Kubernetes to application backends.
6. Create a network attachment that lets a producer VPC network initiate connections to a consumer VPC network.
7. Create an Eventarc trigger that routes Pub/Sub events to the event receiver on your VM instance.
8. Publish a message to a Pub/Sub topic to generate an event, and view the body of the event in the application Pod logs.

Costs

In this document, you use the following billable components of Google Cloud:

• Compute Engine
• Eventarc
• Google Kubernetes Engine
• Pub/Sub
• VPC

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
2. Install the Google Cloud CLI.

3. To initialize the gcloud CLI, run the following command:

   gcloud init

4. Create or select a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   • Create a Google Cloud project:

     gcloud projects create PROJECT_ID

     Replace PROJECT_ID with a name for the Google Cloud project you are creating.

   • Select the Google Cloud project that you created:

     gcloud config set project PROJECT_ID

     Replace PROJECT_ID with your Google Cloud project name.

5. Make sure that billing is enabled for your Google Cloud project.

6. Enable the Cloud Resource Manager, Compute Engine, Eventarc, GKE, and Pub/Sub APIs:

   gcloud services enable compute.googleapis.com container.googleapis.com cloudresourcemanager.googleapis.com eventarc.googleapis.com pubsub.googleapis.com

7. Install the Google Cloud CLI.

8. To initialize the gcloud CLI, run the following command:

   gcloud init

9. Create or select a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   • Create a Google Cloud project:

     gcloud projects create PROJECT_ID

     Replace PROJECT_ID with a name for the Google Cloud project you are creating.

   • Select the Google Cloud project that you created:

     gcloud config set project PROJECT_ID

     Replace PROJECT_ID with your Google Cloud project name.

10. Make sure that billing is enabled for your Google Cloud project.

11. Enable the Cloud Resource Manager, Compute Engine, Eventarc, GKE, and Pub/Sub APIs:

    gcloud services enable compute.googleapis.com container.googleapis.com cloudresourcemanager.googleapis.com eventarc.googleapis.com pubsub.googleapis.com

12. Update Google Cloud CLI components:

    gcloud components update

13. Sign in using your account:

    gcloud auth login

14. If you are the project creator, you are granted the basic Owner role (roles/owner). By default, this Identity and Access Management (IAM) role includes the permissions necessary for full access to most Google Cloud resources and you can skip this step.

    If you are not the project creator, required permissions must be granted on the project to the appropriate principal. For example, a principal can be a Google Account (for end users) or a service account (for applications and compute workloads). For more information, see the Roles and permissions page for your event destination.

    Required permissions

    To get the permissions that you need to complete this quickstart, ask your administrator to grant you the following IAM roles on your project:

    • Compute Network Admin (roles/compute.networkAdmin)
    • Compute Security Admin (roles/compute.securityAdmin)
    • Eventarc Admin (roles/eventarc.admin)
    • Kubernetes Engine Admin (roles/container.admin)
    • Logs View Accessor (roles/logging.viewAccessor)
    • Project IAM Admin (roles/resourcemanager.projectIamAdmin)
    • Pub/Sub Publisher (roles/pubsub.publisher)
    • Service Account Admin (roles/iam.serviceAccountAdmin)
    • Service Account User (roles/iam.serviceAccountUser)
    • Service Usage Admin (roles/serviceusage.serviceUsageAdmin)

    For more information about granting roles, see Manage access.

    You might also be able to get the required permissions through custom roles or other predefined roles.

15. The Compute Engine default service account is automatically created after enabling or using a Google Cloud service that uses Compute Engine.

    For test purposes, you can attach this service account to an Eventarc trigger to represent the identity of the trigger. Note the email format to use when creating a trigger:

    [e-mail]
    

    Replace PROJECT_NUMBER with your Google Cloud project number. You can find your project number on the Welcome page of the Google Cloud console or by running the following command:

    gcloud projects describe PROJECT_ID --format='value(projectNumber)'

    The Compute Engine service account is automatically granted the basic Editor role (roles/editor) on your project. However, if automatic role grants have been disabled, refer to the applicable Roles and permissions instructions on to create a new service account and grant it the required roles.

    Caution: We recommend that you use the Compute Engine default service account for test purposes only. For production environments, we strongly recommend creating a new service account and granting it one or more IAM roles that contain the minimum permissions required and follow the principle of least privilege.
16. If you enabled the Cloud Pub/Sub service agent on or before April 8, 2021, to support authenticated Pub/Sub push requests, grant the Service Account Token Creator role (roles/iam.serviceAccountTokenCreator) to the Google-managed service account. Otherwise, this role is granted by default:

    gcloud projects add-iam-policy-binding PROJECT_ID \
        --member=serviceAccount:[e-mail] \
        --role=roles/iam.serviceAccountTokenCreator

Create a proxy-only subnet

Unless you create an organizational policy that prohibits it, new projects start with a default network (an auto mode VPC network) that has one subnetwork (subnet) in each region. Each VPC network consists of one or more IP address ranges called subnets. Subnets are regional resources, and have IP address ranges associated with them.

1. Use the gcloud compute networks subnets create command to create a proxy-only subnet in the default network.

   gcloud compute networks subnets create proxy-only-subnet \
       --purpose=REGIONAL_MANAGED_PROXY \
       --role=ACTIVE \
       --region=us-central1 \
       --network=default \
       --range=10.10.10.0/24
   

   Note that a subnet with purpose=REGIONAL_MANAGED_PROXY is reserved for Envoy-based load balancers and that the range must provide 64 or more IP addresses.

2. Create a firewall rule that matches the proxy-only subnet's range and that allows traffic on TCP port 8080.

   gcloud compute firewall-rules create allow-proxy-connection \
       --allow tcp:8080 \
       --source-ranges 10.10.10.0/24 \
       --network=default
   

Create a private GKE cluster

Use the gcloud container clusters create-auto command to create a private GKE cluster in Autopilot mode that has private nodes, and that has no client access to the public endpoint.

The following example creates a private GKE cluster named private-cluster and also creates a subnet named my-subnet:

gcloud container clusters create-auto private-cluster \
    --create-subnetwork name=my-subnet \
    --enable-master-authorized-networks \
    --enable-private-nodes \
    --enable-private-endpoint \
    --region=us-central1

Note the following:

• --enable-master-authorized-networks specifies that access to the public endpoint is restricted to IP address ranges that you authorize.
• --enable-private-nodes indicates that the cluster's nodes don't have external IP addresses.
• --enable-private-endpoint indicates that the cluster is managed using the internal IP address of the control plane API endpoint.

It might take several minutes for the creation of the cluster to complete. Once the cluster is created, the output should indicate that the status of the cluster is RUNNING.

Create a VM instance in a specified subnet

A Compute Engine VM instance is a virtual machine that is hosted on Google's infrastructure. The terms Compute Engine instance, VM instance, and VM are synonymous and are used interchangeably. VM instances include GKE clusters, App Engine flexible environment instances, and other Google Cloud products built on Compute Engine VMs.

Use the gcloud compute instances create command to create a Compute Engine VM instance in the subnet you created previously. Attach a service account and set the VM's access scope to cloud-platform.

gcloud compute instances create my-vm \
    --service-account=[e-mail] \
    --scopes=https://www.googleapis.com/auth/cloud-platform \
    --zone=us-central1-a \
    --subnet=my-subnet

For more information, see Create and start a VM instance.

Deploy an event receiver on the VM

Using a prebuilt image, us-docker.pkg.dev/cloudrun/container/hello, deploy a service on your VM that listens on port 80, and that receives and logs events.

1. Establish an SSH connection to your VM instance by running the following command:

   gcloud compute ssh my-vm --project=PROJECT_ID --zone=us-central1-a
   

   After a connection to the SSH server is established, run the remaining commands on your VM instance.

2. If necessary, install kubectl and any required plugins.

3. From your VM instance, use the get-credentials command to enable kubectl to work with the cluster you created.

   gcloud container clusters get-credentials private-cluster \
       --region=us-central1 \
       --internal-ip
   

4. Use a Kubernetes command, kubectl create deployment, to deploy an application to the cluster.

   kubectl create deployment hello-app \
       --image=us-docker.pkg.dev/cloudrun/container/hello
   

   This creates a Deployment named hello-app. The Deployment's Pod runs the hello container image.

5. After deploying the application, you can expose your application to traffic by creating a Kubernetes Service. Run the following kubectl expose command:

   kubectl expose deployment hello-app \
       --type ClusterIP \
       --port 80 \
       --target-port 8080
   

   You should see service/hello-app exposed in the output.

   You can ignore any messages similar to the following:

   E0418 14:15:33.970933    1129 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
   

Configure Kubernetes traffic routing

A Gateway resource represents a data plane that routes traffic in Kubernetes. A Gateway can represent many different kinds of load balancing and routing depending on the GatewayClass it is derived from. For more information, see Deploying Gateways. An HTTPRoute manifest is deployed to create Routes and send traffic to application backends.

1. Deploy a Gateway in your cluster.

   kubectl apply -f - <<EOF
   kind: Gateway
   apiVersion: gateway.networking.k8s.io/v1beta1
   metadata:
     name: internal-http
   spec:
     gatewayClassName: gke-l7-rilb
     listeners:
     - name: http
       protocol: HTTP
       port: 80
   EOF
   

   Note the following:

   • gatewayClassName: gke-l7-rilb specifies the GatewayClass that this Gateway is derived from. gke-l7-rilb corresponds to the internal Application Load Balancer.
   • port: 80 specifies that the Gateway exposes only port 80 for listening for HTTP traffic.

2. Validate that the Gateway has deployed correctly. It might take a few minutes for it to deploy all of its resources.

   kubectl describe gateways.gateway.networking.k8s.io internal-http
   

   The output is similar to the following:

   Name:         internal-http
   Namespace:    default
   ...
   API Version:  gateway.networking.k8s.io/v1beta1
   Kind:         Gateway
   ...
   Spec:
     Gateway Class Name:  gke-l7-rilb
     Listeners:
       Allowed Routes:
         Namespaces:
           From:  Same
       Name:      http
       Port:      80
       Protocol:  HTTP
   Status:
     Addresses:
       Type:   IPAddress
       Value:  10.36.172.5
   ...
   Events:
     Type    Reason  Age                From                   Message
     ----    ------  ----               ----                   -------
     Normal  ADD     80s                sc-gateway-controller  default/internal-http
     Normal  UPDATE  20s (x3 over 80s)  sc-gateway-controller  default/internal-http
     Normal  SYNC    20s                sc-gateway-controller  SYNC on default/internal-http was a success
   

3. Deploy an HTTPRoute manifest to route HTTP traffic to the hello-app service at port 80.

   kubectl apply -f - <<EOF
   kind: HTTPRoute
   apiVersion: gateway.networking.k8s.io/v1beta1
   metadata:
     name: hello-app-route
   spec:
     parentRefs:
     - kind: Gateway
       name: internal-http
     rules:
     - backendRefs:
       - name: hello-app
         port: 80
   EOF
   

Create a network attachment

A network attachment is a resource that lets a producer VPC network initiate connections to a consumer VPC network through a Private Service Connect interface.

To publish events, Eventarc uses the network attachment to establish a connection to the internal HTTP endpoint hosted in a VPC network.

Note: Eventarc reserves the subnet CIDR IP range 240.0.0.0/20. The subnet used for the network attachment shouldn't overlap with this range.

You can create a network attachment that automatically accepts connections from any Private Service Connect interface that refers to the network attachment. Create the network attachment in the same network and region containing the HTTP destination service.

gcloud compute network-attachments create my-network-attachment \
    --region=us-central1 \
    --subnets=my-subnet\
    --connection-preference=ACCEPT_AUTOMATIC

For more information, see About network attachments.

Create an Eventarc trigger

Create an Eventarc trigger that creates a new Pub/Sub topic and routes events to the event receiver deployed on the VM when a message is published to the Pub/Sub topic.

1. Retrieve the Gateway address.

   GATEWAY_ADDRESS=$(kubectl get gateways.gateway.networking.k8s.io internal-http -o=jsonpath="{.status.addresses[0].value}")
   

2. Create a trigger.

   gcloud eventarc triggers create my-trigger \
       --location=us-central1 \
       --destination-http-endpoint-uri="http://$GATEWAY_ADDRESS:80/" \
       --network-attachment="projects/PROJECT_ID/regions/us-central1/networkAttachments/my-network-attachment" \
       --event-filters="type=google.cloud.pubsub.topic.v1.messagePublished" \
       --service-account=[e-mail]
   

   Replace PROJECT_NUMBER with your Google Cloud project number. You can find your project number on the Welcome page of the Google Cloud console or by running the following command:

   gcloud projects describe PROJECT_ID --format='value(projectNumber)'
   

For more information about configuring your trigger, see Route events to an internal HTTP endpoint in a VPC network.

Note: It might take several minutes to create your trigger, and then it can take up to two minutes for the trigger to be fully functional.

Generate and view a Pub/Sub topic event

You can generate an event by publishing a message to a Pub/Sub topic.

1. Find and set the Pub/Sub topic as an environment variable.

   export MY_TOPIC=$(gcloud eventarc triggers describe my-trigger \
       --location=us-central1 \
       --format='value(transport.pubsub.topic)')
   

2. Publish a message to the Pub/Sub topic to generate an event.

   gcloud pubsub topics publish $MY_TOPIC --message "Hello World"
   

   The Eventarc trigger routes the event to the internal HTTP endpoint in the private GKE cluster.

3. Check the application Pod logs and verify the event delivery.

   POD_NAME=$(kubectl get pod --selector app=hello-app --output=name)
   kubectl logs $POD_NAME
   

   The body of the event should be similar to the following:

   2024/04/18 20:31:43 Hello from Cloud Run! The container started successfully and is listening for HTTP requests on $PORT
   {"severity":"INFO","eventType":"google.cloud.pubsub.topic.v1.messagePublished","message":"Received event of type google.cloud.pubsub.topic.v1.messagePublished.
   Event data: Hello World","event":{"specversion":"1.0","id":"10935738681111260","source":"//pubsub.googleapis.com/projects/my-project/topics/eventarc-us-central1-my-trigger-224","type":"google.cloud.pubsub.topic.v1.messagePublished","datacontenttype":"application/json","time":"2024-04-18T20:40:03Z","data":
   {"message":{"data":"SGVsbG8gV29ybGQ=","messageId":"10935738681111260","publishTime":"2024-04-18T20:40:03Z"}}}}
   

You have successfully deployed an event receiver service to an internal HTTP endpoint in a private GKE cluster, created an Eventarc trigger, generated an event from Pub/Sub, and confirmed that the event was routed as expected by the trigger to the target endpoint.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

Delete a Google Cloud project:

gcloud projects delete PROJECT_ID

Delete individual resources

1. Delete the Eventarc trigger:

     gcloud eventarc triggers delete my-trigger --location=us-central1

2. Exit the VM and then delete the VM instance:

     gcloud compute instances delete my-vm --zone=us-central1-a

3. Delete the network attachment:

     gcloud compute network-attachments delete my-network-attachment --region=us-central1

4. Delete the firewall rule:

     gcloud compute firewall-rules delete allow-proxy-connection

5. Delete the cluster:

     gcloud container clusters delete private-cluster --region=us-central1
     

6. Delete the subnet:

     gcloud compute networks subnets delete proxy-only-subnet --region=us-central1

What's next

• Receive Pub/Sub events at an internal HTTP endpoint in a VPC network
• Manage Eventarc triggers
• View all event providers and destinations