Setting up a Pub/Sub proxy for mobile clients on GKE

---

This tutorial shows you how to publish messages from mobile or client-side apps to Pub/Sub by using a proxy that handles authentication and authorization logic instead of client-side credentials.

While you can authenticate messages from the client to Pub/Sub by using Identity and Access Management (IAM), such long-lived credentials never expire. In client-side apps, these credentials can be discovered through techniques like app decompiling and reverse engineering.

Instead, you can offload authentication and authorization logic to a proxy that performs the following tasks:

• Authenticates incoming requests to validate the user.
• Forwards requests to Pub/Sub along with appropriate IAM permissions.

This tutorial shows how to implement a Pub/Sub proxy on Google Kubernetes Engine (GKE). This tutorial is intended for application developers and system architects who define and implement the design for mobile or client-side applications. It assumes you understand fundamental Kubernetes concepts and are familiar with Cloud Endpoints.

Request flow for this tutorial

To understand how Pub/Sub fits into a streaming pipeline, consider a clickstream analysis. In this use case, you might want to understand how users interact with your mobile app. To gain these insights, you capture user activity in real time. The following diagram shows the flow of data.

The data captured by the app is pushed to Pub/Sub through a proxy. Pub/Sub can have subscribers downstream, such as Dataflow or Dataproc, which aggregate the data so that you can perform meaningful analysis.

The following diagram shows a detailed view of the request flow that this tutorial follows.

The next sections explain how the various components in this diagram interact.

User authentication

Mobile apps can use various methods to authenticate users. The authentication flow is specific to your app. This tutorial shows one such solution for authenticating users. An implementation of this solution accompanies this tutorial.

Requests from the client app to the Pub/Sub proxy

The app backend generates a short-lived authentication token that the client stores locally (for example, by using the Android Keystore system or iOS keychain services). This tutorial uses OpenID Connect (OIDC) ID tokens to authenticate the client app. Google issues and signs the OIDC ID token.

The client-side app sends a request to the Pub/Sub proxy by using the OIDC ID token. The Pub/Sub proxy validates the token and forwards the request to Pub/Sub along with the appropriate IAM credentials.

Caution: If you plan to store sensitive information such as authentication tokens on the client side, read about storing secrets in Android and iOS devices. The storage and management of tokens is outside the scope of this tutorial.

Publishing messages

After the client app is successfully authenticated, the Pub/Sub proxy sends a publish request to Pub/Sub. Using IAM, Pub/Sub helps ensure that the caller (the Pub/Sub proxy) has the right permissions to send publish requests. In this tutorial, the Pub/Sub proxy uses the Compute Engine default service account to authenticate with Pub/Sub. The Compute Engine default service account has the Editor IAM role (roles/editor), which provides publisher access to the Pub/Sub proxy.

Objectives

• Create a GKE cluster to run a Pub/Sub proxy.
• Create a Pub/Sub topic.
• Deploy the Pub/Sub proxy.
• Configure Endpoints to authenticate requests to the Pub/Sub proxy.
• Verify that messages are published to Pub/Sub.

Costs

In this document, you use the following billable components of Google Cloud:

• Pub/Sub
• GKE
• Cloud Build
• Endpoints
• Container Registry

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

1. In the Google Cloud console, go to the project selector page.

   Go to project selector

2. Select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

3. Make sure that billing is enabled for your Google Cloud project.

4. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

5. Define the environment variables that you need for this tutorial:

       export PROJECT=$(gcloud config get-value project)
       export REGION=us-central1
       export ZONE=${REGION}-b
       export CLUSTER=pubsub-proxy
       export TOPIC=proxy-test
       export SERVICE_ACCOUNT=publish-test
       export ENDPOINTS_SERVICE="pubtest.endpoints.${PROJECT}.cloud.goog"
       export GENERATE_TOKEN="https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts"

6. Enable the APIs for Cloud Build, Compute Engine, Google Kubernetes Engine, Artifact Analysis, Container Registry, Endpoints, Service Management, Service Control, and Pub/Sub:

       gcloud services enable \
           cloudbuild.googleapis.com \
           compute.googleapis.com \
           container.googleapis.com \
           containeranalysis.googleapis.com \
           containerregistry.googleapis.com \
           endpoints.googleapis.com \
           servicemanagement.googleapis.com \
           servicecontrol.googleapis.com \
           pubsub.googleapis.com

Creating a Pub/Sub topic

• In Cloud Shell, create a Pub/Sub topic that you publish messages to:

  gcloud pubsub topics create $TOPIC
  

Creating a GKE cluster

1. In Cloud Shell, create a GKE cluster:

   gcloud container clusters create $CLUSTER \
       --zone $ZONE \
       --scopes "https://www.googleapis.com/auth/cloud-platform"
   

2. Get credentials for the running cluster:

   gcloud container clusters get-credentials $CLUSTER \
       --zone $ZONE \
       --project $PROJECT
   

Building a container image

1. In Cloud Shell, clone the code repository:

   git clone https://github.com/GoogleCloudPlatform/solutions-pubsub-proxy-rest
   

2. Use Cloud Build to build a container image from source, and then save it in Container Registry:

   cd solutions-pubsub-proxy-rest && \
       gcloud builds submit --tag gcr.io/$PROJECT/pubsub-proxy:v1
   

Creating a static external IP address

1. In Cloud Shell, create a static external IP address that is later assigned to the Pub/Sub proxy load balancer:

   gcloud compute addresses create service-ip --region $REGION
   

2. Store the static IP address in an environment variable, PROXY_IP:

   PROXY_IP=$(gcloud compute addresses describe service-ip \
       --region $REGION --format='value(address)')
   

Deploying Endpoints

The Pub/Sub proxy uses Endpoints to authenticate requests from users. Endpoints uses the Extensible Service Proxy (ESP) to provide API management features such as authentication, monitoring, tracing, and API lifecycle management. This tutorial uses Endpoints only for authenticating incoming requests to the Pub/Sub proxy.

In this tutorial, you deploy ESP as a sidecar with the Pub/Sub proxy. ESP intercepts and authenticates incoming requests before it forwards them to the Pub/Sub proxy.

1. In Cloud Shell, replace the [PROJECT_ID] placeholder with your Google Cloud project ID in the openapi.yaml file:

   sed -i -e "s/\[PROJECT_ID\]/$PROJECT/g" openapi.yaml
   

2. In the OpenAPI manifest file, replace the [IP_ADDRESS] placeholder with the value of PROXY_IP:

   sed -i -e "s/\[IP_ADDRESS\]/$PROXY_IP/g" openapi.yaml
   

3. Deploy the OpenAPI service definition to Endpoints:

   gcloud endpoints services deploy openapi.yaml
   

   The preceding command creates the following:

   • A managed service with the name that you have specified in the host field of the openapi.yaml file (pubtest.endpoints.project-id.cloud.goog), where project-id is the ID of your Google Cloud project.
   • A DNS A record that uses the service name and the Pub/Sub proxy load balancer IP address mapping that is defined in the x-google-endpoints extension in the openapi.yaml file.

   During deployment, you see a warning that you can ignore because this tutorial uses OIDC ID tokens for authentication instead of API keys.

   WARNING: openapi.yaml: Operation 'post' in path '/publish': Operation does
   not require an API key; callers may invoke the method without specifying an
   associated API-consuming project. To enable API key all the
   SecurityRequirement Objects
   (https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#security-requirement-object)
   inside security definition must reference at least one SecurityDefinition
   of type : 'apiKey'.
   

4. Check to see if the service is deployed correctly:

   gcloud endpoints services describe ${ENDPOINTS_SERVICE}
   

   The output is similar to the following:

   [...]
   producerProjectId: project-id
   serviceConfig:
     documentation:
       summary: Pub/Sub proxy exposed as an Endpoint API
   [...]
     name: pubtest.endpoints.project-id.cloud.goog
     title: PubSub Proxy
     usage: {}
   serviceName: pubtest.endpoints.project-id.cloud.goog
   

   In the output:

   • project-id: The ID of your Google Cloud project.

Deploying a proxy

1. In Cloud Shell, generate a self-signed SSL certificate to allow HTTPS connections to the proxy.

   Note: This step creates a self-signed certificate for the domain name that Endpoints provides (pubtest.endpoints.project-id.cloud.goog). However, you can also use your own domain name to expose the Pub/Sub proxy.

   openssl req -x509 -nodes -days 365 \
       -newkey rsa:2048 -keyout ./nginx.key \
       -out ./nginx.crt \
       -subj "/CN=${ENDPOINTS_SERVICE}"
   

2. Create a Kubernetes secret by using the SSL certificate and private key:

   kubectl create secret generic nginx-ssl \
       --from-file=./nginx.crt \
       --from-file=./nginx.key
   

3. Replace the [PROJECT_ID] placeholder in the deployment manifest file with your Google Cloud project ID:

   sed -i -e "s/\[PROJECT_ID\]/$PROJECT/g" kube/deployment.yaml
   

4. Replace the [IP_ADDRESS] placeholder in the service manifest file with the value of PROXY_IP:

   sed -i -e "s/\[IP_ADDRESS\]/$PROXY_IP/g" kube/service.yaml
   

5. Deploy the proxy:

   kubectl apply -f kube/
   

6. Verify that the deployment is successful:

   kubectl rollout status deployment/pubsub-proxy
   

   The output is similar to the following:

   [...]
   deployment "pubsub-proxy" successfully rolled out
   

7. Ensure that two containers (ESP and Pub/Sub proxy) are running in the Pod:

   kubectl get pods $(kubectl get pod \
       -l app=pubsub-proxy \
       -o jsonpath="{.items[0].metadata.name}") \
       -o jsonpath={.spec.containers[*].name}
   

   The output is similar to the following:

   esp  pubsub-proxy
   

8. Watch for the value of EXTERNAL-IP to change from <pending> to the static external IP address that you created earlier:

   kubectl get svc pubsub-proxy -w
   

   The output is similar to the following:

   NAME          TYPE          CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
   pubsub-proxy  LoadBalancer  10.7.247.212  <pending>     443:31104/TCP  6m32s
   pubsub-proxy  LoadBalancer  10.7.247.212  <PROXY_IP>    443:31104/TCP  6m5s
   

   To stop watching, press CTRL+C.

   After the Pub/Sub proxy is successfully deployed, it is exposed at https://${ENDPOINTS_SERVICE}/publish. It might take a few minutes for the new DNS configuration to propagate.

9. Verify the DNS configuration:

   watch nslookup ${ENDPOINTS_SERVICE}
   

   The output is similar to the following:

   Server:   169.254.169.254
   Address:  169.254.169.254#53
   
   Non-authoritative answer:
   Name: pubtest.endpoints.project-id.cloud.goog
   Address: gke-load-balancer-ip
   

   In the output:

   • gke-load-balancer-ip: The IP address of your GKE load balancer (proxy IP).

   To stop watching, press CTRL+C.

If any of the preceding steps results in an error, see the troubleshooting steps.

Generating an authentication token

The following procedure for generating an authentication token is intended as an example. For your production environment, you need a way for users to generate their own authentication tokens. For example, you can find sample code for obtaining an OIDC ID token programmatically in the Identity-Aware Proxy documentation.

To generate an authentication token, do the following:

1. Create a Google Cloud service account for which you generate an OIDC ID token:

   gcloud iam service-accounts create \
       $SERVICE_ACCOUNT \
       --display-name $SERVICE_ACCOUNT
   

2. Get the email identity of the service account:

   SA_EMAIL=${SERVICE_ACCOUNT}@${PROJECT}.iam.gserviceaccount.com
   

3. Grant the Service Account Token Creator IAM role (roles/iam.serviceAccountTokenCreator) for the service account:

   gcloud iam service-accounts add-iam-policy-binding $SA_EMAIL \
       --member user:$(gcloud config get-value account) \
       --role roles/iam.serviceAccountTokenCreator
   

   Note: It takes a few minutes for the policy update to take effect. Wait for 2-3 minutes before you proceed to the next step.

4. Using the IAM credentials API, generate an OIDC ID token.

   TOKEN=$(curl -s ${GENERATE_TOKEN}/${SA_EMAIL}:generateIdToken \
       -H "Content-Type: application/json" \
       -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
       -d '{"audience": "'${ENDPOINTS_SERVICE}'", "includeEmail": "true"}' | jq -r ".token")
   

   The Endpoints service name is specified in the audience field. The audience claim identifies the recipient that the token is intended for.

5. Verify that the token is successfully created:

   echo $TOKEN
   

   The JSON Web Token (JWT) is similar to the following:

   eyJhbGciOiJSUzI1NiIsImtpZCI6IjY4NjQyODlm[...].eyJhdWQiOiJwdWJ0ZXN0LmVuZHBvaW50cy52aXR
   hbC1vY3RhZ29uLTEwOTYxMi5jbG91ZC5nb[...].SjBI4TZjZAlYo6lFKkrvfAcVUp_AJzFKoSsjNbmD_n[...]
   

Calling Pub/Sub by using a proxy

1. In Cloud Shell, publish a test message:

   curl -i -k -X POST https://${ENDPOINTS_SERVICE}/publish \
       -H "Authorization: Bearer $TOKEN" \
       -H "Content-Type: application/json" \
       -d '{"topic": "'$TOPIC'", "messages": [ {"attributes": {"key1": "value1", "key2" : "value2"}, "data": "test data"}]}'
   

   Note: Because this tutorial uses a self-signed certificate, you can turn off certificate verification for the curl command by using the -k or -- insecure option. When you deploy to production, get an SSL/TLS certificate from a public certificate authority if you do not already have one. If you want to deploy the API by using a custom domain, you need to configure Cloud Endpoints Portal and the DNS settings in your domain name registrar as described in Setting up a custom domain.

   The output is similar to the following:

   HTTP/2 200
   server: nginx
   date: Sun, 02 Jun 2019 03:53:46 GMT
   ...
   

2. Check whether the message successfully published to the Pub/Sub topic:

   kubectl logs -f --tail=5 deployment/pubsub-proxy -c pubsub-proxy
   

   The Pub/Sub proxy deployment logs display the message Successfully published:

   2019-06-02 03:49:39.723:INFO:oejs.Server:main: Started @2554ms
   Jun 02, 2019 3:53:44 AM com.google.pubsub.proxy.publish.PublishMessage
   getPublisher
   INFO: Creating new publisher for: proxy-test
   Jun 02, 2019 3:53:47 AM
   com.google.pubsub.proxy.publish.PublishMessage$1 onSuccess
   INFO: Successfully published: 569006136173844
   

Troubleshooting

1. In Cloud Shell, check the state of both containers in the Pub/Sub proxy Pod:

   kubectl describe pods $(kubectl get pod -l app=pubsub-proxy \
       -o jsonpath="{.items[0].metadata.name}")
   

   In the log output, the status of the containers is Running:

   [...]
   Containers:
     esp:
   [...]
     State:  Running
       Started:  Fri, 21 Jun 2019 16:41:30 +0530
     Ready:  True
     Restart Count:  0
   [...]
     pubsub-proxy:
       State:  Running
         Started:  Fri, 21 Jun 2019 16:41:42 +0530
       Ready:  True
       Restart Count:  0
   [...]
   

2. (Optional) Check the container logs to see whether there are other errors. For example, to check the Pub/Sub proxy logs, run the following command:

   kubectl logs -f --tail=10 deployment/pubsub-proxy -c pubsub-proxy
   

For help with troubleshooting, see the following documents:

• Troubleshooting Endpoints with GKE
• Troubleshooting JWT validation
• Troubleshooting response errors

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, you can delete the Google Cloud project that you created for this tutorial, or delete the resources associated with this tutorial.

Delete the Google Cloud project

The easiest way to eliminate billing is to delete the project you created for the tutorial.

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the resources

If you want to keep the Google Cloud project that you used in this tutorial, delete the individual resources:

1. In Cloud Shell, delete the GKE cluster:

   gcloud container clusters delete $CLUSTER --zone $ZONE --async
   

2. Delete downloaded code, artifacts, and other dependencies:

   cd .. && rm -rf solutions-pubsub-proxy-rest
   

3. Delete the image in Container Registry:

   gcloud container images list-tags \
       gcr.io/$PROJECT/pubsub-proxy \
       --format 'value(digest)' | \
       xargs -I {} gcloud container images delete \
       --force-delete-tags --quiet \
       gcr.io/${PROJECT}/pubsub-proxy@sha256:{}
   

4. Delete the Pub/Sub topic:

   gcloud pubsub topics delete $TOPIC
   

5. Delete the service account:

   gcloud iam service-accounts delete $SA_EMAIL
   

6. Delete Endpoints:

   gcloud endpoints services delete ${ENDPOINTS_SERVICE}
   

7. Delete the static IP address:

   gcloud compute addresses delete service-ip --region $REGION
   

What's next

• Learn about authentication with Endpoints.
• Learn about architecture for using Pub/Sub for long-running tasks.
• Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.