Using Google Cloud Platform Service Broker (Beta, Deprecated) to access Pub/Sub

This tutorial demonstrates how to use the Kubernetes Service Catalog and the Google Cloud Service Broker to connect an application running on GKE with Pub/Sub.

Why use Service Catalog?

Service Catalog allows for applications running on a cluster to easily discover and connect to external services without having to manually import information such as credentials or endpoints. External service dependencies are modeled as Kubernetes resources, which can be easily integrated into your existing deployment process.

In this tutorial, we will use the following features of the Service Catalog:

  • Discovering Google Cloud services in Service Broker via Service Catalog.
  • Provisioning a service instance to create the Cloud resource.
  • Binding the provisioned service instance in the Kubernetes application to inject credentials into the cluster.
  • Using the service binding in the application to access the service instance, which points to the Cloud resource.

For more information on Service Catalog, refer to the Kubernetes Service Catalog documentation.

Objectives

This tutorial covers the following steps:

  1. Create a namespace
  2. Create a Cloud Identity and Access Management service account
  3. Create a Pub/Sub topic
  4. Create a Pub/Sub subscription to the topic
  5. Test the Pub/Sub service with a demo app
  6. Clean up

Before you begin

Take the following steps to enable the Kubernetes Engine API:
  1. Visit the Kubernetes Engine page in the Google Cloud Console.
  2. Create or select a project.
  3. Wait for the API and related services to be enabled. This can take several minutes.

  4. Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

Install the following command-line tools used in this tutorial:

  • gcloud is used to create and delete Kubernetes Engine clusters. gcloud is included in the Google Cloud SDK.
  • kubectl is used to manage Kubernetes, the cluster orchestration system used by Kubernetes Engine. You can install kubectl using gcloud: 
    gcloud components install kubectl

Set defaults for the gcloud command-line tool

To save time typing your project ID and Compute Engine zone options in the gcloud command-line tool, you can set the defaults: 
gcloud config set project [PROJECT_ID]
gcloud config set compute/zone [COMPUTE_ENGINE_ZONE]

Set up Service Catalog

  • Install Service Catalog and Service Broker in your GKE cluster.

  • Verify that the Service Catalog is running by using kubectl in the command-line to list all the Service Catalog processes:

    kubectl get deployment -n service-catalog

    All the deployments should report as AVAILABLE.

  • (Optional) Install the svcat CLI tool, which improves the user experience around working with the Service Catalog resources.

Step 1: Create a Namespace

A Namespace is optional, but it is recommended to make cleanup of this tutorial easier.

Alias the Namespace name as an environment variable:

NAMESPACE=pubsub-app

Create the Namespace for the application using Pub/Sub. Run the following command:

kubectl create namespace [NAMESPACE]

It should output the following message:

namespace "pubsub-app" created

Step 2: Create a Cloud IAM service account

You need a Cloud Identity and Access Management service account in order to use Google Cloud services in your application. For this tutorial, we will create one with the ID pubsub-sa.

Otherwise, you can create a Cloud IAM service account using Service Catalog. This can be done by provisioning and binding to a service instance of the cloud-iam-service-account service class.

Provision a Cloud IAM service account instance

Use the following command to store the service account name in an environment variable:

SERVICE_ACCOUNT_ID=pubsub-sa

svcat

Use the svcat command to provision a service account instance:

svcat provision gcp-iam \
    --class cloud-iam-service-account \
    --plan beta \
    --namespace [NAMESPACE] \
    --param accountId=$SERVICE_ACCOUNT_ID

The command should output the following message: 

  Name:        gcp-iam
  Namespace:   pubsub-app
  Status:
  Class:       cloud-iam-service-account
  Plan:        beta



Parameters:
  accountId: pubsub-sa

Check the status of the service instance with the following command:

svcat get instance gcp-iam --namespace [NAMESPACE]

The output should appear similar to the following:

   NAME     NAMESPACE              CLASS             PLAN   STATUS
+---------+------------+---------------------------+------+--------+
  gcp-iam   pubsub-app   cloud-iam-service-account   beta   Ready

The service instance is successfully provisioned when its STATUS is Ready.

kubectl

Create a ServiceInstance YAML file called iam_instance.yaml with the following spec. Make sure the value for serviceAccount field is the same as echo $SERVICE_ACCOUNT_ID:

apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceInstance
metadata:
  name: gcp-iam
  namespace: pubsub-app
spec:
  # This should match with a SERVICE EXTERNAL NAME
  # in the list of available services.
  clusterServiceClassExternalName: cloud-iam-service-account
  # This should match with a PLAN EXTERNAL NAME
  # in the list of available service plans.
  clusterServicePlanExternalName: beta
  parameters:
    accountId: pubsub-sa

Use the kubectl command to provision the service account instance using the YAML file:

kubectl apply -f iam_instance.yaml

The command should output the following message:

serviceinstance "gcp-iam" created

Check the status of the service instance with the following command:

kubectl get serviceinstances gcp-iam -n [NAMESPACE] -o 'custom-columns=INSTANCE-NAME:.metadata.name,SERVICE:.spec.clusterServiceClassExternalName,PLAN:.spec.clusterServicePlanExternalName,STATUS:.status.conditions[0].reason'

The output should appear similar to the following:

INSTANCE-NAME   SERVICE                     PLAN      STATUS
gcp-iam         cloud-iam-service-account   beta      ProvisionedSuccessfully

Once the service instance has been provisioned successfully, you can also check the corresponding Cloud IAM service account pubsub-sa@[PROJECT_ID].iam.gserviceaccount.com in the Cloud Console.

Bind to the Cloud IAM service account instance

Binding to the service account instance will create a secret in the same namespace. The secret will contain the service account name and a private key, which can be easily referenced by your application.

Use the following command to store the secret name in an environment variable:

SA_SECRET_NAME=pubsub-credentials

svcat

Use the svcat command to bind to the service account instance:

svcat bind gcp-iam --namespace [NAMESPACE] --secret-name $SA_SECRET_NAME

The command should output the following message: 

  Name:        gcp-iam
  Namespace:   pubsub-app
  Status:
  Instance:    gcp-iam



Parameters:
  {}

Check the status of the service binding with the following command:

svcat get binding gcp-iam --namespace [NAMESPACE]

The output should appear similar to the following:

   NAME     NAMESPACE    INSTANCE   STATUS
+---------+------------+----------+--------+
  gcp-iam   pubsub-app   gcp-iam    Ready

Wait until the service binding's status is Ready before continuing. This could take a minute.

kubectl

Create a ServiceBinding YAML file called iam_binding.yaml with the following spec:

apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceBinding
metadata:
  name: gcp-iam
  namespace: pubsub-app
spec:
  instanceRef:
    name: gcp-iam
  secretName: pubsub-credentials

Run the following command to create the service binding:

kubectl apply -f iam_binding.yaml

Check the status of the service binding by running the following command:

kubectl get servicebindings gcp-iam -n [NAMESPACE] -o 'custom-columns=BINDING-NAME:.metadata.name,SERVICE-INSTANCE:.spec.instanceRef.name,STATUS:.status.conditions[0].reason,OUTPUT-SECRET:.spec.secretName'

The output should be similar to the following:

BINDING-NAME      SERVICE-INSTANCE   STATUS               OUTPUT-SECRET
gcp-iam           gcp-iam            InjectedBindResult   gcp-iam

Once the service binding has been successfully created, confirm the secret exists with the following kubectl command:

kubectl get secrets $SA_SECRET_NAME --namespace [NAMESPACE]

The output should appear similar to the following:

NAME                 TYPE      DATA      AGE
pubsub-credentials   Opaque    2         1m

You can also view the secret containing the service account credentials:

kubectl get secrets $SA_SECRET_NAME --namespace [NAMESPACE] -o yaml

Notice that the privateKeyData value is Base64 encoded and contains the service account private key.

Step 3: Create a Pub/Sub topic

Provision a Pub/Sub service instance

Now we will use Service Catalog to provision a Pub/Sub service instance. Doing so will create a Pub/Sub topic for our application to publish to and read from.

svcat

Use the svcat command to provision a Pub/Sub service instance:

svcat provision gcp-pubsub \
    --class cloud-pubsub \
    --plan beta \
    --namespace [NAMESPACE] \
    --param topicId=pubsub-app

The command should output the following message: 

  Name:        gcp-pubsub
  Namespace:   pubsub-app
  Status:
  Class:       cloud-pubsub
  Plan:        beta



Parameters:
  topicId: pubsub-app

Check the status of the service instance with the following command:

svcat get instance gcp-pubsub --namespace [NAMESPACE]

The output should appear similar to the following:

     NAME      NAMESPACE       CLASS       PLAN   STATUS
+------------+------------+--------------+------+--------+
  gcp-pubsub   pubsub-app   cloud-pubsub   beta   Ready

The service instance is successfully provisioned when its STATUS is Ready.

kubectl

Create a ServiceInstance YAML file called pubsub_instance.yaml with the following spec:

apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceInstance
metadata:
  name: gcp-pubsub
  namespace: pubsub-app
spec:
  clusterServiceClassExternalName: cloud-pubsub
  # This should match with a PLAN EXTERNAL NAME
  # in the list of available service plans for Pub/Sub.
  clusterServicePlanExternalName: beta
  parameters:
    topicId: pubsub-app

Provision a Pub/Sub service instance with the following command:

kubectl create -f pubsub_instance.yaml

It should output the following message:

serviceinstance "gcp-pubsub" created

Check the Pub/Sub service instance status with the following command:

kubectl get serviceinstances gcp-pubsub -n [NAMESPACE] -o 'custom-columns=INSTANCE-NAME:.metadata.name,SERVICE:.spec.clusterServiceClassExternalName,PLAN:.spec.clusterServicePlanExternalName,STATUS:.status.conditions[0].reason'

It should output a status message similar to the following:

INSTANCE-NAME   SERVICE        PLAN      STATUS
gcp-pubsub      cloud-pubsub   beta      ProvisionedSuccessfully

Once the Pub/Sub service instance has been provisioned successfully, you can also check the corresponding Pub/Sub topic projects/[PROJECT_ID]/topics/pubsub-app in the Cloud Console.

Bind to the Pub/Sub service instance

Binding to a Pub/Sub service instance does the following:

  • Grant the specified Cloud IAM service account appropriate permissions to access the Pub/Sub service.
  • Make the information necessary to access the service instance available to the Kubernetes application in the form of a Kubernetes secret.
  • Optionally, create a subscription to the Pub/Sub topic.

This is required for the demo application, because the app will need to be permitted to access the service in order to execute as the assumed identity of the service account to publish messages to a Pub/Sub topic.

The Pub/Sub service class has multiple types of service bindings. First, we will create a service binding of type publisher, which contains two roles: roles/pubsub.publisher and roles/pubsub.viewer. The binding will allow our service account to publish to this topic and view the messages.

Tip: For more information on Pub/Sub's Cloud IAM roles and permissions, please refer to Pub/Sub's [access control documentation](/pubsub/docs/access-control#tbl_roles).

svcat

Use the svcat command to bind to the Pub/Sub topic as a publisher:

svcat bind gcp-pubsub --namespace [NAMESPACE] --params-json '{
  "serviceAccount": "'${SERVICE_ACCOUNT_ID}'",
  "roles": [
    "roles/pubsub.publisher",
    "roles/pubsub.viewer"
  ]
}'

The command should output the following message (serviceAccount value may be the name of the service account you created before): 

  Name:        gcp-pubsub
  Namespace:   pubsub-app
  Status:
  Instance:    gcp-pubsub



Parameters:
  roles:



    

  • roles/pubsub.publisher
    • 

  • roles/pubsub.viewer
serviceAccount: pubsub-sa

Check the status of the service binding with the following command:

svcat get binding gcp-pubsub --namespace [NAMESPACE]

The output should appear similar to the following:

     NAME      NAMESPACE     INSTANCE    STATUS
+------------+------------+------------+--------+
  gcp-pubsub   pubsub-app   gcp-pubsub   Ready

Wait until the service binding's status is Ready before continuing. This could take a minute.

kubectl

Create a ServiceBinding YAML file called pubsub_binding.yaml with the following spec to bind to the Pub/Sub topic as a publisher. Make sure the value for serviceAccount field is the same as echo $SERVICE_ACCOUNT_ID:

apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceBinding
metadata:
  name: gcp-pubsub
  namespace: pubsub-app
spec:
  instanceRef:
    name: gcp-pubsub
  parameters:
    serviceAccount: pubsub-sa
    roles:
    - roles/pubsub.publisher
    - roles/pubsub.viewer

Bind to the Pub/Sub service instance:

kubectl create -f pubsub_binding.yaml

Check the status of the service binding by running the following command:

kubectl get servicebindings gcp-pubsub -n [NAMESPACE] -o 'custom-columns=BINDING-NAME:.metadata.name,SERVICE-INSTANCE:.spec.instanceRef.name,STATUS:.status.conditions[0].reason,OUTPUT-SECRET:.spec.secretName'

It should display a InjectedBindResult status.

The gcp-pubsub secret created as a result of binding contains the project (projectId), service account (serviceAccount), and topic ID (topicId). You can check the secret by running the following command:

kubectl get secrets gcp-pubsub --namespace [NAMESPACE] -o yaml

Step 4: Create a Pub/Sub subscription to the topic

Like binding to the Pub/Sub service instance for publishing to the topic, the demo application also needs a service binding in order to subscribe to and read messages from the topic.

Now we will create another service binding to the Pub/Sub service instance, this time of type subscriber, which contains a subscription and two roles: roles/pubsub.subscriber and roles/pubsub.viewer. This will create a Pub/Sub subscription to this topic and allow our service account to pull and view messages from this subscription.

svcat

Use the svcat command to bind to the Pub/Sub topic as a subscriber:

svcat bind gcp-pubsub \
    --name gcp-pubsub-subscription \
    --namespace [NAMESPACE] \
    --params-json '{
      "serviceAccount": "'${SERVICE_ACCOUNT_ID}'",
      "roles": [
        "roles/pubsub.subscriber",
        "roles/pubsub.viewer"
      ],
      "subscription": {
        "subscriptionId": "pubsub-app"
      }
    }'

The command should output the following message (serviceAccount value may be the name of the service account you created before): 

  Name:        gcp-pubsub-subscription
  Namespace:   pubsub-app
  Status:
  Instance:    gcp-pubsub



Parameters:
  roles:



    

  • roles/pubsub.subscriber
    • 

  • roles/pubsub.viewer
serviceAccount: pubsub-sa
subscription:
subscriptionId: pubsub-app

Check the status of the service binding with the following command:

svcat get binding gcp-pubsub-subscription --namespace [NAMESPACE]

The output should appear similar to the following:

           NAME             NAMESPACE     INSTANCE    STATUS
+-------------------------+------------+------------+--------+
  gcp-pubsub-subscription   pubsub-app   gcp-pubsub   Ready

Wait until the service binding's status is Ready before continuing. This could take a minute.

kubectl

Create a ServiceBinding YAML file called pubsub_subscription_binding.yaml that binds to the Pub/Sub topic as a subscriber. Make sure the value for serviceAccount field is the same as echo $SERVICE_ACCOUNT_ID:

apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceBinding
metadata:
  name: gcp-pubsub-subscription
  namespace: pubsub-app
spec:
  instanceRef:
    name: gcp-pubsub
  parameters:
    serviceAccount: pubsub-sa
    roles:
    - roles/pubsub.subscriber
    - roles/pubsub.viewer
    subscription:
      subscriptionId: pubsub-app

Bind to the Pub/Sub service instance:

kubectl create -f pubsub_subscription_binding.yaml

Check the status of the service binding by running the following command:

kubectl get servicebindings gcp-pubsub-subscription -n [NAMESPACE] -o 'custom-columns=BINDING-NAME:.metadata.name,SERVICE-INSTANCE:.spec.instanceRef.name,STATUS:.status.conditions[0].reason,OUTPUT-SECRET:.spec.secretName'

It should display a InjectedBindResult status.

The new subscription projects/[PROJECT_ID]/subscriptions/pubsub-app can be found in the Cloud Console. The gcp-pubsub-subscription secret created as a result of binding contains the project (projectId), service account(serviceAccount), topic ID (topicId), and subscription ID (subscriptionId). You can check the secret by running the following command:

kubectl get secrets gcp-pubsub-subscription --namespace [NAMESPACE] -o yaml

Step 5: Test the Pub/Sub service with a demo app

Run the demo app in your GKE cluster to test the connection with the Pub/Sub service.

Create a Kubernetes job file called hello_pubsub_world.yaml with the following spec (replace the value of secretName to [SA_SECRET_NAME] if you manually created the secret to store the service account private key):

apiVersion: batch/v1
kind: Job
metadata:
  name: hello-pubsub-world
  namespace: pubsub-app
spec:
  template:
    spec:
      volumes:
        # Make the 'pubsub-credentials' secret available as volume
        # 'google-cloud-key'.
        - name: google-cloud-key
          secret:
            secretName: pubsub-credentials
      restartPolicy: OnFailure
      containers:
        - name: hello-pubsub-world
          image: gcr.io/gcp-services/samples/hello-pubsub-world
          volumeMounts:
            # Mount the 'google-cloud-key' volume into the container file
            # system.
            - name: google-cloud-key
              mountPath: /var/secrets/google
          env:
            # Pass the path to the private key JSON file from the mounted volume
            # to the environment variable.
            - name: "GOOGLE_APPLICATION_CREDENTIALS"
              value: /var/secrets/google/privateKeyData

            # The two environment variables below come from the 'gcp-pubsub'
            # secret and, together, point at the Cloud Pub/Sub topic to use.
            - name: "GOOGLE_CLOUD_PROJECT_ID"
              valueFrom:
                secretKeyRef:
                  # Use the projectId value from the 'gcp-pubsub' secret created
                  # as a result of binding to the Pub/Sub service instance.
                  name: gcp-pubsub
                  key: projectId
            - name: "GOOGLE_CLOUD_PUBSUB_TOPIC"
              valueFrom:
                secretKeyRef:
                  # Use the topicId value from the 'gcp-pubsub' secret created
                  # as a result of binding to the Pub/Sub service instance.
                  name: gcp-pubsub
                  key: topicId

            # The environment variable below come from the
            # 'gcp-pubsub-subscription' secret and, together with the
            # GOOGLE_CLOUD_PROJECT_ID above, point at the Cloud Pub/Sub
            # subscription to use.
            - name: "GOOGLE_CLOUD_PUBSUB_SUBSCRIPTION"
              valueFrom:
                secretKeyRef:
                  # Use the subscriptionId value from the
                  # 'gcp-pubsub-subscription' secret created as a result of the
                  # subscription binding to the Pub/Sub service instance.
                  name: gcp-pubsub-subscription
                  key: subscriptionId

Note how it uses values from the [SA_SECRET_NAME] (should be pubsub-credentials if you created a new Cloud IAM service account), gcp-pubsub, and gcp-pubsub-subscription secrets, which were created via bindings to the Cloud IAM service account and Pub/Sub service instances.

Create the job:

kubectl apply -f hello_pubsub_world.yaml

The command should output the following message:

job "hello-pubsub-world" created

View the list of pods:

kubectl get pods --namespace [NAMESPACE] --show-all

It should display the details of the pod running in that namespace, similar to the following:

NAME                       READY     STATUS      RESTARTS   AGE
hello-pubsub-world-zh8hm   0/1       Completed   0          1m

Wait for the pod to have a status of Completed, and then check the logs of that pod with the following command:

kubectl -n [NAMESPACE] logs [POD_NAME]

You should see lines that contain entries similar to:

Publishing 10 messages.
Publishing done.
Subscribing 10 messages.
Got message: "hello world #9"
Got message: "hello world #0"
Got message: "hello world #1"
Got message: "hello world #2"
Got message: "hello world #3"
Got message: "hello world #4"
Got message: "hello world #5"
Got message: "hello world #6"
Got message: "hello world #7"
Got message: "hello world #8"
Subscription done.

Cleaning up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:

  1. Delete the demo namespace, which will delete the app, service bindings, and service instances:

    kubectl delete namespace [NAMESPACE]

  2. For more information about cleaning up Service Catalog, please refer to this guide

What's next