This example uses Cloud Pub/Sub, although the instructions can be applied to any Cloud Platform service. The example application in this tutorial authenticates to Cloud Pub/Sub using a service account and subscribes to messages published to a Pub/Sub topic from a Python-based application.
This tutorial covers the following steps:
- How to create a service account
- How to assign necessary roles for your service account to work with Cloud Pub/Sub
- How to save the account key as a Kubernetes Secret
- How to use the service account to configure and deploy an application.
The sample application used in this tutorial subscribes to a Pub/Sub Topic and
prints the messages published to the standard output. You must configure the
application with correct permissions, use
gcloud to publish messages and
inspect the container’s output stream to observe the messages are received
Why use Service Accounts?
Each node in a container cluster is a Compute Engine instance. Therefore, applications running on a container cluster by default inherit the scopes of the Compute Engine instances to which they are deployed.
Google Cloud Platform automatically creates a service account named “Compute Engine default service account” and associates it with any compute instances that are created. Depending on how your project is configured, the default service account may or may not have permissions to use other Cloud Platform APIs. Updating this account’s permissions is not the recommended way to authenticate to other Cloud Platform services from the Kubernetes Engine.
The recommended way to authenticate to Google Cloud Platform services from applications running on Kubernetes Engine is to create your own service accounts. Having service accounts that are distinct from the “Compute Engine default service account” provides the following benefits:
Better visibility into, and auditing of, the API requests your application makes.
The ability to revoke keys for particular applications, instead of sharing a service account and having to revoke API access of all applications at once.
Reduced exposure in case of a potential security incident where the credentials of the service account are compromised.
Before you beginTake the following steps to enable the Google Kubernetes Engine API:
- Visit the Kubernetes Engine page in the Google Cloud Platform Console.
- Create or select a project.
- Wait for the API and related services to be enabled. This can take several minutes.
- Enable billing for your project.
Install the following command-line tools used in this tutorial:
gcloudis used to create and delete Kubernetes Engine clusters.
gcloudis included in the Google Cloud SDK.
kubectlis used to manage Kubernetes, the cluster orchestration system used by Kubernetes Engine. You can install
gcloud components install kubectl
Set defaults for the
To save time typing your project ID and Compute Engine zone
options in the
gcloud command-line tool
gcloudcommand-line tool, you can set default configuration values by running the following commands:
gcloud config set project [PROJECT_ID] gcloud config set compute/zone us-central1-b
Enable Pub/Sub API
For this tutorial, you need to enable the Cloud Pub/Sub API on your project as the sample application uses the Pub/Sub API to receive messages from the Pub/Sub topic:
Create a container cluster
Create a container cluster named
pubsub-test to deploy the Pub/Sub subscriber
gcloud container clusters create pubsub-test
Step 1: Create a Pub/Sub topic
The Pub/Sub subscriber application you will deploy uses a subscription named
echo-read on a Pub/Sub topic called
echo. Create these resources before
deploying the application:
gcloud pubsub topics create echo gcloud pubsub subscriptions create echo-read --topic=echo
Step 2: Deploy Pub/Sub subscriber application
The next step is to deploy the application container to retrieve the messages published to the Pub/Sub topic. This application is written in Python using Google Cloud Pub/Sub client libraries and you can find the source code on GitHub.
The following manifest file describes a Deployment that runs a single instance of this application’s Docker image:
apiVersion: apps/v1beta1 kind: Deployment metadata: name: pubsub spec: template: metadata: labels: app: pubsub spec: containers: - name: subscriber image: gcr.io/google-samples/pubsub-sample:v1
To deploy this manifest, download it to your machine as
pubsub.yaml, and run:
kubectl apply -f pubsub.yaml
Once the application is deployed, query the pods by running:
$ kubectl get pods -l app=pubsub NAME READY STATUS RESTARTS AGE pubsub-2009462906-1l6bh 0/1 CrashLoopBackOff 1 30s
You can see that the container is failing to start and went into a
CrashLoopBackOff state. Inspect the logs from the Pod by running:
$ kubectl logs -l app=pubsub ... google.gax.errors.RetryError: GaxError(Exception occurred in retry method that was not classified as transient, caused by <_Rendezvous of RPC that terminated with (StatusCode.PERMISSION_DENIED, Request had insufficient authentication scopes.)>)
The stack trace and the error message indicates that the application does not have permissions to query the Cloud Pub/Sub service. This is because the “Compute Engine default service account” is not assigned any roles giving it permission to Cloud Pub/Sub.
To avoid using the node instance default service account or sharing a service account in multiple services, you must create a service account specifically for this application.
Step 3: Create service account credentials
To give the applications running on Kubernetes Engine access to Google Cloud Platform services, you need to use service accounts.
To create service account, go to Service Accounts on GCP Console and click Create Service Account:
- Specify a Service Account Name (for example,
- In the Role dropdown, select “Pub/Sub → Subscriber”.
- Check Furnish a new private key and choose key type as JSON.
- Click Create.
Once the service account is created, a JSON key file containing the credentials of the service account is downloaded to your computer. You will use this key file to configure the application to authenticate to the Cloud Pub/Sub API.
Step 4: Import credentials as a Secret
Kubernetes offers the Secret resource type to store credentials inside the container cluster and use them in the applications deployed on Kubernetes Engine directly.
To save the JSON key file a Secret named
pubsub-key, run the following command
with the path to the downloaded service account credentials file:
kubectl create secret generic pubsub-key --from-file=key.json=<PATH-TO-KEY-FILE>.json
This command creates a Secret named
pubsub-key that has a
key.json file with
the contents of the private key you downloaded from GCP Console. Once
you create the Secret, you should remove the key file from your computer.
Step 5: Configure the application with the Secret
To use the
pubsub-key Secret in your application, you need to modify the
Deployment specification to:
- Define a volume with the secret.
- Mount the secret volume to the application container.
- Set the
GOOGLE_APPLICATION_CREDENTIALSenvironment variable to point to the key file in the secret volume mount.
The updated manifest file looks like the following:
apiVersion: apps/v1beta1 kind: Deployment metadata: name: pubsub spec: template: metadata: labels: app: pubsub spec: volumes: - name: google-cloud-key secret: secretName: pubsub-key containers: - name: subscriber image: gcr.io/google-samples/pubsub-sample:v1 volumeMounts: - name: google-cloud-key mountPath: /var/secrets/google env: - name: GOOGLE_APPLICATION_CREDENTIALS value: /var/secrets/google/key.json
This manifest file defines the following to make the credentials available to the application:
a volume named
google-cloud-keywhich uses the Secret named
a volume-mount, making the
/var/secrets/googledirectory inside the container
GOOGLE_APPLICATION_CREDENTIALSenvironment variable set as
/var/secrets/google/key.json, which will contain the credentials file when the secret is mounted to the container as a volume
Note that the
GOOGLE_APPLICATION_CREDENTIALS environment variable is
automatically recognized by Google Cloud client libraries, in this case
the Cloud Pub/Sub client for Python.
To deploy this manifest, download it to your machine as
pubsub-with-secret.yaml, and run:
kubectl apply -f pubsub-with-secret.yaml
Once it is deployed correctly, the Pod status should be listed as
$ kubectl get pods -l app=pubsub NAME READY STATUS RESTARTS AGE pubsub-652482369-2d6h2 1/1 Running 10 29m
Step 6: Test receiving Pub/Sub messages
Now that you configured the application, publish a message to the Pub/Sub Topic
gcloud pubsub topics publish echo --message="Hello, world!"
Within a few seconds, the message should be picked up by the application and printed to the output stream. To inspect the logs from the deployed Pod, run:
$ kubectl logs -l app=pubsub Pulling messages from Pub/Sub subscription... [2017-06-19 12:31:42.501123] ID=130941112144812 Data=Hello, world!
You have successfully configured an application on Kubernetes Engine to authenticate to Pub/Sub API using service account credentials!
To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:
Clean up the Pub/Sub subscription and topic:
gcloud pubsub subscriptions delete echo-read gcloud pubsub topics delete echo
Delete the container cluster:
gcloud container clusters delete pubsub-test