Edit on GitHub
Report issue
Page history

Author(s): @{[ username ]}  Published: {[ TutorialCtrl.tutorial.date_published | date:'mediumDate' ]}

title: Pub/Sub API Logging for Preemptible VM Shutdowns description: Send notifications of pVM shutdowns to Pub/Sub for logging. author: mkahn5 tags: PubSub, Compute Engine date_published: 2017-10-03


Pub/Sub API logging for Preemptible VM (pVM) shutdowns

This tutorial demonstrates sending notifications from Preemptible Virtual Machines (pVMS) to Goolge Cloud Pub/Sub for logging purposes.

Creating the instance

To get started, we'll use a regular instance and will configure the Python libraries and the script to make sure it works as expected. We'll then copy the disk to an image for use in pVMs that we create in the future.

Creating the Pub/Sub topic via Google Cloud Shell

Next, create a topic called shutdown-log to hold all of the shutdown timestamps for the pVMs:

mikekahn@mikekahn-sandbox:~$ gcloud beta pubsub topics create shutdown-log
Created topic [shutdown-log].mikekahn@mikekahn-sandbox:~$

Creating a device credential

To enable auth for the Pub/Sub API, click the IAM & admin section of the console. Then, in the Service accounts section, click Create service account.

Pubsub Role

Give the service account a Pub/Sub role. If this script is just publishing, you can just give it the Publisher role. If you plan to manage subscriptions from the instance, then give it Admin or a higher privileged role.

Be sure to download a new JSON key. Now the device has a credential in the form of a file. Copy the JSON key on your instance. Copy and paste the JSON from the key exported in the last step, or move it over via SCP to your instance.

Using the credential on the device

We are going to use a Google client library to talk to Pub/Sub. In order for this client library to find and use the credential file we copied it will look for an environment variable which we set on the device. Add the variable to bashrc so it loads on its own (debian).

mikekahn@instance-1:/$ echo "export GOOGLE_APPLICATION_CREDENTIALS=/shutdown-log-key.json" >> ~/.bashrc

We are now good to go with the Pub/Sub API. Next, make sure you have the most updated Google Cloud Python library via pip and it works:

mikekahn@instance-1:/$ sudo wget https://bootstrap.pypa.io/get-pip.py
mikekahn@instance-1:/$ sudo python get-pip.py
Collecting pip
Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)    100% |████████████████████████████████| 1.3MB 1.0MB/s
Collecting setuptools  
Downloading setuptools-36.5.0-py2.py3-none-any.whl (478kB)    100% |████████████████████████████████| 481kB 2.4MB/s
Collecting wheel  
Downloading wheel-0.30.0-py2.py3-none-any.whl (49kB)    100% |████████████████████████████████| 51kB 9.8MB/s
Installing collected packages: pip, setuptools, wheel
Successfully installed pip-9.0.1 setuptools-36.5.0 wheel-0.30.0
mikekahn@instance-1:/$ sudo apt-get install python-dev
mikekahn@instance-1:/$ sudo pip install --upgrade google-cloud
Successfully installed dill-0.2.7.1 future-0.16.0
gapic-google-cloud-datastore-v1-0.15.3
gapic-google-cloud-error-reporting-v1beta1-0.15.3
gapic-google-cloud-logging-v2-0.91.3
gapic-google-cloud-pubsub-v1-0.15.4
gapic-google-cloud-spanner-admin-database-v1-0.15.3
gapic-google-cloud-spanner-admin-instance-v1-0.15.3
gapic-google-cloud-spanner-v1-0.15.3
google-cloud-0.27.0 google-cloud-bigquery-0.26.0 google-cloud-bigtable-0.26.0 google-cloud-core-0.26.0
google-cloud-datastore-1.2.0 google-cloud-dns-0.26.0 google-cloud-error-reporting-0.26.0 google-cloud-language-0.27.0 google
cloud-logging-1.2.0 google-cloud-monitoring-0.26.0 google-cloud-pubsub-0.27.0 google-cloud-resource-manager-0.26.0 google
cloud-runtimeconfig-0.26.0 google-cloud-spanner-0.26.0 google-cloud-speech-0.28.0 google-cloud-storage-1.3.2 google-cloud
translate-1.1.0 google-cloud-videointelligence-0.25.0 google-cloud-vision-0.26.0 google-gax-0.15.15 google-resumable-media
0.2.3 grpc-google-iam-v1-0.11.4 httplib2-0.10.3 monotonic-1.3 oauth2client-3.0.0 ply-3.8 proto-google-cloud-datastore-v1
0.90.4 proto-google-cloud-error-reporting-v1beta1-0.15.3 proto-google-cloud-logging-v2-0.91.3 proto-google-cloud-pubsub-v1
0.15.4 proto-google-cloud-spanner-admin-database-v1-0.15.3 proto-google-cloud-spanner-admin-instance-v1-0.15.3 proto-google
cloud-spanner-v1-0.15.3 tenacity-4.4.0
mikekahn@instance-1:/$
mikekahn@instance-1:/$ sudo pip install --upgrade google-cloud-pubsub

At the time of writing this article I had an issue with the Python pubsub module 0.27.1. If you are having issues, try using 0.28.3, as it worked fine for me.

mikekahn@instance-1:/$ sudo pip install google-cloud-pubsub==0.28.3

Pub/Sub publish script

import socket
import os
import time

from google.cloud import pubsub_v1
from datetime import datetime

from subprocess import check_output

publisher = pubsub_v1.PublisherClient()

hostname = socket.gethostname()
ips = check_output(['hostname', '--all-ip-addresses'])
timestamp = time.strftime("%c")

topic = 'projects/{project_id}/topics/{topic}'.format(
    project_id='mikekahn-sandbox',
    topic='shutdown-log',  # Set this to something appropriate.
)
publisher.publish(topic, timestamp, ips=ips)

Very simply, this script publishes a message to the pubsub topic shutdown-log with a timestamp and ip address of the server. IPs can be interchangeable with hostname in the publisher statement. Replace the project_id and topic variables if you use the above script. Now run the script and check the topic to see if your message came through.

Check the topic for your messages

mikekahn@mikekahn-sandbox:~$ gcloud beta pubsub subscriptions list
┌──────────────────┬──────────────┬──────────────┬──────┬──────────────┐
│     PROJECT      │ SUBSCRIPTION │    TOPIC     │ TYPE │ ACK_DEADLINE
│├──────────────────┼──────────────┼──────────────┼──────┼──────────────┤
│ mikekahn-sandbox │ shutdown-log │ shutdown-log │ PULL │ 10
│└──────────────────┴──────────────┴──────────────┴──────┴──────────────┘
mikekahn@mikekahn-sandbox:~$

mikekahn@mikekahn-sandbox:~$ gcloud beta pubsub subscriptions pull projects/mikekahn-sandbox/subscriptions/shutdown-log
┌──────────────────────────┬─────────────────┬──────────────────┬────────────────────────────────────────────────────────────
───────────────────────────────────────────────────────────────────────────────────────────────────────┐
│           DATA           │    MESSAGE_ID   │    ATTRIBUTES    │  ACK_ID
│├──────────────────────────┼─────────────────┼──────────────────┼───────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────────────────────────────────────────
┤│ Mon Sep 25 23:50:57 2017 │ 155794142007574 │ ips=10.128.0.3   │
XkASTCcYRElTK0MLKlgRTgQhIT4wPkVTRFAGFixdRkhRNxkIaFEOT14jPzUgKEUVCQgUBXx9cEJTdV9UcmhRDRlyfWBwPwgbUwoXB35cURIHaE5tdSVuDBx3emhxa
14RAQJCVnlbc_SJyvQ2ZiU9XxJLLD5-LC1FQQ
│└──────────────────────────┴─────────────────┴──────────────────┴───────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Here our pubsub message shows the timestamp Mon Sept 25… and the instance IP 10.128.0.3. Great, everything works! So now let's move this over to a pVM and test once more.

Configure shutdown script

You can invoke a shutdown script directly or provide a shutdown script file for instances on GCE. In this case the script is on the image so I am providing contents directly. Next step (not shown) I saved the python pubsub api message script above, configured tested python library and the json API key to a compute image called shutdown-log. So now with my image created I can start building pVMs ready to notify me when they go offline.

Build a pVM with the shutdown script

mikekahn@mikekahn-sandbox:~$ gcloud compute instances create pubsubshutdown6 --preemptible --image shutdown-log --zone us
west1-a --metadata shutdown-script="#! /bin/bash
sudo su -
python /pub-sub-publish.py"

Created [https://www.googleapis.com/compute/v1/projects/mikekahn-sandbox/zones/us-west1-a/instances/pubsubshutdown2].
NAME             ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
pubsubshutdown2  us-west1-a  n1-standard-1  true         10.138.0.4   35.203.140.135  RUNNING
mikekahn@mikekahn-sandbox:~$

Result: pVM Instance Details - Shutdown Script

Verify logging is setup

Now pVM instances built with image that contains the script in this post will publish their IP and timestamp to the Pub/Sub topic as they are shut down. This can be used for logging and notification whenever pVMs are taken offline for infrastructure managers or to trigger other events in application workflow.

mikekahn@mikekahn-sandbox:~$ gcloud beta pubsub subscriptions pull projects/mikekahn-sandbox/subscriptions/shutdown-log
┌──────────────────────────┬─────────────────┬──────────────────┬─────────────────────────────────────────────────────────────
───────────────────────────────────────────────────────────────────────────────────────────────────────┐│   DATA │
MESSAGE_ID   │    ATTRIBUTES    │ ACK_ID
│├──────────────────────────┼─────────────────┼──────────────────┼───────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────────────────────────────────────────┤│ Fri Sep 29 00:53:54
2017 │ 156359192560439 │ ips=10.138.0.4   │
XkASTCcYRElTK0MLKlgRTgQhIT4wPkVTRFAGFixdRkhRNxkIaFEOT14jPzUgKEUaCQgUBXx9cVtedV5ZGgdRDRlyfGQgOQsVUAURUy1VWhENem1cVzhQCB9xeGh0Y
gWBwJBUHd32cmqwsBtZho9XxJLLD5-LC1FQQ
│└──────────────────────────┴─────────────────┴──────────────────┴───────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────────────────────────────────────────┘
See more by @{[ username ]} and more tagged {[ tag ]}{[ $last ? '' : ', ' ]}

Submit a Tutorial

Share step-by-step guides

SUBMIT A TUTORIAL

Request a Tutorial

Ask for community help

SUBMIT A REQUEST

GCP Tutorials

Tutorials published by GCP

VIEW TUTORIALS

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.