Scripting with gcloud: a beginner’s guide to automating GCP tasks

If you work with Google Cloud Platform (GCP), chances are you’ve probably used either the web-based Console or the command-line based Cloud SDK to manage resources and applications hosted on GCP. If you’ve downloaded Cloud SDK, which comes with the gcloud command-line tool, you might consider scripting with gcloud to get more out of your GCP usage!

Scripting with  gcloud commands lets you automate any number of GCP tasks—deploying code to App Engine, analyzing log data, and updating a Compute Engine network, among other things. And with the help of filter and format flags, you can structure your output from the gcloud command-line tool (and not rely on messages to stdout/stderr that can change in future releases!), easily extract information, and combine multiple commands to manage your GCP resources and projects non-interactively.

In this post, we’ll create a few simple scripts that utilize gcloud’s functionality, to create a collection of automated, predefined, easily reusable GCP tasks.

Starting with scripts (with a little help from --filter & --format)

To get started, let’s write a script to automatically delete terminated VMs.

First, run gcloud compute instances list --format=text --limit=1 to get a sense of the fields you can filter by to extract the status of an available VM instance. A quick look confirms there’s a helpfully named field, status.

Now, we just need to iterate through all these instances and delete them. gcloud compute instances delete should do the trick. Note that this command takes a zone flag; we’ll need to extract this information from the terminated VM instance before we can delete it. We could use another command-line utility like sed or a structured query tool such as jq but the gcloud CLI tool provides a simple but powerful built-in language optimized for this kind of need: the --format flag. We can retrieve just the zone field alone by running:

Then, to delete an instance without prompting, we’ll need something along the lines of:

Where $instance is the instance name of a terminated VM instance and $zone is its respective zone.

Scripting without supervision

This script is written to work in “unattended mode”, i.e.,  it works without a user at the terminal by including the --quiet flag. In addition to reducing the amount of output to standard error, this flag bypasses all interactive prompting, in which case, default answers for interactive questions and default values of flags (or their underlying properties) are used. Note that if a required flag isn’t set and it doesn’t have a default value, the command will fail with errors. Using --quiet also means, in this scenario, that you won’t get asked for confirmation before terminated VMs are deleted—a destructive action that cannot be undone. Use --quiet and its equivalent environment variable, CLOUDSDK_CORE_DISABLE_PROMPTS, wisely!

Putting it all together

Now that we’ve got the basics out of the way, we can construct a small but mighty script to automatically delete terminated VMs.

We can also choose to construct this script to get attributes from the list command in one go:

We can crunch this down into a one-liner with some gcloud command nesting (and without anything being written to stdout):

Alternatively, the following command accomplishes the same without using a pipe:

More scripting, more power

Now that you’ve gotten a taste of using gcloud commands in a straightforward script, let’s try something a little more ambitious—let’s restart VMs of a particular zone that have memory usage numbers above a certain threshold. We’ll need to execute four actions: list all VM instances in a specified zone, ssh into each one to get memory usage data, stop each instance and bring them up again. The right candidates for these jobs should be gcloud compute instances list (with a handy combination of  filter and format), gcloud compute ssh, gcloud compute instances stop, and gcloud compute instances start.

Like the previous script, we need to iterate through a list of viable VM instances:

For each instance, we have to run free -m and pull out information about how much free space the instance has left and check if it’s less than our threshold:

And if it has insufficient free space left, all that’s left to do is to simply stop and start the instance:

Let’s tie that all together into a working script:

And there you have it—a simple script that restarts VMs with memory usage numbers above a specified threshold.

Now that you’ve got the basics of scripting with gcloud down, you can accomplish all sorts of things: create a large number of Google Cloud projects for many people, move a file from Cloud Storage to a specified location on your filesystem, or execute a gcloud command to run on all your VMs and collect the output in a common file. All you’ll need is a couple of gcloud commands, the --quiet flag and when in doubt, --help.

Additional resources

For an interactive guide on using --filter and --format flags, launch this tutorial. For more on scripting with gcloud, give the Scripting gcloud commands page a read.