Partnering on open source: Managing Google Cloud Platform with Chef
Managing cloud resources is a critical part of the application lifecycle. That’s why today, we released and open sourced a set of comprehensive cookbooks for Chef users to manage Google Cloud Platform (GCP) resources.
Chef is a continuous automation platform powered by an awesome community. Together, Chef and GCP enable you to drive continuous automation across infrastructure, compliance and applications.
The new cookbooks allow you to define an entire GCP infrastructure using Chef recipes. The Chef server then creates the infrastructure, enforces it, and ensures it stays in compliance. The cookbooks are idempotent, meaning you can reapply them when changes are required and still achieve the same result.
The new cookbooks support the following products:
- Google Container Engine: install / docs | source
- Google Compute Engine: install / docs | source
- Google Cloud SQL: install / docs | source
- Google Cloud DNS: install / docs | source
- Google Cloud Storage: install / docs | source
We also released a unified authentication cookbook that provides a single authentication mechanism for all the cookbooks.
We tested the cookbooks on CentOS, Debian, Ubuntu, Windows and other operating systems. Refer to the operating system support matrix for compatibility details. The cookbooks work with Chef Client, Chef Server, Chef Solo, Chef Zero, and Chef Automate.
To learn more about these Chef cookbooks, register for the webinar with myself and Chef’s JJ Asghar on 15 October 2017.
Getting started with Chef on GCPUsing these new cookbooks is as easy as following these four steps:
- Install the cookbooks.
- Get a service account with privileges for the GCP resources that you want to manage and enable the the APIs for each of the GCP services you will use.
- Describe your GCP infrastructure in Chef:
- Define a gauth_credential resource
- Define your GCP infrastructure
- Run Chef to apply the recipe.
1. Install the cookbooksYou can find all the GCP cookbooks for Chef on Chef Supermarket. We also provide a “bundle” cookbook that installs every GCP cookbook at once. That way you can choose the granularity of the code you pull into your infrastructure.
Note: These Google cookbooks require neither administrator privileges nor special privileges/scopes on the machines that Chef runs on. You can install the cookbooks either as a regular user on the machine that will execute the recipe, or on your Chef server; the latter option distributes the cookbooks to all clients.
The authentication cookbook requires a few of our gems. You can install them using various methods, including using Chef itself:
For more details on how to install the gems, please visit the authentication cookbook documentation.
Now, you can go ahead and install the Chef cookbooks. Here’s how to install them all with a single command:
knife cookbook site install google-cloud
Or, you can install only the cookbooks for select products:
knife cookbook site install google-gcompute # Google Compute Engine
knife cookbook site install google-gcontainer # Google Container Engine
knife cookbook site install google-gdns # Google Cloud DNS
knife cookbook site install google-gsql # Google Cloud SQL
knife cookbook site install google-gstorage # Google Cloud Storage
2. Get your service account credentials and enable APIsTo ensure maximum flexibility and portability, you must authenticate and authorize GCP resources using service account credentials. Using service accounts allows you to restrict the privileges to the minimum necessary to perform the job.
Note: Because service accounts are portable, you don’t need to run Chef inside GCP. Our cookbooks run on any computer with internet access, including other cloud providers. You might, for example, execute deployments from within a CI/CD system pipeline such as Travis or Jenkins, or from your own development machine.
Also make sure to enable the the APIs for each of the GCP services you intend to use.
3a. Define your authentication mechanismOnce you have your service account, add the following resource block to your recipe to begin authenticating with it. The resource name, here 'mycred' is referenced in the objects in the credential parameter.
gauth_credential 'mycred' do
For further details on how to setup or customize authentication visit the Google Authentication cookbook documentation.
3b. Define your resourcesYou can manage any resource for which we provide a type. The example below creates an SQL instance and database in Cloud SQL. For the full list of resources that you can manage, please refer to the respective cookbook documentation link or to this aggregate summary view.
gsql_instance ‘my-app-sql-server’ do
gsql_database 'webstore' do
Note that the above code has to be described in a recipe within a cookbook. We recommend you have a “profile” wrapper cookbook that describes your infrastructure, and reference the Google cookbooks as a dependency.
4. Apply your recipeNext, we direct Chef to enforce the recipe in the “profile” cookbook. For example:
$ chef-client -z --runlist ‘recipe[mycloud::myapp]’
In this example,
mycloud is the “profile” cookbook, and
myapp is the recipe that contains the GCP resource declarations.
Please note that you can apply the recipe from anywhere that Chef can execute recipes (client, server, automation), once or multiple times, or periodically in the background using an agent.