This solution describes best practices for running Spinnaker on Google Compute Engine, such as deploying, securing and maintaining your Spinnaker deployment.
When you need to set up a continuous delivery pipeline, Spinnaker provides the following benefits:
Spinnaker is built on the principle of immutable infrastructure, providing a foundation for fast, reliable deployments of cloud-native apps.
Deployments are predictable, safe, and can be easily rolled back when needed.
You can create complex deployment pipelines that span multiple development environments such as testing, QA, and production.
Spinnaker is an open source tool for orchestrating continuous delivery pipelines that deploy software to cloud resources. The codebase is written in Java and Groovy, and leverages the Spring Boot framework. Spinnaker provides an intuitive user interface, and a flexible REST API for fine-grained control over tasks in each pipeline stage and the dependencies between stages. With Spinnaker, it's easy to map your software release process to a set of discrete steps that reliably delivers your software to end users.
The following image describes Spinnaker's release process.
Software is built, then tested. If all tests pass, an immutable image is "baked" and made available in the cloud. After the image is available it can be deployed to sets of instances to update their running software.
Spinnaker is an application stack that includes microservice components, each
handling a discrete piece of functionality. Each component generates logs in a
separate directory in the top-level
Spinnaker's component configuration files
exist in the
The following components interact directly with Google Cloud Platform (GCP):
The Clouddriver component interacts with the Compute Engine API to deploy applications and update their load balancers.
The Front50 component handles persistent storage for Spinnaker-specific configuration and metadata such as pipeline definitions. Configuration data is stored in Google Cloud Storage.
The Rosco component builds Compute Engine images using Packer during the bake stage of a pipeline.
For a full list of Spinnaker components, see the Spinnaker documentation.
You can deploy the following architecture by using the Spinnaker Deployment Manager templates.
Each major component such as Spinnaker, Jenkins, and Redis runs on its own virtual machine for independent scaling. Each virtual machine runs in its own instance group to ensure that at least one instance runs at all times. Using this setup ensures that if a machine fails or terminates, the Compute Engine instance group manager brings up another instance in its place.
A distinct deployment network and subnetwork isolates Spinnaker from the rest of the project’s infrastructure. Static IP addresses keep each component at a well-known network location so that other components can access them reliably. The network firewall restricts access except for traffic that Spinnaker needs to operate, and for administrators to manage Spinnaker. Inside Spinnaker’s network, only the Spinnaker instance can access Jenkins and Redis.
The architecture uses Cloud Storage to store Spinnaker’s configuration settings, because Cloud Storage is highly available and provides data replication.
Each Spinnaker component has a corresponding configuration file that customizes
its functionality. For example, you can configure the orchestration component
run a generic script stage in Jenkins,
which allows you to have a
database migration stage in your pipeline that is defined by a Bash script
within your source code repository. After installing Spinnaker, you can find
these configuration files in the
spinnaker-local.yml file unifies common parameters between components so
that common values can be reused by different components.
The following diagram illustrates the hierarchy of Spinnaker's configuration files.
You can set environment variables in the
/etc/default/spinnaker directory for
certain configuration options in the
spinnaker-local.yml file. To see which
options can be set using environment variables, look for variables represented
by the following notation.
If you don't override these variables,
default-value is used.
Create environment variables as key-value pairs separated by an equals sign, as shown in the following examples.
SPINNAKER_GOOGLE_ENABLED=true SPINNAKER_GOOGLE_PROJECT_ID=myproject-name SPINNAKER_DEFAULT_STORAGE_BUCKET=my-spinnaker-bucket SPINNAKER_GOOGLE_DEFAULT_REGION=us-west1 SPINNAKER_GOOGLE_DEFAULT_ZONE=us-west1-a
Configuring image creation
One of Spinnaker’s critical use cases is orchestrating pipelines that deploy immutable images to VM clusters. Immutable images provide several benefits.
You can roll back the images to specific, previous releases.
Scaling an existing version produces an identical copy of the application.
Your application can boot faster because of minimal startup steps.
Spinnaker leverages Packer to create
immutable images. The Rosco
component manages the image build process. Rosco is configured with a default
Packer configuration template,
located in the
/opt/rosco/config/packer/gce.json file. This configuration can
be overridden when creating pipelines by configuring an explicit configuration
file. If you'd like to change the default template, you can either update the
gce.json file, or
create a configuration file
and specify it when configuring the bake stage of your pipeline.
Spinnaker uses the Igor component as its interface with continuous integration (CI) systems such as Jenkins and Travis CI, and Git repositories such as Github and Stash. The component handles polling pipeline triggers, and triggering external jobs as part of a pipeline. Igor can be configured to interface with multiple independent CI systems for distinct build and integration testing. For pipelines to be triggered by a repository push, Igor must be configured to poll a CI job that in turn polls the Git repository, or must be configured to poll Github or Stash directly.
Spinnaker can send notifications based on the triggering or results of a pipeline stage. You can configure the echo component to send a notification through email or SMS messages, or to Slack or Hipchat.
Security and authorization
This section describes security and authorization best practices for Spinnaker on GCP.
Authorizing Spinnaker’s access to GCP
You can authorize Spinnaker's access to GCP for deploying to a single or multiple projects.
Deploying to a single project
When deploying to a single project, Spinnaker can use the default service account to get credentials for Compute Engine requests. The Spinnaker VM requires the following IAM roles.
- Service Account Actor
- Compute Instance Admin
- Compute Network Admin
- Compute Storage Admin
- Storage Admin
Deploying to multiple projects
Many customers choose to run one Spinnaker deployment per GCP project, but if this use case doesn’t work for you, you can authenticate the Spinnaker service account for multiple projects. When using the Deployment Manager templates, the Spinnaker VM launches with the default service account that belongs to the project. Complete the following tasks to give Spinnaker access to deploy to multiple projects.
Update the Spinnaker Instance Group to use the same email address as the service account.
For each project you’ll deploy to, authorize the service account with the roles listed in Deploying to a single project.
clouddriver.ymlconfiguration file to have multiple accounts, each mapping to a single project.
For more info on service accounts and GCP, see understanding service accounts.
Authenticating and authorizing users for the UI
You can configure Spinnaker to use Google OAuth to authenticate users. When a user first accesses the UI, they are prompted to authenticate with a Google account, and then forwarded on to the UI. Accounts with 2-factor authentication enabled are required to use both forms of authentication to be granted access to the UI.
G Suite customers can help secure their Spinnaker installation by implementing authorization rules based on the groups in their domain. Spinnaker allows you to define a list of groups that a user must belong to for their modifications to be accepted. By default, all users are allowed to view all resources in the UI. When an account has group membership requirements, only members of that group are permitted to make modifications to resources in that account. For example, you can configure a group of users named "spinnaker-admins" that have the ability to:
- Create and delete pipelines.
- Kick off pipelines.
- Destroy clusters.
Users that have existing identity federation systems can leverage Spinnaker's SAML integration.
Configuring network access
You usually don't need Spinnaker to be accessible from the Internet, because users typically access Spinnaker through your on-premises network. One way to access Spinnaker from your on-premises environment is to set up a Google Cloud VPN instance in the network where you’ve deployed Spinnaker. The following image describes the setup.
Google Cloud VPN provides an encrypted tunnel for traffic between your environment and Compute Engine instances, allowing anyone on your corporate network to access Spinnaker’s private IP address.If you launch Spinnaker without a public IP address, you need to create a NAT gateway that can be used to reach the Internet for repository and dependency downloads.
For situations where only several users interact directly with Spinnaker, you can set up SSH port forwarding from each user’s workstation to the Spinnaker instance. Forward the Deck (9000), Gate (8084), and Rosco (8087) ports to use Spinnaker from a remote machine. For more information about how to set up port forwarding, see port forwarding over SSH.
You can lock down internal network traffic by creating firewall rules that prevent unnecessary access between VMs. Allow traffic only from Spinnaker to Redis, and from Spinnaker to Jenkins. Limit SSH traffic to the Spinnaker VM.
Upgrade Spinnaker regularly to receive the latest feature enhancements and bug fixes. To upgrade your Spinnaker installation to the latest release, run a package upgrade. For example, you can run the following commands to update Ubuntu 14.04.
sudo apt-get update sudo apt-get upgrade spinnaker-clouddriver \ spinnaker-deck spinnaker-echo spinnaker-front50 \ spinnaker-gate spinnaker-igor spinnaker-orca \ spinnaker-rosco spinnaker sudo restart spinnaker
Note that restarting Spinnaker terminates any currently running pipelines, and potentially restarts them when Spinnaker is back up and running.
Backing up and restoring Spinnaker
Spinnaker stores important data such as pipeline, application and project configurations in Cloud Storage. This data is highly durable, replicated across regions by default, and encrypted at rest. Spinnaker uses Cloud Storage's versioning capabilities to enable a pipeline revision history. The Spinnaker user interface shows the differences between your pipeline versions, and allows you to restore a previous state.
To backup the data, use the
gcloud sync command to replicate the data to a
bucket in another region. To restore the data, sync the data back to a bucket
in a region where you intend to redeploy Spinnaker.
Spinnaker uses Redis as ephemeral storage to keep track of ongoing pipeline tasks. It isn’t critical to backup this data because pipelines can be restarted if the state is lost. Losing the Redis state can cause pipelines to automatically retrigger. Ensure that your deployment pipelines have gating for important or destructive stages by using manual judgement stages and Spinnaker's built in notifications. You can have multiple Redis instances to provide high availability, in case one host goes down. If you choose to backup this data, you can configure Redis to periodically take state snapshots and transfer the files to Cloud Storage. For more information about Redis persistence and disaster recovery, see Redis Persistence.