Creating a CI/CD pipeline with VSTS and Compute Engine

In this tutorial, you learn how to use Visual Studio Team Services (VSTS) and Compute Engine to create a continuous integration/continuous deployment (CI/CD) pipeline. The tutorial uses Orchard CMS, an open source content management system as a sample application. Orchard CMS is based on ASP.NET MVC and runs on Windows Server 2016.

The CI/CD pipeline uses two separate environments, one for testing and one for production. When a developer commits a change to the Git repository, the commit causes the source code to be built and a new Windows Server 2016–based virtual machine (VM) image to be created by using Packer. The new image is then automatically released to the development environment by using a rolling update. After testing, a release manager can then promote the release so that it's deployed into the production environment.

Conceptual diagram of CI/CD pipeline showing how developers and end users interact with the application

The tutorial assumes that you have basic knowledge of the .NET Framework, Windows Server, Microsoft Internet Information Services (IIS), VSTS, and Compute Engine. The tutorial also requires you to have administrative access to a VSTS account and a Visual Studio 2017 installation that's already connected to your VSTS account.

Objectives

  • Run a private VSTS agent on Compute Engine and connect it to VSTS.
  • Use Packer with Compute Engine to create Windows images.
  • Use Compute Engine Managed Instance Groups to implement rolling deployments.
  • Set up a CI/CD pipeline in VSTS to orchestrate the building, creating, and deployment processes.

Costs

This tutorial uses billable components of Google Cloud Platform, including:

Use the Pricing Calculator to generate a cost estimate based on your projected usage. Check the VSTS pricing page for any fees that might apply for using VSTS.

Before you begin

Deploying an application to Google Cloud Platform (GCP) requires a GCP project. It is usually advisable to use separate projects for CI, development, and production workloads so that identity and access management (IAM) roles and permissions can be granted individually.

However, for the sake of simplicity this tutorial uses a single project for the CI, development, and production environments.

  1. Sign in to your Google Account.

    If you don't already have one, sign up for a new account.

  2. Select or create a GCP project.

    Go to the Manage resources page

  3. Make sure that billing is enabled for your project.

    Learn how to enable billing

  4. Make sure you have a VSTS account and have administrator access to it. If you don't yet have a VSTS account, you can sign up on the VSTS home page.
  5. Make sure you have Visual Studio 2017 installed and that it's connected to your VSTS account.

Creating a VSTS project

Using Orchard CMS as an example, you will use VSTS to manage the source code, run builds and tests, and orchestrate the deployment to Compute Engine.

To begin, create a new project in your VSTS account:

  1. Go to the VSTS home page (https://[YOUR_VSTS_ACCOUNT_NAME].visualstudio.com/_projects).
  2. Click New Project, or Create Project if you're using the new navigation UI.
  3. Enter a project name, such as Orchard.
  4. Choose Git for version control, and then click Create.
  5. After the project has been created, in the top menu, click Code. You're prompted to clone, push, import, or initialize a repository.
  6. Click Import to fork the Orchard CMS repository from GitHub. Set the following values:

    • Source type: Git
    • Clone URL: https://github.com/OrchardCMS/Orchard.git
    • Leave the Requires authorization checkbox unselected.

    Screenshot of the 'Import a Git repository' dialog box

  7. Click Import.

  8. Select Code > Files. You now see the source code of Orchard CMS.

Continuous integration

You can now use VSTS to set up continuous integration. For each commit that you push to the Git repository, VSTS will build the code and publish the resulting build artifact to internal VSTS storage.

Creating a testing branch

To make sure that the instructions in this tutorial work, you need to create a branch that's based on a specific version of the source code. This step will help make sure that future changes to the code on GitHub don't break this tutorial.

  1. In VSTS, select Code > Tags.
  2. In the list of tags, right-click the icon next to 1.10.2.
  3. Select New branch.
  4. In the Name box, enter testing as the branch name, and confirm by clicking Create branch.

    Screenshot of the 'Create a branch' dialog box in VSTS

Creating a build definition

After you create the branch, you can define the build. Because Orchard CMS is an ASP.NET application that's written in Visual Studio, defining the build includes the following steps:

  • Restoring the NuGet package dependencies.
  • Building the solution (src\Orchard.sln).
  • Publishing the build artifacts of the Orchard.Web project.

You can automate these steps as follows:

  1. In VSTS, navigate to Build and Release > Builds, and then click New Definition. (New pipeline in the new navigation UI.)
  2. Select VSTS Git.
  3. In the Default branch for manual and scheduled builds list, select testing, and then click Continue.
  4. Select the ASP.NET template, and then click Apply. Be careful to choose ASP.NET and not ASP.NET Core or ASP.NET Core (.NET Framework).
  5. On the next page, apply the following settings:

    • Name: Orchard-CI
    • Agent queue: Hosted VS2017
    • Path to solution or packages.config: src\Orchard.sln
  6. In the left navigation pane, click Get sources and apply the following settings:

    • Shallow fetch: Enabled
    • Depth: 1

    These settings speed up the process of cloning the repository.

  7. Right-click the Test Assemblies build step, and then click Remove selected task(s). Orchard CMS uses NUnit for unit testing, which requires the use of a custom test adapter. Configuring this adapter is beyond the scope of this tutorial.

  8. Click Publish Artifact and configure the following setting:

    • Path to publish: $(build.artifactstagingdirectory)/Orchard.Web.zip
  9. Click the Triggers tab and select the Enable continuous integration checkbox.

  10. At the top of the screen, click Save & queue.
  11. In the dialog, enter a comment if you want, and then confirm by clicking Save & queue.
  12. Follow the link at the top of the screen (Build #nnn, where nnn is the build number) to track the progress of the build. If the build does not complete within about 2 minutes, check the build output in the Logs tab for any error messages.

The build takes up to seven minutes to complete. At the end of the build, the file Orchard.Web.zip, which contains all files of the web application, will be available in the internal VSTS artifact storage area.

With the source code being built continuously, the next step is to create the VM images automatically. This automation includes the following steps:

  • Launching a new, temporary VM instance that uses the Windows Server 2016 image.
  • Installing and configuring IIS.
  • Deploying Orchard CMS.

You use Packer to automate this process. Packer needs access to the VM instance, but you don't want to expose the instance to the public internet. Therefore, you run Packer on a private VSTS agent within the GCP network.

Deploying a private VSTS agent

Running Packer on a private agent in GCP avoids exposing resources to the public internet and thereby creating a less secure environment. However, it also increases administrative overhead. To minimize this overhead, you deploy the agent in a managed instance group. Using this approach has these advantages:

  • A managed instance group allows you to re-create the VM, install all required components, and register with VSTS automatically.
  • You can apply Windows updates by deleting the VM and letting the managed instance group re-create the environment with the latest, fully patched Windows image as a base.

Follow these steps:

  1. In the GCP Console, switch to your newly created project.
  2. Open Cloud Shell.

    Go to Cloud Shell

  3. To save time, set default values for your project ID and Compute Engine zone:

    gcloud config set project [PROJECT_NAME]
    gcloud config set compute/zone [ZONE]

    Replace [PROJECT_NAME] with the name of your GCP project, and replace [ZONE] with the name of the zone that you're going to use for creating resources. If you are unsure about which zone to pick, use us-central1-a.

    Example:

    gcloud config set project vsts-test-project-12345
    gcloud config set compute/zone us-central1-a

  4. Enable the Compute Engine API:

    gcloud services enable compute.googleapis.com

  5. In VSTS, in the menu at the top of the page, hold the mouse over the gear icon and click Agent Queues. If you're using the new navigation UI, click Project Settings at the bottom of the menu, and then choose Agent queues.

  6. On the menu, click New pool (New queue in the new navigation UI) at the upper left.
  7. Enter Google Cloud as the pool name and click OK. This pool will be the queue for the private build agent that is running on Compute Engine.
  8. In the left pane, select the newly created Google Cloud pool and click Download agent.
  9. Under Download the agent, click Copy to copy the download URL.
  10. In Cloud Shell, initialize an environment variable. For the URL value, paste the URL that you copied in the previous step.

    export VSTS_AGENT_URL=[PASTE URL FROM CLIPBOARD]

  11. Switch back to VSTS, hold the mouse over the user avatar in the upper-right corner of the screen, and then click Security.

  12. Under Personal access tokens, click Add.
  13. Configure the following settings:

    • Description: Google Cloud Agent
    • Authorized Scopes: All scopes
  14. Click Create Token.

  15. Copy the token to the clipboard.
  16. In Cloud Shell, initialize another environment variable by running the following command. For the URL, paste the URL that you just copied.

    export VSTS_TOKEN=[PASTE TOKEN FROM CLIPBOARD]

  17. Initialize another environment variable to contain the VSTS URL. Replace [ACCOUNT] with your account name, as displayed in the address bar in VSTS.

    export VSTS_URL=https://[ACCOUNT].visualstudio.com

  18. Run the following command to create a specialization script. The script downloads and installs the VSTS agent package, Packer, and Cloud SDK, and registers the agent with VSTS.

    cat | envsubst '$VSTS_AGENT_URL $VSTS_TOKEN $VSTS_URL' > specialize.ps1 << 'EOF'
    # Create an installation directory for the VSTS agent
    New-Item -ItemType directory -Path $env:programfiles\vsts-agent
    # Create a work directory for the VSTS agent New-Item -ItemType directory -Path $env:programdata\vsts-agent
    # Download and install the VSTS agent package Invoke-WebRequest ` -Uri "$VSTS_AGENT_URL" ` -OutFile $env:TEMP\vsts-agent.zip Add-Type -AssemblyName System.IO.Compression.FileSystem [System.IO.Compression.ZipFile]::ExtractToDirectory( ` "$env:TEMP\vsts-agent.zip", ` "$env:programfiles\vsts-agent")
    # Download and install Packer [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 Invoke-WebRequest ` -Uri "https://releases.hashicorp.com/packer/1.2.2/packer_1.2.2_windows_amd64.zip" ` -OutFile $env:TEMP\packer.zip [System.IO.Compression.ZipFile]::ExtractToDirectory( ` "$env:TEMP\packer.zip", ` "$env:programfiles\packer")
    # Download and install the Cloud SDK Invoke-WebRequest ` -Uri https://dl.google.com/dl/cloudsdk/channels/rapid/GoogleCloudSDKInstaller.exe ` -OutFile $env:TEMP\cloudsdk.exe Start-Process -Wait $env:TEMP\cloudsdk.exe -arg "/S /noreporting /nostartmenu /nodesktop"
    # Add Packer and the Cloud SDK installation directory to global path [Environment]::SetEnvironmentVariable( ` "Path", $env:Path + ";$env:programfiles\packer;${env:ProgramFiles(x86)}\Google\Cloud SDK\google-cloud-sdk\bin", ` [System.EnvironmentVariableTarget]::Machine)
    # Install gcloud beta commands $env:CLOUDSDK_PYTHON=gcloud components copy-bundled-python Start-Process -Wait gcloud -arg "components install beta --quiet"
    # Configure the VSTS agent & $env:programfiles\vsts-agent\bin\Agent.Listener configure ` --url $VSTS_URL ` --agent "GCE Agent" ` --work $env:programdata\vsts-agent ` --pool "Google Cloud" ` --replace ` --runAsService ` --windowsLogonAccount "NT AUTHORITY\NETWORK SERVICE" ` --auth PAT ` --token $VSTS_TOKEN EOF

  19. Create an instance template for the VSTS agent. Configure the instance template so that the VM instance runs specialize.ps1 as a specialize script during startup.

    gcloud compute instance-templates create vsts-agent \
        --machine-type n1-standard-1 \
        --image-family windows-2016-core \
        --image-project windows-cloud \
        --metadata-from-file sysprep-specialize-script-ps1=specialize.ps1 \
        --scopes "https://www.googleapis.com/auth/compute,https://www.googleapis.com/auth/devstorage.read_write"

  20. Create a managed instance group that's based on this instance template. It might take around three minutes for the new VM instance to start and for the VSTS agent to register.

    gcloud compute instance-groups managed create vsts-agent \
        --template=vsts-agent \
        --size=1

  21. In VSTS, in the menu at the top of the page, hold the mouse over the gear icon and click Agent Queues. If you're using the new navigation UI, go to the VSTS home page, click Admin settings at the bottom of the menu, and select Agent pools.

  22. Select the Google Cloud pool, and then verify that the agent is registered and its state shows as Online.

    Screenshot of the 'Agent Queues' dialog box in VSTS

Troubleshooting

If the agent doesn't register within 10 minutes, there might be an issue. Try the following:

  1. In the GCP Console, navigate to Compute Engine > VM Instances.
  2. Open the details for the VM that has the prefix vsts-agent.
  3. Click Serial port 1 (console).

The console log contains the output of the specialization script, which might help with troubleshooting. Even after the agent shows up in VSTS, it might initially appear as Offline. In that case, wait another two minutes for the status to change to Online.

Testing the setup

If you want, test the resiliency of this setup:

  1. Go to Compute Engine > VM Instances.
  2. Delete the VM that has the prefix vsts-agent.

The managed instance group will immediately spawn a new instance, and a few minutes later, a new agent will be operational again.

Creating VM images

To automate the VM image creation process, you need to create a Packer template. This template is a JSON file that you should keep with the project's source code.

Creating a Packer template

Before you create the template, you need to check out the source code.

  1. In Visual Studio, open Team Explorer.
  2. In the menu, click the Manage Connections icon.
  3. Click Manage Connections > Connect to a Project.

    Screenshot of the 'Connect to a Project' option in the Team Explorer pane of Visual Studio

  4. In the next dialog, select the Orchard Git repository, and then click Clone.

    Screenshot of the 'Orchard' Git repository selected in the 'Connect to a Project' dialog in Visual Studio

  5. Open Team Explorer again.

  6. Click Branches and then expand the remotes/origin folder.
  7. Right-click the testing branch, and then click Checkout.
  8. After the code has been checked out, in Visual Studio, open Solution Explorer.
  9. In the root of the solution, create a new file named packer.json.
  10. Copy the following code into the newly created file, and then save the file:

    {
      "variables": {
        "gcp_project": "",
        "gcp_zone": "",
        "windows_user": "packer_user",
        "windows_password": "Packer123",
        "image_name": "vsts",
        "image_family": "vsts",
        "app_package": ""
      },
      "builders": [
        {
          "type": "googlecompute",
          "project_id": "{{user `gcp_project`}}",
          "source_image_family": "windows-2016",
          "disk_size": "50",
          "instance_name": "{{user `image_name`}}",
          "image_name": "{{user `image_name`}}",
          "image_family": "{{user `image_family`}}",
          "machine_type": "n1-standard-2",
          "communicator": "winrm",
          "winrm_username": "{{user `windows_user`}}",
          "winrm_password": "{{user `windows_password`}}",
          "winrm_insecure": true,
          "winrm_use_ssl": true,
          "winrm_port": 5986,
          "metadata": {
            "windows-startup-script-cmd": "winrm quickconfig -quiet & net user /add {{user `windows_user`}} {{user `windows_password`}} & net localgroup administrators {{user `windows_user`}} /add & winrm set winrm/config/service/auth @{Basic=\"true\"}"
          },
          "zone": "{{user `gcp_zone`}}",
          "use_internal_ip": true,
          "state_timeout": "8m",
          "scopes": [ "https://www.googleapis.com/auth/devstorage.read_only" ]
        }
      ],
      "provisioners": [
        {
          "type": "powershell",
          "inline": [
            "$ErrorActionPreference = \"Stop\"",
    "# Download application package from Cloud Storage", "gsutil cp {{user `app_package`}} $env:TEMP\\app.zip",
    "# Install IIS", "Enable-WindowsOptionalFeature -Online -FeatureName NetFx4Extended-ASPNET45", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-WebServerRole", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-WebServer", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-CommonHttpFeatures", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-HttpErrors", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-HttpRedirect", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-ApplicationDevelopment", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-HealthAndDiagnostics", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-HttpLogging", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-LoggingLibraries", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-RequestMonitor", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-HttpTracing", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-Security", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-RequestFiltering", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-Performance", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-WebServerManagementTools", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-IIS6ManagementCompatibility", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-Metabase", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-DefaultDocument", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-ApplicationInit", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-NetFxExtensibility45", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-ISAPIExtensions", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-ISAPIFilter", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-ASPNET45", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-HttpCompressionStatic",
    "# Extract application package to wwwroot", "New-Item -ItemType directory -Path $env:TEMP\\app", "Add-Type -AssemblyName System.IO.Compression.FileSystem", "[System.IO.Compression.ZipFile]::ExtractToDirectory(\"$env:TEMP\\app.zip\", \"$env:TEMP\\app\")", "Remove-Item $env:TEMP\\app.zip", "Move-Item -Path $(dir -recurse $env:TEMP\\app\\**\\PackageTmp | % { $_.FullName }) -Destination c:\\inetpub\\wwwroot\\app -force",
    "# Configure IIS web application pool and application", "Import-Module WebAdministration", "New-WebAppPool orchard-net4", "Set-ItemProperty IIS:\\AppPools\\orchard-net4 managedRuntimeVersion v4.0", "New-WebApplication -Name Orchard -Site 'Default Web Site' -PhysicalPath c:\\inetpub\\wwwroot\\app -ApplicationPool orchard-net4",
    "# Grant read/execute access to the application pool user", "&icacls C:\\inetpub\\wwwroot\\app\\ /grant \"IIS AppPool\\orchard-net4:(OI)(CI)(RX)\"",
    "# Create data folder and grant write access to the application pool user", "New-Item -ItemType directory -Path C:\\inetpub\\wwwroot\\app\\App_Data\\", "&icacls C:\\inetpub\\wwwroot\\app\\App_Data\\ /grant \"IIS AppPool\\orchard-net4:(OI)(CI)M\"",
    "# Disable searching for Windows updates", "New-ItemProperty -Path HKLM:\\SOFTWARE\\Policies\\Microsoft\\Windows\\WindowsUpdate\\AU -Name NoAutoUpdate -Value 1 -PropertyType DWORD -Force",
    "# Disable provisioning user", "disable-localuser {{user `windows_user`}}",
    "# Generalize the image", "& \"$Env:Programfiles\\Google\\Compute Engine\\sysprep\\gcesysprep\"" ] } ] }

The Packer template has three sections:

  • The variables section defines the variables that are used in the template. The values for these variables will be passed as command-line arguments to Packer.
  • The builders section contains the settings for creating and communicating with the temporary VM instance.
  • The provisioners section contains a sequence of commands that are run on the VM to configure it after it has been created.

The preceding template uses the Packer Google Compute Builder to create a VM instance that runs Windows Server 2016. When you specify the source image for Packer to use, it's important to use an image that already includes the latest security patches. To prevent you from having to change the source_image setting every time that Google releases a new version of the Windows Server 2016 image, the configuration obtains the most recent image from the windows-2016-core image family.

Using a startup script, the template enables Windows Remote Management (WinRM) and creates a temporary user. After the VM is up and running, Packer uses WinRM and this user to run the PowerShell commands that are specified in the provisioners section. Using the use_internal_ip setting, you ensure that WinRM communicates over the local network rather than over the internet, which is possible because the corresponding build step will run on the private build agent that you provisioned earlier. Finally, the builders section also defines an IAM scope for the VM, which permits access to Cloud Storage.

The PowerShell commands that are in the provisioners section configure the VM to run Orchard CMS. This configuration involves the following:

  • Downloading the application package from Cloud Storage.
  • Installing IIS.
  • Creating an IIS application pool and application for Orchard CMS.
  • Extracting the application package to the IIS webroot folder and configuring the access control lists (ACLs) for the folder.
  • Disabling the search for Windows updates, which is unnecessary if you apply updates by rerunning the CI/CD pipeline periodically.
  • Disabling the provisioning user.
  • Generalizing the image by using GCESysprep to ensure that each VM that's created from the image is assigned a unique security identifier.

To commit the file to Git, do the following:

  1. In Visual Studio, open Team Explorer.
  2. Click the Home icon at upper left to switch to the Home view.
  3. Click Changes.
  4. Under Changes, right-click packer.json and then click Stage.
  5. Enter a commit message like Add Packer template.
  6. Click Commit Staged and Push.

Creating a Cloud Storage bucket for build artifacts

Using the File Provisioner in Packer, you can copy files from the machine that's running Packer to the VM. If you use WinRM, this operation can be slow. Therefore, the template that you created in the previous section looks for the Orchard CMS application package on Cloud Storage, which offers substantially better performance.

To create a bucket in Cloud Storage for this purpose, run the following command:

gsutil mb gs://$(gcloud config get-value core/project)-artifacts

If you don't want to keep the artifacts of all builds, you might consider configuring an object lifecycle rule to delete files that are past a certain age.

Extending the build definition

You have now checked the Packer template checked in to Git and created the Cloud Storage bucket. You can now integrate Packer into your VSTS build process.

So far, the VSTS build definition that you've created uses a single phase. In VSTS, all tasks that are part of the same phase run on the same build agent. In your case, that agent is Hosted VS2017. To start using the private build agent that is provisioned in Compute Engine, you must add a second phase to the build definition.

Within a single phase, tasks share a working directory and have access to artifacts from previous tasks. This access is not possible across phases, because to consume artifacts from previous stages, you must publish those artifacts to VSTS in the first phase and then download them in the next phase. Therefore, extend the build definition as follows:

  1. In VSTS, navigate to Build and Release > Builds.
  2. Switch to the Definitions tab and click Orchard-CI.
  3. Click Edit to open the definition.
  4. In the Process pane on the left, click , and select Add agent phase.
  5. Configure the following settings for the new phase. These settings ensure that the phase is not run before the first phase has completed, and let all build steps run on your private build agent.

    • Display name: Create VM image
    • Agent queue: Private > Google Cloud
    • Dependencies: Phase 1
  6. Click + to add a step to the newly added agent phase, select Download Build Artifacts, click Add, and configure the following setting for the new task:

    • Artifact name: drop
  7. Click + to add a step, select Command Line, click Add, and configure the following settings for the new task:

    • Version: 1.x
    • Display name: Publish artifact to Cloud Storage
    • Tool: gsutil
    • Arguments: cp $(System.ArtifactsDirectory)\drop\Orchard.Web.zip gs://$(Packer.Project)-artifacts/Orchard.Web-$(Build.BuildId).zip
  8. Click + to add another step, select Command Line, click Add, and configure the following settings for the new task:

    • Version: 1.x
    • Display name: Create image
    • Tool: packer
    • Arguments: build -var "gcp_project=$(Packer.Project)" -var "gcp_zone=$(Packer.Zone)" -var "image_family=orchard" -var "image_name=orchard-$(Build.BuildId)" -var "app_package=gs://$(Packer.Project)-artifacts/Orchard.Web-$(Build.BuildId).zip" $(Build.SourcesDirectory)/packer.json
  9. Verify that the order of steps is the same as in the following screenshot:

    Screenshot of the 'Phase 1' build tasks in VSTS

  10. Switch to the Variables tab and add the following variables.

    Name Value
    Packer.Project The name of your GCP project.
    Packer.Zone The zone that you specified earlier when you ran gcloud config set compute/zone (for example, us-central1-a)

  11. At the top of the screen, click Save & queue to trigger a new build.

  12. In the dialog, enter a comment if you want, and confirm by clicking Save & queue. Allow 10 to 15 minutes for the build to complete.
  13. In GCP, navigate to Compute Engine > Images and confirm that an image named orchard-N has been created, where N represents the VSTS build ID.

Continuous deployment

Now that VSTS is automatically building your code and creating a new VM image for each commit, you can turn your attention toward deployment.

Configuring the development environment

With Orchard CMS, you can use either SQL Server or an embedded database that stores data locally. For the sake of simplicity, use the default configuration that relies on the embedded database, although it comes with two restrictions:

  • Only a single VM instance can run at a time. Otherwise, users might see different data depending on which VM instance is serving content to them.
  • Any data changes are lost whenever the VM instance is restarted, unless you change the deployment to use Cloud Filestore for data storage. (We do not cover this scenario in the tutorial.)

Before you can configure the steps in VSTS to automate the deployment, you must prepare the development environment. This preparation includes creating a managed instance group that will manage the web server VM instances. It also includes creating an HTTP load balancer.

  1. In Cloud Shell, create an instance template that uses a standard Windows Server 2016 Core image (not one that's been customized). You will use this template only initially, because each build will produce a new template.

    gcloud compute instance-templates create orchard-initial \
        --machine-type n1-standard-2 \
        --image-family windows-2016-core \
        --image-project windows-cloud

  2. Create an HTTP health check. Because Orchard CMS does not have a dedicated health check endpoint, you can query the path /.

    gcloud compute http-health-checks create orchard-dev-http \
        --check-interval=10s --unhealthy-threshold=10 \
        --request-path=/

  3. Create a managed instance group that's based on the initial instance template. For simplicity's sake, the following commands create a zonal managed instance group. However, you can use the same approach for regional managed instance groups that distribute VM instances across more than one zone.

    gcloud beta compute instance-groups managed create orchard-dev \
        --template=orchard-initial \
        --http-health-check=orchard-dev-http \
        --initial-delay=2m \
        --size=1 && \
    gcloud compute instance-groups set-named-ports orchard-dev --named-ports http:80

  4. Create a load balancer backend service that uses the HTTP health check and managed instance group that you created previously:

    gcloud compute backend-services create orchard-dev-backend \
        --http-health-checks orchard-dev-http \
        --port-name http --protocol HTTP --global && \
    gcloud compute backend-services add-backend orchard-dev-backend \
        --instance-group orchard-dev --global \
        --instance-group-zone=$(gcloud config get-value compute/zone)

  5. Create a load balancer frontend:

    gcloud compute url-maps create orchard-dev --default-service orchard-dev-backend && \
    gcloud compute target-http-proxies create orchard-dev-proxy --url-map=orchard-dev && \
    gcloud compute forwarding-rules create orchard-dev-fw-rule --global --target-http-proxy orchard-dev-proxy --ports=80

  6. Create a firewall rule that allows the Google load balancer to send HTTP requests to instances that have been annotated with the gclb-backend tag. You will later apply this tag to the web service VM instances.

    gcloud compute firewall-rules create gclb-backend --source-ranges=130.211.0.0/22,35.191.0.0/16 --target-tags=gclb-backend --allow tcp:80

Configuring the production environment

Setting up the production environment requires a sequence of steps similar to those for configuring the development environment.

  1. In Cloud Shell, create an HTTP health check. Because Orchard CMS does not have a dedicated health check endpoint, you can query the path /.

    gcloud compute http-health-checks create orchard-prod-http \
        --check-interval=10s --unhealthy-threshold=10 \
        --request-path=/

  2. Create another managed instance group that is based on the initial instance template that you created earlier:

    gcloud beta compute instance-groups managed create orchard-prod \
      --template=orchard-initial \
      --http-health-check=orchard-prod-http \
      --initial-delay=2m \
      --size=1 && \
    gcloud compute instance-groups set-named-ports orchard-prod --named-ports http:80

  3. Create a load balancer backend service that uses the HTTP health check and managed instance group that you created previously:

    gcloud compute backend-services create orchard-prod-backend --http-health-checks orchard-prod-http --port-name http --protocol HTTP --global && \
    gcloud compute backend-services add-backend orchard-prod-backend --instance-group orchard-prod --global --instance-group-zone=$(gcloud config get-value compute/zone)

  4. Create a load balancer frontend:

    gcloud compute url-maps create orchard-prod --default-service orchard-prod-backend && \
    gcloud compute target-http-proxies create orchard-prod-proxy --url-map=orchard-prod && \
    gcloud compute forwarding-rules create orchard-prod-fw-rule --global --target-http-proxy orchard-prod-proxy --ports=80

Configuring the release pipeline

Unlike some other continuous integration systems, VSTS makes a distinction between building and deploying, and provides a specialized set of tools labeled Release Management for all of the deployment-related tasks.

VSTS Release Management is built around these concepts:

  • A release refers to set of artifacts that make up a specific version of your application and that are usually the result of a build process.
  • Deployment refers to the process of taking a release and deploying it into a specific environment.
  • A deployment performs a set of tasks, which can be grouped in phases.
  • Environments allow you to distinguish between different sets of systems to deploy.
  • A pipeline allows orchestrating deployments into multiple environments.

Usually, a VSTS release consumes an artifact such as a zip file from a build. In this tutorial, the build produces a VM image in GCP, so there is no artifact like that to consume. However, by using the build ID, which is passed to the release as an environment variable, you can locate the corresponding VM image and use it for the deployment.

Creating a release definition

The first step is to create a new release definition.

  1. In VSTS, select Build and Release > Releases.
  2. Click New Definition. (New pipeline in new navigation UI.)
  3. From the list of templates, select Empty process.
  4. When you're prompted for an environment name, enter Dev.
  5. At the top of the screen, name the release Orchard-ComputeEngine.
  6. In the pipeline diagram, next to Artifacts, click Add.
  7. Select Build and add the following settings:

    • Source: Orchard-CI
    • Default version: Latest
  8. Click Add.

  9. On the Artifact box, click the lightning bolt icon to add a deployment trigger.
  10. Under Continuous deployment trigger, set the switch to Enabled.
  11. Click Save.
  12. Enter a comment if you want, and confirm by clicking Save.

The pipeline now looks like this:

Screenshot of the pipeline in VSTS

Deploying the development environment

Now that you have created the release definition, you can add the steps to initiate a rolling deployment.

  1. In VSTS, switch to the Tasks tab.
  2. Click Agent phase.
  3. Change the agent queue to Private > Google Cloud.
  4. Next to Agent phase, click the + icon to add a step to the phase.
  5. Select the Command Line task, click Add, and configure the following settings:

    • Version: 1.*
    • Display name: Create instance template
    • Tool: gcloud
    • Arguments: compute instance-templates create orchard-$(Build.BuildId)-$(Release.ReleaseId) --machine-type n1-standard-2 --image orchard-$(Build.BuildId) --image-project $(Packer.Project) --tags=gclb-backend

    This command creates a new instance template that uses the VM image that you previously built with Packer. The command applies the gclb-backend tag so that the load balancer can reach instances that are created from this template.

  6. Add another Command Line task and configure the following settings:

    • Version: 1.*
    • Display name: Associate instance template
    • Tool: gcloud
    • Arguments: beta compute instance-groups managed set-instance-template orchard-dev --template=orchard-$(Build.BuildId)-$(Release.ReleaseId) --zone $(Deployment.Dev.Zone)

    This command updates the existing instance group to use the new instance template. Note that this command does not yet cause any of the existing VMs to be replaced or updated. Instead, it ensures that any future VMs in this instance group are created from the new template.

  7. Add another Command Line task and configure the following settings:

    • Version: 1.*
    • Display name: Start rolling update
    • Tool: gcloud
    • Arguments: beta compute instance-groups managed rolling-action start-update orchard-dev --version template=orchard-$(Build.BuildId)-$(Release.ReleaseId) --type proactive --min-ready 2m --max-unavailable 0 --zone $(Deployment.Dev.Zone)

    This command causes the existing instance group to replace existing VMs with new VMs in a rolling fashion.

  8. Click the Variables tab, and add the following variables:

    Name Value
    Packer.Project The name of your GCP project.
    Deployment.Dev.Zone The zone that you specified earlier when running gcloud config set compute/zone (for example, us-central1-a)

  9. Click Save.

  10. Enter a comment if you want, and confirm by clicking OK.

Deploying the production environment

Finally, you need to configure the deployment to the GKE production cluster.

  1. In VSTS, switch to the Pipeline tab.
  2. In the Environments box, select Add > New environment.
  3. From the list of templates, select Empty process.
  4. When you're prompted for an environment name, enter Prod.
  5. Click the lightning bolt icon of the newly created environment.
  6. Configure the following settings:

    • Select trigger: After environment
    • Environment: Dev
    • Pre-deployment approvals: (enabled)
    • Approvers: Select your own user name or group.
  7. Hold the mouse over the Tasks tab and click Tasks > Prod.

  8. Click Agent phase.
  9. Change the Agent queue value to Private > Google Cloud.
  10. Next to Agent phase, click the + icon to add a step to the phase.
  11. Add another Command Line task and configure the following settings:

    • Version: 1.*
    • Display name: Associate instance template
    • Tool: gcloud
    • Arguments: beta compute instance-groups managed set-instance-template orchard-prod --template=orchard-$(Build.BuildId)-$(Release.ReleaseId) --zone $(Deployment.Prod.Zone)

    This command updates the existing instance group to use the instance template that you created during the deployment to the Dev environment. Reusing the same instance template ensures that you are deploying the exact same image.

  12. Add another Command Line task, and configure the following settings:

    • Version: 1.*
    • Display name: Start rolling update
    • Tool: gcloud
    • Arguments: beta compute instance-groups managed rolling-action start-update orchard-prod --version template=orchard-$(Build.BuildId)-$(Release.ReleaseId) --type proactive --min-ready 2m --max-unavailable 0 --zone $(Deployment.Prod.Zone)

    This command causes the existing instance group to replace existing VMs with new VMs in a rolling fashion.

  13. Switch to the Variables tab.

  14. Add a variable:

    • Name: Deployment.Prod.Zone
    • Value: Zone that you specified earlier when running gcloud config set compute/zone (for example: us-central1-a)
  15. Click Save.

  16. Enter a comment if you want, and confirm by clicking OK.

Running the pipeline

Now that you've configured the entire pipeline, it's time to test it. In the Packer template that you created previously, you used windows-2016 as the source image family. When you want to run a web server, running a full-featured Windows Server distribution might increase resource consumption with no benefit. Instead you will change the template to use Windows Server 2016 Core, and use this change to exercise the entire CI/CD pipeline.

  1. In Visual Studio, open the file packer.json.
  2. Change the source image family:

    "source_image_family": "windows-2016-core",

  3. Open Team Explorer, and switch to the Home view.

  4. Click Changes.
  5. Under Changes, right-click packer.json and then click Stage.
  6. Enter a commit message like Use Windows Server Core.
  7. Click Commit All and Push.
  8. In VSTS, select Build and Release > Builds and observe that a build has been triggered automatically:

    Screenshot showing list of builds underway, with the music store build in progress

    It might take 10 to 15 minutes before the status switches to Succeeded.

  9. When the build is finished, select Build and Release > Releases and observe that a release process has been initiated:

    Screenshot showing that the release process has started

  10. Click Release-1 to open the details page, and wait for the status of the Dev environment to switch to Succeeded. You might need to refresh the status by clicking the Refresh button in the menu or by reloading the browser page.

  11. In Cloud Shell, run the following command to obtain the IP address of the load balancer for the development environment:

    gcloud compute forwarding-rules list | grep orchard-dev | awk '{print $2}'

  12. In the browser, go to the Orchard installation using the URL that you got in the previous step:

    http://[DEV_IP]/orchard/

    You might see an error at first because the load balancer takes a few minutes to become available. When it’s ready, observe that Orchard CMS has been deployed successfully:

    Screenshot showing the Orchard CMS app running in a browser page

  13. In VSTS, in the yellow shaded bar at the top of the release page, click Approve or Reject:

    Screenshot showing the release page and a message 'A pre-deployment approval is pending ... Approve or Reject'

    If you don't see the yellow shaded bar, you might need to first approve or reject a previous release.

  14. Click Approve to trigger the deployment to the production environment.

  15. Wait for the status of the Prod environment to switch to Succeeded. You might need to manually refresh the page in your browser.
  16. In Cloud Shell, run the following command to obtain the IP address of the load balancer for the production environment:

    gcloud compute forwarding-rules list | grep orchard-prod | awk '{print $2}'

  17. In the browser, go to the orchard installation using the URL that you got in the previous step:

    http://[PROD_IP]/orchard/

    Again, you might see an error at first because the load balancer takes a few minutes to become available. When it's ready, you see the Orchard CMS page again, this time running in the production cluster.

Cleaning up

After you've finished the tutorial, clean up the resources you created so you won't be billed for them in the future.

Delete the VSTS project

Delete the project in VSTS. Note that deleting the VSTS project also causes all source code changes to be lost.

Delete the GCP project

  1. In the GCP Console, go to the Projects page.

    Go to the Projects page

  2. In the project list, select the project you want to delete and click Delete project. After selecting the checkbox next to the project name, click
      Delete project
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

Was this page helpful? Let us know how we did:

Send feedback about...