Creating a CI/CD pipeline with Azure Pipelines and Compute Engine

In this tutorial, you learn how to use Azure Pipelines (previously called Visual Studio Team Services) and Compute Engine to create a continuous integration/continuous deployment (CI/CD) pipeline. The tutorial uses Orchard CMS, an open source content management system as a sample application. Orchard CMS is based on ASP.NET MVC and runs on Windows Server 2016.

The CI/CD pipeline uses two separate environments, one for testing and one for production. When a developer commits a change to the Git repository, the commit causes the source code to be built and a new Windows Server 2016–based virtual machine (VM) image to be created by using Packer. The new image is then automatically released to the development environment by using a rolling update. After testing, a release manager can then promote the release so that it's deployed into the production environment.

Conceptual diagram of CI/CD pipeline showing how developers and end users interact with the application

The tutorial assumes that you have basic knowledge of the .NET Framework, Windows Server, Microsoft Internet Information Services (IIS), Azure Pipelines, and Compute Engine. The tutorial also requires you to have administrative access to an Azure DevOps account and a Visual Studio 2017 installation that's already connected to your Azure DevOps account.

Objectives

  • Run a private Azure Pipelines agent on Compute Engine and connect it to Azure Pipelines.
  • Use Packer with Compute Engine to create Windows images.
  • Use Compute Engine Managed Instance Groups to implement rolling deployments.
  • Set up a CI/CD pipeline in Azure Pipelines to orchestrate the building, creating, and deployment processes.

Costs

This tutorial uses billable components of Google Cloud Platform, including:

Use the Pricing Calculator to generate a cost estimate based on your projected usage. Check the Azure DevOps pricing page for any fees that might apply for using Azure DevOps.

Before you begin

Deploying an application to Google Cloud Platform (GCP) requires a GCP project. It is usually advisable to use separate projects for CI, development, and production workloads so that identity and access management (IAM) roles and permissions can be granted individually.

However, for the sake of simplicity this tutorial uses a single project for the CI, development, and production environments.

  1. Sign in to your Google Account.

    If you don't already have one, sign up for a new account.

  2. Select or create a GCP project.

    Go to the Manage resources page

  3. Make sure that billing is enabled for your project.

    Learn how to enable billing

  4. Make sure you have an Azure DevOps account and have administrator access to it. If you don't yet have an Azure DevOps account, you can sign up on the Azure DevOps home page.
  5. Make sure you have Visual Studio 2017 installed and that it's connected to your Azure DevOps account.

Creating an Azure DevOps project

Using Orchard CMS as an example, you will use Azure Pipelines to manage the source code, run builds and tests, and orchestrate the deployment to Compute Engine.

To begin, create a new project in your Azure DevOps account:

  1. Go to the Azure DevOps home page (https://dev.azure.com/[YOUR_AZURE_DEVOPS_ACCOUNT_NAME]).
  2. Click Create Project.
  3. Enter a project name, such as Orchard.
  4. Set Visibility to Private, and then click Create.
  5. After the project has been created, in the menu on the left, click Repos.
  6. Click Import to fork the Orchard CMS repository from GitHub. Set the following values:

    • Source type: Git
    • Clone URL: https://github.com/OrchardCMS/Orchard.git
    • Leave the Requires authorization checkbox unselected.

    Screenshot of the 'Import a Git repository' dialog box

  7. Click Import. When the import process is done, you see the source code for Orchard CMS.

Continuous integration

You can now use Azure Pipelines to set up continuous integration. For each commit that you push to the Git repository, Azure Pipelines will build the code and publish the resulting build artifact to internal Azure Pipelines storage.

Creating a testing branch

To make sure that the instructions in this tutorial work, you need to create a branch that's based on a specific version of the source code. This step will help make sure that future changes to the code on GitHub don't break this tutorial.

  1. In the Azure DevOps menu, select Repos > Tags.
  2. In the list of tags, right-click the icon next to 1.10.2.
  3. Select New branch.
  4. In the Name box, enter testing as the branch name, and confirm by clicking Create branch.

    Screenshot of the 'Create a branch' dialog box in Azure Pipelines

By default, Azure Pipelines expects your code to reside in the master branch. In order to have it use the testing branch, you need to change the default branch:

  1. In the Azure DevOps menu, select Project settings.
  2. Select Code > Repositories.
  3. In the list of repositories, select the Git repository you imported previously. It should have the same name as your Azure DevOps project.
  4. Expand the list of branches by clicking the arrow next to Branches.
  5. Select the testing branch. This causes a ... button to appear right next to the name of the branch.
  6. Click ... and select Set as default branch.

Creating a build definition

After you create the branch, you can define the build. Because Orchard CMS is an ASP.NET application that's written in Visual Studio, defining the build includes the following steps:

  • Restoring the NuGet package dependencies.
  • Building the solution (src\Orchard.sln) on a Hosted VS2017 build agent.
  • Publishing the build artifacts of the Orchard.Web project.

You can automate these steps as follows:

  1. In the Azure DevOps menu, select Pipelines > Builds, and then click New pipeline.
  2. Select Azure Repos as the code location.
  3. Select the Git repository you imported previously. It should have the same name as your Azure DevOps project.
  4. Select Starter pipeline.
  5. Remove the example code generated for azure-pipelines.yml and copy the following code into the editor:

    resources:
    - repo: self
      fetchDepth: 1
    variables:
      artifactName: 'Orchard.Web'
    phases:
    - phase: Phase_1
      displayName: Build application
      condition: succeeded()
      queue:
        name: Hosted VS2017
        demands: 
        - msbuild
        - visualstudio
      variables:
        solution: 'src\Orchard.sln'
        buildPlatform: 'Any CPU'
        buildConfiguration: 'Release'
      steps:
      - task: NuGetToolInstaller@0
        displayName: 'Use NuGet 4.4.1'
        inputs:
          versionSpec: 4.4.1
      - task: NuGetCommand@2
        displayName: 'NuGet restore'
        inputs:
          restoreSolution: '$(solution)'
      - task: VSBuild@1
        displayName: 'Build solution'
        inputs:
          solution: '$(solution)'
          msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactstagingdirectory)\\"'
          platform: '$(buildPlatform)'
          configuration: '$(buildConfiguration)'
      - task: PublishSymbols@2
        displayName: 'Publish symbols path'
        continueOnError: true
        inputs:
          SearchPattern: '**\bin\**\*.pdb'
          PublishSymbols: false
      - task: PublishBuildArtifacts@1
        displayName: 'Publish Artifact'
        inputs:
          PathtoPublish: '$(build.artifactstagingdirectory)/Orchard.Web.zip'
          ArtifactName: '$(artifactName)'
    
  6. Click Save and run.

  7. Enter a custom commit message, and then confirm by clicking Save and run.

    This commits azure-pipelines.yml to the testing branch of your repository and triggers a build. Any further changes to the build definition have to be made by changing the file in the Git repository.

The build takes up to seven minutes to complete. At the end of the build, the file Orchard.Web.zip, which contains all files of the web application, will be available in the internal Azure Pipelines artifact storage area.

With the source code being built continuously, the next step is to create the VM images automatically. This automation includes the following steps:

  • Launching a new, temporary VM instance that uses the Windows Server 2016 image.
  • Installing and configuring IIS.
  • Deploying Orchard CMS.
  • Stopping the VM.
  • Creating an image.
  • Deleting the temporary VM.

You use Packer to automate this process. Packer needs access to the VM instance, but you don't want to expose the instance to the public internet. Therefore, you run Packer on a private Azure Pipelines agent within the GCP network.

Deploying a private Azure Pipelines agent

Running Packer on a private agent in GCP avoids exposing resources to the public internet and thereby creating a less secure environment. However, it also increases administrative overhead. To minimize this overhead, you deploy the agent in a managed instance group. Using this approach has these advantages:

  • A managed instance group allows you to re-create the VM, install all required components, and register with Azure Pipelines automatically.
  • You can apply Windows updates by deleting the VM and letting the managed instance group re-create the environment with the latest, fully patched Windows image as a base.

Follow these steps:

  1. In the GCP Console, switch to your newly created project.
  2. Open Cloud Shell.

    Go to Cloud Shell

  3. To save time, set default values for your project ID and Compute Engine zone:

    gcloud config set project [PROJECT_NAME]
    gcloud config set compute/zone [ZONE]

    Replace [PROJECT_NAME] with the name of your GCP project, and replace [ZONE] with the name of the zone that you're going to use for creating resources. If you are unsure about which zone to pick, use us-central1-a.

    Example:

    gcloud config set project devops-test-project-12345
    gcloud config set compute/zone us-central1-a
  4. Enable the Compute Engine API:

    gcloud services enable compute.googleapis.com
  5. In the Azure DevOps menu, select Project settings, and then select Pipelines > Agent pools.

  6. Click New agent pool.

  7. Enter Google Cloud as the pool name and click OK. This pool will be the queue for the private build agent that is running on Compute Engine.

  8. In the left pane, select the newly created Google Cloud pool and click Download agent.

  9. Under Download the agent, click Copy to copy the download URL.

  10. In Cloud Shell, initialize an environment variable. For the URL value, paste the URL that you copied in the previous step.

    export AZURE_DEVOPS_AGENT_URL=[PASTE URL FROM CLIPBOARD]
  11. Switch back to Azure Pipelines, click the user avatar in the upper-right corner of the screen, and then click Security.

  12. Under Security > Personal access tokens, click New Token.

  13. Configure the following settings:

    • Name: Google Cloud Agent
    • Scopes: Full access
  14. Click Create.

  15. Copy the token to the clipboard.

  16. In Cloud Shell, initialize another environment variable by running the following command. For the URL, paste the URL that you just copied.

    export AZURE_DEVOPS_TOKEN=[PASTE TOKEN FROM CLIPBOARD]
  17. Initialize another environment variable to contain the Azure Pipelines URL. Replace [ACCOUNT] with your account name, as displayed in the address bar in Azure Pipelines.

    export AZURE_DEVOPS_URL=https://[ACCOUNT].visualstudio.com
  18. Run the following command to create a specialize script. The script downloads and installs the Azure Pipelines agent package, Packer, and Cloud SDK, and registers the agent with Azure Pipelines.

    cat | envsubst '$AZURE_DEVOPS_AGENT_URL $AZURE_DEVOPS_TOKEN $AZURE_DEVOPS_URL' > specialize.ps1 << 'EOF'
    # Create an installation directory for the Azure Pipelines agent
    New-Item -ItemType directory -Path $env:programfiles\vsts-agent
    
    # Create a work directory for the Azure Pipelines agent
    New-Item -ItemType directory -Path $env:programdata\vsts-agent
    
    # Download and install the Azure Pipelines agent package
    Invoke-WebRequest `
      -Uri "$AZURE_DEVOPS_AGENT_URL" `
      -OutFile $env:TEMP\vsts-agent.zip
    Add-Type -AssemblyName System.IO.Compression.FileSystem
    [System.IO.Compression.ZipFile]::ExtractToDirectory( `
      "$env:TEMP\vsts-agent.zip", `
      "$env:programfiles\vsts-agent")
    
    # Download and install Packer
    [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
    Invoke-WebRequest `
      -Uri "https://releases.hashicorp.com/packer/1.2.2/packer_1.2.2_windows_amd64.zip" `
      -OutFile $env:TEMP\packer.zip
    [System.IO.Compression.ZipFile]::ExtractToDirectory( `
      "$env:TEMP\packer.zip", `
      "$env:programfiles\packer")
    
    # Download and install the Cloud SDK
    Invoke-WebRequest `
      -Uri https://dl.google.com/dl/cloudsdk/channels/rapid/GoogleCloudSDKInstaller.exe `
      -OutFile $env:TEMP\cloudsdk.exe
    Start-Process -Wait $env:TEMP\cloudsdk.exe -arg "/S /noreporting /nostartmenu /nodesktop"
    
    # Add Packer and the Cloud SDK installation directory to global path
    [Environment]::SetEnvironmentVariable( `
      "Path", $env:Path + ";$env:programfiles\packer;${env:ProgramFiles(x86)}\Google\Cloud SDK\google-cloud-sdk\bin", `
      [System.EnvironmentVariableTarget]::Machine)
    
    # Install gcloud beta commands
    $env:CLOUDSDK_PYTHON=gcloud components copy-bundled-python
    Start-Process -Wait gcloud -arg "components install beta --quiet"
    
    # Configure the Azure Pipelines agent
    & $env:programfiles\vsts-agent\bin\Agent.Listener configure `
      --url $AZURE_DEVOPS_URL `
      --agent "GCE Agent" `
      --work $env:programdata\vsts-agent `
      --pool "Google Cloud" `
      --replace `
      --runAsService `
      --windowsLogonAccount "NT AUTHORITY\NETWORK SERVICE" `
      --auth PAT `
      --token $AZURE_DEVOPS_TOKEN
    EOF
    
  19. Create an instance template for the Azure Pipelines agent. Configure the instance template so that the VM instance runs specialize.ps1 as a specialize script during startup.

    gcloud compute instance-templates create vsts-agent \
        --machine-type n1-standard-1 \
        --image-family windows-2016-core \
        --image-project windows-cloud \
        --metadata-from-file sysprep-specialize-script-ps1=specialize.ps1 \
        --scopes "https://www.googleapis.com/auth/compute,https://www.googleapis.com/auth/devstorage.read_write"
    
  20. Create a managed instance group that's based on this instance template. It might take around three minutes for the new VM instance to start and for the Azure Pipelines agent to register.

    gcloud compute instance-groups managed create vsts-agent \
        --template=vsts-agent \
        --size=1
    
  21. In Azure DevOps, navigate back to your project.

  22. In the menu, select Project settings, and then select Pipelines > Agent pools.

  23. Select the Google Cloud pool, and then verify that the agent is registered and its state shows as Online.

    Screenshot of the 'Agent Queues' dialog box in Azure Pipelines

Troubleshooting

If the agent doesn't register within 10 minutes, there might be an issue. Try the following:

  1. In the GCP Console, navigate to Compute Engine > VM Instances.
  2. Open the details for the VM that has the prefix vsts-agent.
  3. Click Serial port 1 (console).

The console log contains the output of the specialize script, which might help with troubleshooting. Even after the agent shows up in Azure Pipelines, it might initially appear as Offline. In that case, wait another two minutes for the status to change to Online.

Testing the setup

If you want, test the resiliency of this setup:

  1. Go to Compute Engine > VM Instances.
  2. Delete the VM that has the prefix vsts-agent.

The managed instance group will immediately spawn a new instance, and a few minutes later, a new agent will be operational again.

Creating VM images

To automate the VM image creation process, you need to create a Packer template. This template is a JSON file that you should keep with the project's source code.

Creating a Packer template

Before you create the template, you need to check out the source code.

  1. In Visual Studio, open Team Explorer.
  2. In the menu, click the Manage Connections icon.
  3. Click Manage Connections > Connect to a Project.

    Screenshot of the 'Connect to a Project' option in the Team Explorer pane of Visual Studio

  4. In the next dialog, select the Orchard Git repository, and then click Clone.

    Screenshot of the 'Orchard' Git repository selected in the 'Connect to a Project' dialog in Visual Studio

  5. After the code has been checked out, in Visual Studio, open Solution Explorer.

  6. In the root of the solution, create a new file named packer.json.

  7. Copy the following code into the newly created file, and then save the file:

    {
      "variables": {
        "gcp_project": "",
        "gcp_zone": "",
        "windows_user": "packer_user",
        "windows_password": "Packer123",
        "image_name": "vsts",
        "image_family": "vsts",
        "app_package": ""
      },
      "builders": [
        {
          "type": "googlecompute",
          "project_id": "{{user `gcp_project`}}",
          "source_image_family": "windows-2016",
          "disk_size": "50",
          "instance_name": "{{user `image_name`}}",
          "image_name": "{{user `image_name`}}",
          "image_family": "{{user `image_family`}}",
          "machine_type": "n1-standard-2",
          "communicator": "winrm",
          "winrm_username": "{{user `windows_user`}}",
          "winrm_password": "{{user `windows_password`}}",
          "winrm_insecure": true,
          "winrm_use_ssl": true,
          "winrm_port": 5986,
          "metadata": {
            "windows-startup-script-cmd": "winrm quickconfig -quiet & net user /add {{user `windows_user`}} {{user `windows_password`}} & net localgroup administrators {{user `windows_user`}} /add & winrm set winrm/config/service/auth @{Basic=\"true\"}"
          },
          "zone": "{{user `gcp_zone`}}",
          "use_internal_ip": true,
          "state_timeout": "8m",
          "scopes": [ "https://www.googleapis.com/auth/devstorage.read_only" ]
        }
      ],
      "provisioners": [
        {
          "type": "powershell",
          "inline": [
            "$ErrorActionPreference = \"Stop\"",
    
            "# Download application package from Cloud Storage",
            "gsutil cp {{user `app_package`}} $env:TEMP\\app.zip",
    
            "# Install IIS",
            "Enable-WindowsOptionalFeature -Online -FeatureName NetFx4Extended-ASPNET45",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-WebServerRole",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-WebServer",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-CommonHttpFeatures",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-HttpErrors",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-HttpRedirect",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-ApplicationDevelopment",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-HealthAndDiagnostics",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-HttpLogging",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-LoggingLibraries",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-RequestMonitor",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-HttpTracing",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-Security",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-RequestFiltering",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-Performance",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-WebServerManagementTools",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-IIS6ManagementCompatibility",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-Metabase",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-DefaultDocument",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-ApplicationInit",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-NetFxExtensibility45",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-ISAPIExtensions",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-ISAPIFilter",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-ASPNET45",
            "Enable-WindowsOptionalFeature -Online -FeatureName IIS-HttpCompressionStatic",
    "# Extract application package to wwwroot", "New-Item -ItemType directory -Path $env:TEMP\\app", "Add-Type -AssemblyName System.IO.Compression.FileSystem", "[System.IO.Compression.ZipFile]::ExtractToDirectory(\"$env:TEMP\\app.zip\", \"$env:TEMP\\app\")", "Remove-Item $env:TEMP\\app.zip", "Move-Item -Path $(dir -recurse $env:TEMP\\app\\**\\PackageTmp | % { $_.FullName }) -Destination c:\\inetpub\\wwwroot\\app -force", "# Configure IIS web application pool and application", "Import-Module WebAdministration", "New-WebAppPool orchard-net4", "Set-ItemProperty IIS:\\AppPools\\orchard-net4 managedRuntimeVersion v4.0", "New-WebApplication -Name Orchard -Site 'Default Web Site' -PhysicalPath c:\\inetpub\\wwwroot\\app -ApplicationPool orchard-net4", "# Grant read/execute access to the application pool user", "&icacls C:\\inetpub\\wwwroot\\app\\ /grant \"IIS AppPool\\orchard-net4:(OI)(CI)(RX)\"", "# Create data folder and grant write access to the application pool user", "New-Item -ItemType directory -Path C:\\inetpub\\wwwroot\\app\\App_Data\\", "&icacls C:\\inetpub\\wwwroot\\app\\App_Data\\ /grant \"IIS AppPool\\orchard-net4:(OI)(CI)M\"", "# Disable searching for Windows updates", "New-ItemProperty -Path HKLM:\\SOFTWARE\\Policies\\Microsoft\\Windows\\WindowsUpdate\\AU -Name NoAutoUpdate -Value 1 -PropertyType DWORD -Force", "# Disable provisioning user", "disable-localuser {{user `windows_user`}}", "# Generalize the image", "& \"$Env:Programfiles\\Google\\Compute Engine\\sysprep\\gcesysprep\"" ] } ] }

The Packer template has three sections:

  • The variables section defines the variables that are used in the template. The values for these variables will be passed as command-line arguments to Packer.
  • The builders section contains the settings for creating and communicating with the temporary VM instance.
  • The provisioners section contains a sequence of commands that are run on the VM to configure it after it has been created.

The preceding template uses the Packer Google Compute Builder to create a VM instance that runs Windows Server 2016. When you specify the source image for Packer to use, it's important to use an image that already includes the latest security patches. To prevent you from having to change the source_image setting every time that Google releases a new version of the Windows Server 2016 image, the configuration obtains the most recent image from the windows-2016-core image family.

Using a startup script, the template enables Windows Remote Management (WinRM) and creates a temporary user. After the VM is up and running, Packer uses WinRM and this user to run the PowerShell commands that are specified in the provisioners section. Using the use_internal_ip setting, you ensure that WinRM communicates over the local network rather than over the internet, which is possible because the corresponding build step will run on the private build agent that you provisioned earlier. Finally, the builders section also defines an IAM scope for the VM, which permits access to Cloud Storage.

The PowerShell commands that are in the provisioners section configure the VM to run Orchard CMS. This configuration involves the following:

  • Downloading the application package from Cloud Storage.
  • Installing IIS.
  • Creating an IIS application pool and application for Orchard CMS.
  • Extracting the application package to the IIS webroot folder and configuring the access control lists (ACLs) for the folder.
  • Disabling the search for Windows updates, which is unnecessary if you apply updates by rerunning the CI/CD pipeline periodically.
  • Disabling the provisioning user.
  • Generalizing the image by using GCESysprep to ensure that each VM that's created from the image is assigned a unique security identifier.

To commit the file to Git, do the following:

  1. In Visual Studio, open Team Explorer.
  2. Click the Home icon at upper left to switch to the Home view.
  3. Click Changes.
  4. Under Changes, right-click packer.json and then click Stage.
  5. Enter a commit message like Add Packer template.
  6. Click Commit Staged.

Creating a Cloud Storage bucket for build artifacts

Using the File Provisioner in Packer, you can copy files from the machine that's running Packer to the VM. If you use WinRM, this operation can be slow. Therefore, the template that you created in the previous section looks for the Orchard CMS application package on Cloud Storage, which offers substantially better performance.

To create a bucket in Cloud Storage for this purpose, run the following command:

gsutil mb gs://$(gcloud config get-value core/project)-artifacts

If you don't want to keep the artifacts of all builds, you might consider configuring an object lifecycle rule to delete files that are past a certain age.

Extending the build definition

You have now checked the Packer template in to Git and created the Cloud Storage bucket. You can now integrate Packer into your Azure Pipelines build process.

So far, the Azure Pipelines build definition that you've created uses a single phase. In Azure Pipelines, all tasks that are part of the same phase run on the same build agent. In your case, that agent is Hosted VS2017. To start using the private build agent that is provisioned in Compute Engine, you must add a second phase to the build definition.

Within a single phase, tasks share a working directory and have access to artifacts from previous tasks. This access is not possible across phases, because to consume artifacts from previous stages, you must publish those artifacts to Azure Pipelines in the first phase and then download them in the next phase. Therefore, extend the build definition as follows:

  1. In Visual Studio, open azure-pipelines.yml.
  2. Extend the build definition by appending the following piece of code to the file. Replace [PROJECT_NAME] with the name of your GCP project, and replace [ZONE] with the name of the zone that you're going to use for creating resources. If you are unsure about which zone to pick, use us-central1-a.

    - phase: Phase_2
      displayName: Create VM image
      dependsOn: Phase_1
      condition: succeeded()
      queue:
        name: 'Google Cloud'
      variables:
        Packer.Project: '[PROJECT_NAME]'
        Packer.Zone: '[ZONE]'
      steps:
      - task: DownloadBuildArtifacts@0
        displayName: 'Download Build Artifacts'
        inputs:
          artifactName: '$(artifactName)'
      - task: CmdLine@1
        displayName: 'Publish artifact to Cloud Storage'
        inputs:
          filename: gsutil
          arguments: 'cp $(System.ArtifactsDirectory)\$(artifactName)\Orchard.Web.zip gs://$(Packer.Project)-artifacts/Orchard.Web-$(Build.BuildId).zip'
      - task: CmdLine@1
        displayName: 'Create image'
        inputs:
          filename: packer
          arguments: 'build -var "gcp_project=$(Packer.Project)" -var "gcp_zone=$(Packer.Zone)" -var "image_family=orchard" -var "image_name=orchard-$(Build.BuildId)" -var "app_package=gs://$(Packer.Project)-artifacts/Orchard.Web-$(Build.BuildId).zip" $(Build.SourcesDirectory)/packer.json'
    

To commit and push the changes to Azure DevOps, do the following:

  1. In Visual Studio, open Team Explorer.
  2. Click the Home icon at upper left to switch to the Home view.
  3. Click Changes.
  4. Under Changes, right-click azure-pipelines.yml and then click Stage.
  5. Enter a commit message like Extend build definition to create VM image.
  6. Click Commit Staged and Push.
  7. In the Azure DevOps menu, select Pipelines > Build. You should see that a new build has automatically been triggered. Allow 10 to 15 minutes for the build to complete. Note that if the build fails with the error message Could not find a pool with name Google Cloud, you may need to re-save your pipeline.
  8. In GCP, navigate to Compute Engine > Images and confirm that an image named orchard-N has been created, where N represents the Azure Pipelines build ID.

Now that Azure Pipelines is automatically building your code and creating a new VM image for each commit, you can turn your attention toward deployment.

Configuring the development environment

With Orchard CMS, you can use either SQL Server or an embedded database that stores data locally. For the sake of simplicity, use the default configuration that relies on the embedded database, although it comes with two restrictions:

  • Only a single VM instance can run at a time. Otherwise, users might see different data depending on which VM instance is serving content to them.
  • Any data changes are lost whenever the VM instance is restarted, unless you change the deployment to use Cloud Filestore for data storage. (We do not cover this scenario in the tutorial.)

Before you can configure the steps in Azure Pipelines to automate the deployment, you must prepare the development environment. This preparation includes creating a managed instance group that will manage the web server VM instances. It also includes creating an HTTP load balancer.

  1. In Cloud Shell, create an instance template that uses a standard Windows Server 2016 Core image (not one that's been customized). You will use this template only initially, because each build will produce a new template.

    gcloud compute instance-templates create orchard-initial \
        --machine-type n1-standard-2 \
        --image-family windows-2016-core \
        --image-project windows-cloud
  2. Create an HTTP health check. Because Orchard CMS does not have a dedicated health check endpoint, you can query the path /.

    gcloud compute http-health-checks create orchard-dev-http \
        --check-interval=10s --unhealthy-threshold=10 \
        --request-path=/
  3. Create a managed instance group that's based on the initial instance template. For simplicity's sake, the following commands create a zonal managed instance group. However, you can use the same approach for regional managed instance groups that distribute VM instances across more than one zone.

    gcloud beta compute instance-groups managed create orchard-dev \
        --template=orchard-initial \
        --http-health-check=orchard-dev-http \
        --initial-delay=2m \
        --size=1 && \
    gcloud compute instance-groups set-named-ports orchard-dev --named-ports http:80
  4. Create a load balancer backend service that uses the HTTP health check and managed instance group that you created previously:

    gcloud compute backend-services create orchard-dev-backend \
        --http-health-checks orchard-dev-http \
        --port-name http --protocol HTTP --global && \
    gcloud compute backend-services add-backend orchard-dev-backend \
        --instance-group orchard-dev --global \
        --instance-group-zone=$(gcloud config get-value compute/zone)
  5. Create a load balancer frontend:

    gcloud compute url-maps create orchard-dev --default-service orchard-dev-backend && \
    gcloud compute target-http-proxies create orchard-dev-proxy --url-map=orchard-dev && \
    gcloud compute forwarding-rules create orchard-dev-fw-rule --global --target-http-proxy orchard-dev-proxy --ports=80
  6. Create a firewall rule that allows the Google load balancer to send HTTP requests to instances that have been annotated with the gclb-backend tag. You will later apply this tag to the web service VM instances.

    gcloud compute firewall-rules create gclb-backend --source-ranges=130.211.0.0/22,35.191.0.0/16 --target-tags=gclb-backend --allow tcp:80

Configuring the production environment

Setting up the production environment requires a sequence of steps similar to those for configuring the development environment.

  1. In Cloud Shell, create an HTTP health check. Because Orchard CMS does not have a dedicated health check endpoint, you can query the path /.

    gcloud compute http-health-checks create orchard-prod-http \
        --check-interval=10s --unhealthy-threshold=10 \
        --request-path=/
  2. Create another managed instance group that is based on the initial instance template that you created earlier:

    gcloud beta compute instance-groups managed create orchard-prod \
        --template=orchard-initial \
        --http-health-check=orchard-prod-http \
        --initial-delay=2m \
        --size=1 && \
    gcloud compute instance-groups set-named-ports orchard-prod --named-ports http:80
  3. Create a load balancer backend service that uses the HTTP health check and managed instance group that you created previously:

    gcloud compute backend-services create orchard-prod-backend --http-health-checks orchard-prod-http --port-name http --protocol HTTP --global && \
    gcloud compute backend-services add-backend orchard-prod-backend --instance-group orchard-prod --global --instance-group-zone=$(gcloud config get-value compute/zone)
  4. Create a load balancer frontend:

    gcloud compute url-maps create orchard-prod --default-service orchard-prod-backend && \
    gcloud compute target-http-proxies create orchard-prod-proxy --url-map=orchard-prod && \
    gcloud compute forwarding-rules create orchard-prod-fw-rule --global --target-http-proxy orchard-prod-proxy --ports=80

Configuring the release pipeline

Unlike some other continuous integration systems, Azure Pipelines makes a distinction between building and deploying, and provides a specialized set of tools labeled Release Management for all of the deployment-related tasks.

Azure Pipelines Release Management is built around these concepts:

  • A release refers to set of artifacts that make up a specific version of your application and that are usually the result of a build process.
  • Deployment refers to the process of taking a release and deploying it into a specific environment.
  • A deployment performs a set of tasks, which can be grouped in jobs.
  • Stages allow you to segment your pipeline and can be used to orchestrate deployments to multiple environments, for example development and testing environments.

Usually, an Azure Pipelines release consumes an artifact such as a zip file from a build. In this tutorial, the build produces a VM image in GCP, so there is no artifact like that to consume. However, by using the build ID, which is passed to the release as an environment variable, you can locate the corresponding VM image and use it for the deployment.

Creating a release definition

The first step is to create a new release definition.

  1. In the Azure DevOps menu, select Pipelines > Releases.
  2. Click New pipeline.
  3. From the list of templates, select Empty job.
  4. When you're prompted for a name for the stage, enter Dev.
  5. At the top of the screen, name the release Orchard-ComputeEngine.
  6. In the pipeline diagram, next to Artifacts, click Add.
  7. Select Build and add the following settings:

    • Source: Select the Git repository that contains the azure-pipelines.yml file.
    • Default version: Latest
    • Source alias: Orchard
  8. Click Add.

  9. On the Artifact box, click the lightning bolt icon to add a deployment trigger.

  10. Under Continuous deployment trigger, set the switch to Enabled.

  11. Click Save.

  12. Enter a comment if you want, and confirm by clicking Save.

The pipeline now looks like this:

Screenshot of the pipeline in Azure Pipelines

Deploying the development environment

Now that you have created the release definition, you can add the steps to initiate a rolling deployment.

  1. In Azure Pipelines, switch to the Tasks tab.
  2. Click Agent job.
  3. Change the agent pool to Private > Google Cloud.
  4. Next to Agent job, click the + icon to add a step to the phase.
  5. Select the Command Line task, click Add, and configure the following settings:

    • Version: 1.*
    • Display name: Create instance template
    • Tool: gcloud
    • Arguments: compute instance-templates create orchard-$(Build.BuildId)-$(Release.ReleaseId) --machine-type n1-standard-2 --image orchard-$(Build.BuildId) --image-project $(Packer.Project) --tags=gclb-backend

    This command creates a new instance template that uses the VM image that you previously built with Packer. The command applies the gclb-backend tag so that the load balancer can reach instances that are created from this template.

  6. Add another Command Line task and configure the following settings:

    • Version: 1.*
    • Display name: Associate instance template
    • Tool: gcloud
    • Arguments: beta compute instance-groups managed set-instance-template orchard-dev --template=orchard-$(Build.BuildId)-$(Release.ReleaseId) --zone $(Deployment.Dev.Zone)

    This command updates the existing instance group to use the new instance template. Note that this command does not yet cause any of the existing VMs to be replaced or updated. Instead, it ensures that any future VMs in this instance group are created from the new template.

  7. Add another Command Line task and configure the following settings:

    • Version: 1.*
    • Display name: Start rolling update
    • Tool: gcloud
    • Arguments: beta compute instance-groups managed rolling-action start-update orchard-dev --version template=orchard-$(Build.BuildId)-$(Release.ReleaseId) --type proactive --min-ready 2m --max-unavailable 0 --zone $(Deployment.Dev.Zone)

    This command causes the existing instance group to replace existing VMs with new VMs in a rolling fashion.

  8. Click the Variables tab, and add the following variables:

    Name Value
    Packer.Project The name of your GCP project.
    Deployment.Dev.Zone The zone that you specified earlier when running gcloud config set compute/zone (for example, us-central1-a)
  9. Click Save.

  10. Enter a comment if you want, and confirm by clicking OK.

Deploying the production environment

Finally, you need to configure the deployment to the GKE production cluster.

  1. In Azure Pipelines, switch to the Pipeline tab.
  2. In the Stages box, select Add > New stage.
  3. From the list of templates, select Empty job.
  4. When you're prompted for a name for the stage, enter Prod.
  5. Click the lightning bolt icon of the newly created stage.
  6. Configure the following settings:

    • Select trigger: After stage
    • Stages: Dev
    • Pre-deployment approvals: (enabled)
    • Approvers: Select your own user name or group.
  7. Hold the mouse over the Tasks tab and click Tasks > Prod.

  8. Click Agent job.

  9. Change the Agent pool value to Private > Google Cloud.

  10. Next to Agent phase, click the + icon to add a step to the phase.

  11. Add a Command Line task and configure the following settings:

    • Version: 1.*
    • Display name: Associate instance template
    • Tool: gcloud
    • Arguments: beta compute instance-groups managed set-instance-template orchard-prod --template=orchard-$(Build.BuildId)-$(Release.ReleaseId) --zone $(Deployment.Prod.Zone)

    This command updates the existing instance group to use the instance template that you created during the deployment to the Dev environment. Reusing the same instance template ensures that you are deploying the exact same image.

  12. Add another Command Line task, and configure the following settings:

    • Version: 1.*
    • Display name: Start rolling update
    • Tool: gcloud
    • Arguments: beta compute instance-groups managed rolling-action start-update orchard-prod --version template=orchard-$(Build.BuildId)-$(Release.ReleaseId) --type proactive --min-ready 2m --max-unavailable 0 --zone $(Deployment.Prod.Zone)

    This command causes the existing instance group to replace existing VMs with new VMs in a rolling fashion.

  13. Switch to the Variables tab.

  14. Add a variable:

    • Name: Deployment.Prod.Zone
    • Value: Zone that you specified earlier when running gcloud config set compute/zone (for example: us-central1-a)
  15. Click Save.

  16. Enter a comment if you want, and confirm by clicking OK.

Running the pipeline

Now that you've configured the entire pipeline, it's time to test it. In the Packer template that you created previously, you used windows-2016 as the source image family. When you want to run a web server, running a full-featured Windows Server distribution might increase resource consumption with no benefit. Instead you will change the template to use Windows Server 2016 Core, and use this change to exercise the entire CI/CD pipeline.

  1. In Visual Studio, open the file packer.json.
  2. Change the source image family:

    "source_image_family": "windows-2016-core",
  3. Open Team Explorer, and switch to the Home view.

  4. Click Changes.

  5. Under Changes, right-click packer.json and then click Stage.

  6. Enter a commit message like Use Windows Server Core.

  7. Click Commit All and Push.

  8. In Azure Pipelines, select Build and Release > Builds and observe that a build has been triggered automatically:

    Screenshot showing list of builds underway, with the music store build in progress

    It might take 10 to 15 minutes before the status switches to Succeeded.

  9. When the build is finished, select Build and Release > Releases and observe that a release process has been initiated:

    Screenshot showing that the release process has started

  10. Click Release-1 to open the details page, and wait for the status of the Dev stage to switch to Succeeded. You might need to refresh the status by clicking the Refresh button in the menu or by reloading the browser page.

  11. In Cloud Shell, run the following command to obtain the IP address of the load balancer for the development environment:

    gcloud compute forwarding-rules list | grep orchard-dev | awk '{print $2}'
  12. In the browser, go to the Orchard installation using the URL that you got in the previous step:

    http://[DEV_IP]/orchard/

    You might see an error at first because the load balancer takes a few minutes to become available. When it’s ready, observe that Orchard CMS has been deployed successfully:

    Screenshot showing the Orchard CMS app running in a browser page

  13. In Azure Pipelines, below the Prod stage, click Approve to trigger the deployment to the production environment:

    Screenshot showing the release page and a message 'A pre-deployment approval is pending ... Approve or Reject'

  14. Wait for the status of the Prod stage to switch to Succeeded. You might need to manually refresh the page in your browser.

  15. In Cloud Shell, run the following command to obtain the IP address of the load balancer for the production environment:

    gcloud compute forwarding-rules list | grep orchard-prod | awk '{print $2}'
  16. In the browser, go to the orchard installation using the URL that you got in the previous step:

    http://[PROD_IP]/orchard/

    Again, you might see an error at first because the load balancer takes a few minutes to become available. When it's ready, you see the Orchard CMS page again, this time running in the production cluster.

Cleaning up

After you've finished the tutorial, clean up the resources you created so you won't be billed for them in the future.

Delete the Azure Pipelines project

Delete the project in Azure DevOps. Note that deleting the Azure DevOps project also causes all source code changes to be lost.

Delete the GCP project

  1. In the GCP Console, go to the Projects page.

    Go to the Projects page

  2. In the project list, select the project you want to delete and click Delete project. After selecting the checkbox next to the project name, click
      Delete project
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

Was this page helpful? Let us know how we did:

Send feedback about...