Autoscaling an Instance Group with Stackdriver Custom Metrics

This tutorial shows how to set up a Compute Engine managed instance group that autoscales based on the value of a custom Stackdriver Monitoring metric. You can use the techniques shown in this tutorial for implementing your own monitoring metric to scale a managed instance group.

Objectives

  • Deploy an autoscaling Compute Engine instance group.
  • Create a custom metric used to scale the instance group.
  • Use the Google Cloud Platform Console to visualize the custom metric and instance group size.

Costs

This tutorial uses billable components of GCP including:

  • Cloud Storage
  • Compute Engine
  • Stackdriver Monitoring

    Monitoring features custom metrics, which have associated costs. For more information, see Stackdriver Pricing.

Before you begin

  1. Select or create a GCP project.

    Go to the Project selector page

  2. Make sure that billing is enabled for your Google Cloud Platform project.

    Learn how to enable billing

  3. Enable the Cloud Storage and Stackdriver APIs.

    Enable the APIs

  4. If you don't already have one, create a new Stackdriver account.

Application architecture

This tutorial guides you through creation of a sample autoscaling application. The application uses a Node.js script installed on Compute Engine instances. The script reports a numeric value to a Stackdriver metric. (You do not need to know Node.js or JavaScript to run this tutorial.) In response to the value of the metric, the application autoscales the Compute Engine instance group up or down as needed.

The Node.js script in this tutorial is used as a way to seed a custom metric with values to which the instance group can respond. In a production environment, you would base autoscaling on a metric that is relevant to your use case.

Architecture of application, showing how an instance group uses a template and startup scripts read from Cloud Storage to launch new instances.

The application includes the following components:

  1. Compute Engine instance template—A template used to create each instance in the instance group.
  2. Cloud Storage—A bucket used to host the startup script and other script files.
  3. Compute Engine startup script—A startup script that installs the necessary code components on each instance. The startup script is installed and started automatically when an instance starts. When the startup script runs, it in turn installs and starts code on the instance that writes values to the Stackdriver custom metric.
  4. Compute Engine instance group—An instance group that autoscales based on the Stackdriver Monitoring metric values.
  5. Compute Engine instances—A variable number of Compute Engine instances.
  6. Custom Cloud Storage metric—A custom monitoring metric used as the input value for Compute Engine instance group autoscaling.

Creating the application

Creating the autoscaling application requires downloading the necessary code components, creating a managed instance group, and configuring autoscaling for the managed instance group.

Uploading the script files to Cloud Storage

During autoscaling, the instance group might need to create new Compute Engine instances. When it does, it creates the instances based on an instance template. Each instance needs a startup script. Therefore, the template needs a way to reference the startup script. Compute Engine supports using Cloud Storage buckets as a source for your startup script.

Upload your startup script to a Cloud Storage bucket by following these steps:

  1. Download the following scripts from the GitHub repository for this tutorial:

    • startup.sh
    • writeToCustomMetric.js
    • writeToCustomMetric.sh
    • config.json
    • package.json
  2. In the GCP Console, go to the Cloud Storage page.

    Go to the Cloud Storage page

  3. Click Create bucket and accept the default values to create a new Cloud Storage bucket. For details about how to name a bucket, see the bucket naming guidelines.

  4. Copy the script files to your Cloud Storage bucket.

    1. Click the Cloud Storage bucket that you created to browse to the contents of the bucket.
    2. Click Upload Files, select all of the script files that you downloaded earlier from GitHub (startup.sh, writeToCustomMetric.js, writeToCustomMetric.sh,config.json, package.json), and then upload them.

    After the scripts are uploaded, the files appear in the Cloud Storage bucket:

    Listing of the files in the user's bucket

    The functionality of each of these scripts is explained in the next section.

  5. Make a note of the Cloud Storage file location for the startup script, which will have this format:

    gs://[YOUR_BUCKET_NAME]/startup.sh

Understanding the code components

  • startup.sh—A shell script that installs the necessary components to each Compute Engine instance as the instance is added to the managed instance group.
  • writeToCustomMetric.js—A Node.js snippet that creates a custom monitoring metric whose value triggers scaling. To emulate real-world metric values, this script varies the value over time. In a production deployment, you replace this script with custom code that reports the monitoring metric that you're interested in, such as a processing queue value.
  • config.json—A Node.js config file that specifies the values for the custom monitoring metric and that is used in writeToCustomMetric.js.
  • package.json—A Node.js package file that specifies standard installation and dependencies for writeToCustomMetric.js.
  • writeToCustomMetric.sh—A shell script that continuously runs the writeToCustomMetric.js program on each Compute Engine instance.

Creating an instance template

In this section, you create a template for instances that are created in the instance group using autoscaling. As part of the template, you specify the location (in Cloud Storage) of the startup script that should run when the instance starts.

  1. In the GCP Console, go to the Instance templates page for Compute Engine.

    Go to the Instance templates page

  2. Click Create Instance Template.

  3. Name the instance template autoscaling-instance01 and accept the default values for Machine type, Boot disk, Identity and API access, and Firewall.

    'Create an instance template' image showing the name filled out

  4. Click Management, security, disks, networking, sole tenancy to expand the input options.

  5. In the Metadata section of the Management tab, enter these metadata keys and values:

    • startup-script-url = gs://[YOUR_BUCKET_NAME]/startup.sh
    • gcs-bucket = gs://[YOUR_BUCKET_NAME]

    'Metadata' section of 'Create an instance template' page showing the recommended key/value pairs filled in

  6. Click Create.

Creating the instance group

  1. In the GCP Console, go to the Instance Groups page.

    Go to the Instance Groups page

  2. Click Create a new instance group and name the instance group autoscaling-instance-group-1.

  3. Under Location, select Single-zone.

  4. Under Zone, select your preferred zone.

  5. Under Group type, select Managed instance group.

    'Create instance group' page showing the 'Managed instance group' option selected

  6. Under Instance template, select the instance template you just created.

  7. Set Autoscaling to Off.

    You'll edit the autoscaling setting after the instance group has been created. Leave the other settings at their default values.

    'Create instance group' page showing the 'Autoscaling' option set to 'Off'

  8. Click Create.

You have now created the instance group with a single instance, and that instance should be reporting custom metric values. The next step is to verify the setup.

Verifying that the instance group has been created

  1. In the GCP Console, go to the Instance Groups page.

    Go to the Instance Groups page

  2. Verify that the green status icon is displayed for the instance group named autoscaling-instance-group-1 created using the autoscaling-instance01 template. This indicates that the instance group has been successfully created.

    Listing of instance groups, showing a greens status icon for the 'autoscaling-instance-group-1' group

Verifying that the Node.js script is running

The custom metric custom.googleapis.com/appdemo_queue_depth_01 isn't created until the first instance in the group is created and the instance begins reporting custom metric values.

You can verify that the writeToCustomMetric.js script is running on the first instance in the instance group by checking whether the instance is logging custom metric values.

  1. In the GCP Console, go to the Instance Groups page.

    Go to the Instance Groups page

  2. Click the autoscaling-instance-group-1 to display the instances that are running in the group.

  3. Click any instance. (Because autoscaling has not started additional instances, click the single instance that you currently see.)

    Page that shows details for the instance group, showing a single instance that was started by the instance group

  4. In the VM instance details, page, under Logs click Stackdriver Logging to view the logs for the VM instance.

    Detail of the 'VM instance details' page, showing the 'Logs' heading with the 'Stackdriver Logging' link

  5. Enter nodeapp to filter the logs for this Compute Engine instance.

    If the Node.js script is being executed on the Compute Engine instance, a request is sent to the API, and log entries that say Finished writing time series data appear in the logs. For example, in the following screenshot, entries with this text appear at 10:31:05.000 and 10:30:53.000. If you don't see log entries like this, the Node.js script isn't reporting the custom metric values.

    Listing of log entries

Configure autoscaling for the instance group

After you've verified that the custom metric is successfully reporting data from the first instance, you can configure the instance group to autoscale based on the value of the custom metric.

  1. In the GCP Console, go to the Instance Groups page.

    Go to the Instance Groups page

  2. Select the autoscaling-instance-group-1 group and then click Edit.

    Listing of instance groups, showing the 'autoscaling-instance-group-1' group

  3. Set Autoscaling to On.

    'Instance groups' page in edit mode, with the 'Autoscaling' option set to 'On'

  4. Under Autoscale based on, select Stackdriver monitoring metric.

  5. In the Metric identifier field, enter custom.googleapis.com/appdemo_queue_depth_01.

  6. In the Target field, enter 150.

    When custom monitoring metric values are higher or lower than the Target value, the autoscaler scales the managed instance group, increasing or decreasing the number of instances. The target value can be any double value. This tutorial uses 150 because that value matches the values being reported by the custom monitoring metric.

  7. In the Target mode list, select Gauge.

    The Gauge setting specifies that the autoscaler should compute the average value of the data collected over the last few minutes and compare it to the target value. (By contrast, setting Target mode to DELTA_PER_MINUTE or DELTA_PER_SECOND autoscales based on the observed rate of change rather than an average value.)

  8. For Minimum number of instances enter 1, and for Maximum number of instances enter 5.

    Detail of the 'Instance groups' page in edit mode, showing the recommended settings

  9. Click Save.

Watching the instance group perform autoscaling

The Node.js script varies the custom metric values it reports from each instance over time. As the value of the metric goes up, the instance group scales up by adding Compute Engine instances. If the value goes down, the instance group detects this and scales down by removing instances. (As noted earlier, the script emulates a real-world metric whose value might similarly fluctuate up and down.)

You can see how the instance group is scaling in response to the metric by viewing the Autoscaled size graph.

  1. In the GCP Console, go to the Instance Groups page.

    Go to the Instance Groups page

  2. Click the autoscaling-instance-group-1 instance group in the list.

    The page displays the details about the instance group, including the autoscaling graph.

    The 'Instance groups' page for the selected group, showing a graph for 'Autoscaled size'

    The number of instances depicted in the top graph changes in parallel with the values reported by the custom metric reported in the lower graph.

Cleaning up

After you've finished the tutorial, clean up the resources you created on GCP so you won't be billed for them in the future. The following sections describe how to delete these resources.

Delete the project

The easiest way to eliminate billing is to delete the project you created for the tutorial.

To delete the project:

  1. In the GCP Console, go to the Projects page.

    Go to the Projects page

  2. In the project list, select the project you want to delete and click Delete .
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Deleting all the components

  1. Delete the instance group.
  2. Delete the instance template.
  3. Delete the Cloud Storage bucket.

What's next

Trang này có hữu ích không? Hãy cho chúng tôi biết đánh giá của bạn:

Gửi phản hồi về...