Reduce latency by using compact placement policies

This document explains how to reduce network latency among your virtual machine (VM) instances by creating and applying compact placement policies to them.

A compact placement policy specifies that your VMs should be physically placed closer to each other. This can help improve performance and reduce network latency among your VMs when, for example, you run high performance computing (HPC), machine learning (ML), or database server workloads.

Before you begin

  • If you haven't already, set up authentication. Authentication is the process by which your identity is verified for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine as follows.

    Select the tab for how you plan to use the samples on this page:

    gcloud

    1. Install the Google Cloud CLI, then initialize it by running the following command: 

      gcloud init
    2. Set a default region and zone.

    REST

    To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

      Install the Google Cloud CLI, then initialize it by running the following command: 

      gcloud init

    For more information, see Authenticate for using REST in the Google Cloud authentication documentation.

Required roles

To get the permissions that you need to create and apply a compact placement policy to VMs, ask your administrator to grant you the Compute Instance Admin (v1) (roles/compute.instanceAdmin.v1) IAM role on your project. For more information about granting roles, see Manage access.

This predefined role contains the permissions required to create and apply a compact placement policy to VMs. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to create and apply a compact placement policy to VMs:

  • To create placement policies: compute.resourcePolicies.create on the project
  • To apply a placement policy to existing VMs: compute.instances.addResourcePolicies on the project
  • To create VMs:
    • compute.instances.create on the project
    • To use a custom image to create the VM: compute.images.useReadOnly on the image
    • To use a snapshot to create the VM: compute.snapshots.useReadOnly on the snapshot
    • To use an instance template to create the VM: compute.instanceTemplates.useReadOnly on the instance template
    • To assign a legacy network to the VM: compute.networks.use on the project
    • To specify a static IP address for the VM: compute.addresses.use on the project
    • To assign an external IP address to the VM when using a legacy network: compute.networks.useExternalIp on the project
    • To specify a subnet for the VM: compute.subnetworks.use on the project or on the chosen subnet
    • To assign an external IP address to the VM when using a VPC network: compute.subnetworks.useExternalIp on the project or on the chosen subnet
    • To set VM instance metadata for the VM: compute.instances.setMetadata on the project
    • To set tags for the VM: compute.instances.setTags on the VM
    • To set labels for the VM: compute.instances.setLabels on the VM
    • To set a service account for the VM to use: compute.instances.setServiceAccount on the VM
    • To create a new disk for the VM: compute.disks.create on the project
    • To attach an existing disk in read-only or read-write mode: compute.disks.use on the disk
    • To attach an existing disk in read-only mode: compute.disks.useReadOnly on the disk
  • To create an instance template: compute.instanceTemplates.create on the project
  • To create a regional or zonal MIG: compute.instanceGroupManagers.create on the project
  • To view the details of a VM: compute.instances.get on the project

You might also be able to get these permissions with custom roles or other predefined roles.

Create a compact placement policy

When creating a compact placement policy, you can optionally specify one or both of the following:

  • A maximum distance value. This lets you control the distance among your VMs when you have strict network latency requirements. Depending on the machine type and zone of the VMs you apply the policy to, the following conditions apply:

    • If a zone has capacity available, then a compact placement policy with a higher maximum distance (such as 3) might cause your VMs to be placed closer to each other.

    • If a zone lacks capacity, then a compact placement policy with a lower maximum distance (such as 1) is more likely to prevent the policy from being applied to some VMs.

    Additionally, maximum distance values affect the number of VMs you can apply the compact placement policy to, as well as the VMs' host maintenance policy. For more information, see Restrictions for compact placement policies.

  • A fixed number of VMs. This makes the compact placement policy effective only when you apply it to a specific number of VMs. Attempting to apply the compact placement policy to more VMs than the specified number causes errors. You can't apply compact placement policies with a fixed number of VMs to reservations.

To create a compact placement policy, use the gcloud CLI or REST.

gcloud

To create a compact placement policy, use the gcloud compute resource-policies create group-placement command with the --collocation=collocated flag.

gcloud compute resource-policies create group-placement POLICY_NAME \
    --collocation=collocated \
    --region=REGION

Replace the following:

  • POLICY_NAME: the name of the policy to create.

  • REGION: the region where to create the policy. If you want to apply the compact placement policy to an existing VM, create the policy in a region that contains the zone where the VM is located.

Optionally, you can do one or both of the following:

  • To better control how far apart to place your VMs, use gcloud beta compute resource-policies create group-placement command and include the --max-distance flag.

    gcloud beta compute resource-policies create group-placement POLICY_NAME \
    --collocation=collocated \
    --max-distance=MAX_DISTANCE \
    --region=REGION

    Replace MAX_DISTANCE with the maximum distance configuration for your VMs. The value must be between 1, which specifies to place your VMs in the same rack for the lowest network latency possible, and 3, which specifies to place your VMs in adjacent clusters. If you want to apply the compact placement policy to a reservation, then you can't specify a value of 1.

  • To make the compact placement policy effective only when applied to a specific number of VMs, include the --vm-count flag.

    gcloud compute resource-policies create group-placement POLICY_NAME \
    --collocation=collocated \
    --region=REGION \
    --vm-count=VM_COUNT

    Replace VM_COUNT with the exact number of VMs that you can apply the compact placement policy to. The value must be between 1 and the maximum number of VMs you can apply the compact placement policy to.

REST

To create a compact placement policy, make a POST request to the resourcePolicies.insert method. In the request body, include the collocation field and set it to COLLOCATED.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/resourcePolicies

{
  "name": "POLICY_NAME",
  "groupPlacementPolicy": {
    "collocation": "COLLOCATED"
  }
}

Replace the following:

  • PROJECT_ID: the ID of the project where you want to create the placement policy.

  • REGION: the region where to create the policy. If you want to apply the compact placement policy to an existing VM, create the policy in a region that contains the zone where the VM is located.

  • POLICY_NAME: the name of the compact placement policy to create.

Optionally, you can do one or both of the following:

  • To better control how far apart to place your VMs, make a POST request to the beta.resourcePolicies.insert method. In the request body, include the maxDistance field.

    POST https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/regions/REGION/resourcePolicies

{
  "name": "POLICY_NAME",
  "groupPlacementPolicy": {
    "collocation": "COLLOCATED",
    "maxDistance": "MAX_DISTANCE"
  }
}

    Replace MAX_DISTANCE with the maximum distance configuration for your VMs. The value must be between 1, which specifies to place your VMs in the same rack for the lowest network latency possible, and 3, which specifies to place your VMs in adjacent clusters. If you want to apply the compact placement policy to a reservation, then you can't specify a value of 1.

  • To make the compact placement policy effective only when applied to a specific number of VMs, include the vmCount field in the request body.

    POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/resourcePolicies

{
  "name": "POLICY_NAME",
  "groupPlacementPolicy": {
    "collocation": "COLLOCATED",
    "vmCount": "VM_COUNT"
  }
}

    Replace VM_COUNT with the exact number of VMs that you can apply the compact placement policy to. The value must be between 1 and the maximum number of VMs you can apply the compact placement policy to.

Apply compact placement policies

You can apply a compact placement policy to an existing VM, or when you create VMs, instance templates, MIGs, or reservations of VMs.

To create a Compute Engine resource that specifies a compact placement policy, select one of the following methods:

If you want to specify a compact placement policy when creating a single-project reservation by specifying properties directly, see Create a reservation for a single project instead.

After a compact placement policy is applied to a VM, you can verify the physical location of a VM in relation to other VMs that specify the same policy.

Apply a compact placement policy to an existing VM

You can apply a compact placement policy to an existing VM using the gcloud CLI and REST.

Before applying a compact placement policy to an existing VM, make sure of the following:

  • The VM and the compact placement policy are located in the same project.

  • The VM is located within the region where the compact placement policy exists.

  • The VM specifies a supported machine series and host maintenance policy.

Otherwise, applying the compact placement policy to an existing VM fails.

gcloud

To apply a compact placement policy to an existing VM, use the gcloud compute instances add-resource-policies command with the --resource-policies flag.

gcloud compute instances add-resource-policies VM_NAME \
    --resource-policies=POLICY_NAME \
    --zone=ZONE

Replace the following:

  • VM_NAME: the name of an existing VM.

  • POLICY_NAME: the name of an existing compact placement policy.

  • ZONE: the zone where the VM is located.

REST

To apply a compact placement policy to an existing VM, make a POST request to the instances.addResourcePolicies method with the resourcePolicies field.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/VM_NAME/addResourcePolicies

{
  "resourcePolicies": [
    "projects/PROJECT_ID/regions/REGION/resourcePolicies/POLICY_NAME"
  ]
}

Replace the following:

  • PROJECT_ID: the ID of the project where the compact placement policy and the VM are located.

  • ZONE: the zone where the VM exists, which must be within the region where the compact placement policy is located.

  • VM_NAME: the name of an existing VM.

  • REGION: the region where the compact placement policy is located.

  • POLICY_NAME: the name of an existing compact placement policy.

Create a VM with a compact placement policy

You can create a VM that specifies an existing compact placement policy using the gcloud CLI and REST.

gcloud

To create a VM that specifies a compact placement policy, use the gcloud compute instances create command with the --resource-policies flag.

For example, to create a VM that specifies a c2d-standard-2 machine type, run the following command:

gcloud compute instances create VM_NAME \
    --machine-type=c2d-standard-2 \
    --maintenance-policy=MAINTENANCE_POLICY \
    --resource-policies=POLICY_NAME \
    --zone=ZONE

Replace the following:

  • VM_NAME: the name of the VM to create.

  • MAINTENANCE_POLICY: the host maintenance policy of the VM. If the compact placement policy you specify uses a maximum distance value of 1 or 2, you can only specify TERMINATE. Otherwise, you can specify either MIGRATE or TERMINATE.

  • POLICY_NAME: the name of an existing compact placement policy.

  • ZONE: the zone where to create the VM. You can only create a VM in a zone that is within the region of the compact placement policy you specify.

REST

To create a VM that specifies a compact placement policy, make a POST request to the instances.insert method with the resourcePolicies field.

For example, to create a VM that specifies a c2d-standard-2 machine type, make the following POST request:

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances

{
  "machineType": "zones/ZONE/machineTypes/c2d-standard-2",
  "name": "VM_NAME",
  "resourcePolicies": [
    "projects/PROJECT_ID/regions/REGION/resourcePolicies/POLICY_NAME"
  ],
  "scheduling": {
    "onHostMaintenance": "MAINTENANCE_POLICY"
  }
}

Replace the following:

  • PROJECT_ID: the ID of the project where the compact placement policy is located.

  • ZONE: the zone where to create the VM and where the machine type is located. You can only create a VM in a zone that is within the region of the compact placement policy you specify.

  • VM_NAME: the name of the VM to create.

  • REGION: the region where the compact placement policy is located.

  • POLICY_NAME: the name of an existing compact placement policy.

  • MAINTENANCE_POLICY: the host maintenance policy of the VM. If the compact placement policy you specify uses a maximum distance value of 1 or 2, you can only specify TERMINATE. Otherwise, you can specify either MIGRATE or TERMINATE.

For more information about the configuration options to create a VM, see Create and start a VM instance.

Create VMs in bulk with a compact placement policy

You can create VMs in bulk that specify an existing compact placement policy using the gcloud CLI and REST.

gcloud

To create VMs in bulk that specify a compact placement policy, use the gcloud compute instances bulk create command with the --resource-policies flag.

For example, to create VMs in bulk that all specify a c2d-standard-2 machine type and the same compact placement policy, run the following command:

gcloud compute instances bulk create \
    --async \
    --count=COUNT \
    --machine-type=c2d-standard-2 \
    --maintenance-policy=MAINTENANCE_POLICY \
    --name-pattern=NAME_PATTERN \
    --resource-policies=POLICY_NAME \
    --zone=ZONE

Replace the following:

  • COUNT: the number of VMs to create.

  • MAINTENANCE_POLICY: the host maintenance policy of the VM. If the compact placement policy you specify uses a maximum distance value of 1 or 2, you can only specify TERMINATE. Otherwise, you can specify either MIGRATE or TERMINATE.

  • NAME_PATTERN: the name pattern for the VMs to create. Use the hash character (#) to replace it with a sequence of numbers. For example, specifying vm-# creates VMs with names vm-1, vm-2, and so on, up to the number of VMs specified in COUNT.

  • POLICY_NAME: the name of an existing compact placement policy.

  • ZONE: the zone where to bulk create the VMs. You can only create VMs in zones that are within the region of the compact placement policy you specify.

REST

To create VMs in bulk that specify a compact placement policy, make a POST request to the instances.bulkInsert method with the resourcePolicies field.

For example, to create VMs in bulk that all specify a c2d-standard-2 machine type and the same compact placement policy, make the following POST request:

POST https://www.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/bulkInsert

{
  "count": "COUNT",
  "machineType": "zones/ZONE/machineTypes/c2d-standard-2",
  "namePattern": "NAME_PATTERN",
  "instanceProperties": {
    "resourcePolicies": [
      "projects/PROJECT_ID/regions/REGION/resourcePolicies/POLICY_NAME"
    ],
    "scheduling": {
      "onHostMaintenance": "MAINTENANCE_POLICY"
    }
  }
}

Replace the following:

  • PROJECT_ID: the ID of the project where the compact placement policy is located.

  • ZONE: the zone where to bulk create the VMs and where the machine type is located. You can only create VMs in zones that are within the region of the compact placement policy you specify.

  • COUNT: the number of VMs to create.

  • NAME_PATTERN: the name pattern for the VMs to create. Use the hash character (#) to replace it with a sequence of numbers. For example, specifying vm-# creates VMs with names vm-1, vm-2, and so on, up to the number of VMs specified in COUNT.

  • REGION: the region where the compact placement policy is located.

  • POLICY_NAME: the name of an existing compact placement policy.

  • MAINTENANCE_POLICY: the host maintenance policy of the VM. If the compact placement policy you specify uses a maximum distance value of 1 or 2, you can only specify TERMINATE. Otherwise, you can specify either MIGRATE or TERMINATE.

For more information about the configuration options to create VMs in bulk, see Create VMs in bulk.

Create an instance template with a compact placement policy

You can create an instance template that specifies an existing compact placement policy using the gcloud CLI and REST.

After you create an instance template, you can use it to do the following:

gcloud

To create an instance template that specifies a compact placement policy, use the gcloud compute instance-templates create command with the --resource-policies flag.

For example, to create a global instance template that specifies a c2d-standard-2 machine type and an existing compact placement policy, run the following command:

gcloud compute instance-templates create INSTANCE_TEMPLATE_NAME \
    --machine-type=c2d-standard-2 \
    --maintenance-policy=MAINTENANCE_POLICY \
    --resource-policies=POLICY_NAME

Replace the following:

  • INSTANCE_TEMPLATE_NAME: the name of the instance template.

  • MAINTENANCE_POLICY: the host maintenance policy of the VM. If the compact placement policy you specify uses a maximum distance value of 1 or 2, you can only specify TERMINATE. Otherwise, you can specify either MIGRATE or TERMINATE.

  • POLICY_NAME: the name of an existing compact placement policy.

REST

To create an instance template that specifies a compact placement policy, make a POST request to the instanceTemplates.insert method. In the request body, specify the resourcePolicies field.

For example, to create a global instance template that specifies a c2d-standard-2 machine type and an existing compact placement policy, make the following POST request:

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/instanceTemplates

{
  "name": "INSTANCE_TEMPLATE_NAME",
  "machineType": "zones/ZONE/machineTypes/c2d-standard-2",
  "properties": {
    "resourcePolicies": {
      "POLICY_NAME"
    },
    "scheduling": {
      "onHostMaintenance": "MAINTENANCE_POLICY"
    }
  }
}

Replace the following:

  • PROJECT_ID: the ID of the project where the compact placement policy you want to apply to the instance template is located.

  • ZONE: the zone where the machine type is located.

  • INSTANCE_TEMPLATE_NAME: the name of the instance template.

  • POLICY_NAME: the name of an existing compact placement policy.

  • MAINTENANCE_POLICY: the host maintenance policy of the VM. If the compact placement policy you specify uses a maximum distance value of 1 or 2, you can only specify TERMINATE. Otherwise, you can specify either MIGRATE or TERMINATE.

For more information about the configuration options to create an instance template, see Create instance templates.

Apply a compact placement policy to the VMs in a MIG

After you create an instance template that specifies a compact placement policy, you can use the template to do the following:

If you want to apply a compact placement policy to a MIG, creating or applying the policy to a regional MIG with the any single zone distribution shape is recommended. This way, whenever a regional MIG needs to scale out by creating VMs, it selects the zone where to create the VMs based on your reservations, quotas, and hardware requirements.

Create a MIG with a compact placement policy

You can create a MIG using an instance template that specifies a compact placement policy using the gcloud CLI and REST.

gcloud

To create a MIG using an instance template that specifies a compact placement policy, use the gcloud compute instance-groups managed create command with the --template flag.

For example, to create a regional MIG with an any-single-zone distribution shape, run the following command:

gcloud compute instance-groups managed create MIG_NAME \
    --region=REGION \
    --size=SIZE \
    --target-distribution-shape=any-single-zone \
    --template=INSTANCE_TEMPLATE_NAME

Replace the following:

  • MIG_NAME: the name of the MIG to create.

  • REGION: the region where to create the MIG, which must match with the region where the compact placement policy is located.

  • SIZE: the size of the MIG.

  • INSTANCE_TEMPLATE_NAME: the name of an existing instance template that specifies a compact placement policy.

REST

To create a MIG using an instance template that specifies a compact placement policy, make a POST request to the instanceGroupManagers.insert or regionInstanceGroupManagers.insert methods with the instanceTemplate field set to the name of the existing template.

For example, to create a regional MIG with default VM properties and the any-single-zone distribution shape, make the following POST request:

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/instanceGroupManagers

{
  "name": "MIG_NAME",
  "instanceTemplate": "global/instanceTemplates/INSTANCE_TEMPLATE_NAME",
  "distributionPolicy": {
    "targetShape": "ANY_SINGLE_ZONE"
  },
  "targetSize": SIZE
}

Replace the following:

  • PROJECT_ID: the ID of the project where the compact placement policy and the instance template that specifies the placement policy are located.

  • REGION: the region where to create the MIG, which must match with the region where the compact placement policy is located.

  • MIG_NAME: the name of the MIG to create.

  • INSTANCE_TEMPLATE_NAME: the name of an existing instance template that specifies a compact placement policy.

  • SIZE: the size of the MIG.

For more information about the configuration options to create MIGs, see Basic scenarios for creating MIGs.

Apply a compact placement policy to an existing MIG

You can apply a compact placement policy to an existing MIG using an instance template that specifies the same placement policy using the gcloud CLI and REST.

gcloud

To update a MIG to use an instance template that specifies a compact placement policy, use the gcloud compute instance-groups managed rolling-action start-update command with the --version=template flag.

For example, to update a regional MIG to use an instance template that specifies a compact placement policy and replace the existing VMs from the MIG with new VMs that specify the template's properties, run the following command:

gcloud compute instance-groups managed rolling-action start-update MIG_NAME \
    --region=REGION \
    --type=proactive \
    --version=template=INSTANCE_TEMPLATE_NAME

Replace the following:

  • MIG_NAME: the name of an existing MIG.

  • REGION: the region where the MIG is located. You can only apply the compact placement policy to a MIG that is in the same region.

  • INSTANCE_TEMPLATE_NAME: the name of an existing instance template that specifies a compact placement policy.

REST

To update a MIG to use an instance template that specifies a compact placement policy, and automatically apply the properties of the template and the placement policy to existing VMs in the MIG, make a PATCH request to the instanceGroupManagers.insert or regionInstanceGroupManagers.insert with the instanceTemplate field.

For example, to update a regional MIG to use an instance template that specifies a compact placement policy and replace the existing VMs from the MIG with new VMs that specify the template's properties, make the following PATCH request:

PATCH https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/instanceGroupManagers/MIG_NAME

{
  "instanceTemplate": "global/instanceTemplates/INSTANCE_TEMPLATE_NAME",
  "updatePolicy": {
    "type": "PROACTIVE"
  }
}

Replace the following:

  • PROJECT_ID: the ID of the project where the MIG, the compact placement policy, and the instance template that specifies the placement policy are located.

  • REGION: the region where the MIG is located. You can only apply the compact placement policy to a MIG that is in the same region.

  • MIG_NAME: the name of an existing MIG.

  • INSTANCE_TEMPLATE_NAME: the name of an existing instance template that specifies a compact placement policy.

For more information about the configuration options to update the VMs in a MIG, see Update and apply new configurations to VMs in a MIG.

Verify the physical location of a VM

After you apply a compact placement policy to a VM, you can view how the compact placement policy affected its physical location in relation to other VMs that specify the same policy. This helps you determine if the compact placement policy was correctly applied to a VM and map which VMs are the closest to each other.

You can verify the physical location of a VM that specifies a placement policy using the gcloud CLI and REST.

gcloud

To view the physical location of a VM that specifies a compact placement policy, use the gcloud compute instances describe command with the --format flag.

gcloud compute instances describe VM_NAME \
    --format="table[box,title=VM-Position](resourcePolicies.scope():sort=1,resourceStatus.physicalHost:label=location)" \
    --zone=ZONE

Replace the following:

  • VM_NAME: the name of an existing VM.

  • ZONE: the zone where the VM is located.

The output is similar to the following:

VM-Position

RESOURCE_POLICIES: us-central1/resourcePolicies/example-policy']
PHYSICAL_HOST: /CCCCCCC/BBBBBB/AAAA

The value for the PHYSICAL_HOST field is composed by three parts. These parts each represent the cluster, rack, and host where the VM is located.

When comparing the position of two VMs that specify the same compact placement policy, the more parts in the PHYSICAL_HOST field the VMs share, the closer they are located to each other. For example, if two VMs both specify one of the following sample values for the PHYSICAL_HOST field:

  • /CCCCCCC/xxxxxx/xxxx: the two VMs are placed in the same cluster, which equals a maximum distance value of 2. VMs placed in the same cluster experience low network latency.

  • /CCCCCCC/BBBBBB/xxxx: the two VMs are placed in the same rack, which equals a maximum distance value of 1. VMs placed in the same rack experience lower network latency than VMs placed in the same cluster.

  • /CCCCCCC/BBBBBB/AAAA: the two VMs share the same host. VMs placed in the same host minimize network latency as much as possible.

REST

To view the physical location of a VM that specifies a compact placement policy, make a GET request to the instances.get method.

GET https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/VM_NAME

Replace the following:

  • PROJECT_ID: the ID of the project where the VM is located.

  • ZONE: the zone where the VM is located.

  • VM_NAME: the name of an existing VM that specifies a placement policy.

The output is similar to the following:

{
...
"resourcePolicies": [
  "https://www.googleapis.com/compute/v1/projects/example-project/regions/us-central1/resourcePolicies/example-policy"
],
"resourceStatus": {
  "physicalHost": "/xxxxxxxx/xxxxxx/xxxxx"
},
...
}

The value for the physicalHost field is composed by three parts. These parts each represent the cluster, rack, and host where the VM is located.

When comparing the position of two VMs that specify the same compact placement policy, the more parts in the physicalHost field the VMs share, the closer they are located to each other. For example, if two VMs both specify one of the following sample values for the physicalHost field:

  • /CCCCCCC/xxxxxx/xxxx: the two VMs are placed in the same cluster, which equals a maximum distance value of 2. VMs placed in the same cluster experience low network latency.

  • /CCCCCCC/BBBBBB/xxxx: the two VMs are placed in the same rack, which equals a maximum distance value of 1. VMs placed in the same rack experience lower network latency than VMs placed in the same cluster.

  • /CCCCCCC/BBBBBB/AAAA: the two VMs share the same host. VMs placed in the same host minimize network latency as much as possible.

What's next?