Using the bulk instance API

Linux VMs Windows VMs

This document describes the bulk instance API and how to use it to create multiple, homogeneous VMs that are independent from each other.

Using the bulk instance API is different than aggregating multiple instance insert API requests into one HTTP request by batching API requests. Also, instances created with the bulk instance API are not managed for you.

The following table compares the instances.insert API with the instances.bulkInsert API.

Functionality	`instances.insert`	`instances.bulkInsert`
Zone selection Automatically selected based on considerations such as resource availability and quota	Manual	Automatic when using the regional endpoint
Upfront validation Request fails immediately if it is not feasible	No	With capacity and quota
VM name generation Automatically generated based on a specified name pattern	Manual	Optionally generated automatically
Automatic rollback Request automatically rolled back if Compute Engine cannot create the target number of VMs	No	Optionally enabled
API rate limit How requests affect the API rate limit	Per instance request	Per bulk instance request

If you want to let Compute Engine manage the VMs for you, use managed instance groups.

Considerations

When using the instructions in this document to use the bulk instance API, remember the following:

The bulk instance API lets you issue regional or zonal requests. If the request is regional, Compute Engine determines the zone to create the VMs in based on the zones that have available hardware, taking into consideration the available capacity in each zone as well as any of your reservations. For these regional requests, Compute Engine determines a single zone to place the VMs. To place VMs in different zones, issue separate requests to those zones. This document contains a pseudocode example that shows how to do this.
Requests to the bulk instance API consume the same API rate limit as requests to create single VMs. For the purposes of rate limiting, each bulk instance API request counts as a single request, regardless of the number of VMs being created. If you don't have enough quota, the request fails immediately, you get a rateLimitExceeded error, and Compute Engine does not create any VMs.
To simplify viewing VMs in the Google Cloud Console or to use Cloud Monitoring, consider adding the VMs to an unmanaged instance group. Unmanaged instance groups do not provide management of the VM lifecycle or load balancing. To group the VMs without using an unmanaged instance group, you can use labels.

Pricing

There is no additional charge for following the instructions in this document to use the bulk instance API. As with creating single VMs, billing begins when you create the VMs.

If, when using the bulk instance API, Compute Engine fails to create any VMs, you are only billed for the VMs that Compute Engine successfully creates.

Before you begin

If you want to use the command-line examples in this guide:
1. Install or update to the latest version of the gcloud command-line tool.
2. Set a default region and zone.
If you want to use the API examples in this guide, set up API access.
For VMs and any related resources that you plan to create make sure you have enough quota and the required permissions.

Creating VMs with the bulk instance API

By following the procedures below, you can use the bulk instance API to create multiple VMs in a region or in a zone.

Creating multiple VMs in a region

The following procedures show how to use the bulk instance API to create multiple VMs in a specific region. When doing this, Compute Engine determines the zone.

If you specify a machine type or support for additional hardware such as a GPU or a local SSD, Compute Engine places the VMs in a zone within the region that supports the machine type and the additional hardware.

gcloud

Use the gcloud compute instances bulk create command with the required flags to create multiple VMs in a region.

gcloud compute instances bulk create \
  ( --name-pattern="NAME_PATTERN" | --predefined-names=[PREDEFINED_NAMES] )\
  --region=REGION \
  --count=COUNT \
  [ --min-count=MIN_COUNT ]

Replace the following:

NAME_PATTERN: the name pattern for the VMs. Use a sequence of hash (#) characters for Compute Engine to replace with a sequence of numbers. For example, using vm-# for the name pattern generates VMs with names vm-1, vm-2, and so on, up to the number of VMs specified by --count, which must be less than or equal to the number of VMs that the name pattern allows.

When using a name pattern, Compute Engine tries to avoid name conflicts by checking the names of existing VMs created from previous requests. Additionally, there might be name conflicts if you are using Global DNS. To avoid name conflicts from using Global DNS, switch to Zonal DNS. If you cannot switch to Zonal DNS, avoid using the same name pattern across different regions.
PREDEFINED_NAMES: a list of predefined names for the VMs to create. If using this flag and specifying COUNT, COUNT must equal the number of names provided.
REGION: the region to create the VMs in.
COUNT: the number of VMs to create. This must be less than or equal to the number of VMs allowed by NAME_PATTERN. Or, if using --predefined-names, you do not have to specify COUNT, but if you do, it must be equal to the number of names provided.

MIN_COUNT: an optional flag that specifies the minimum number of VMs to create. The following table describes the behavior of the request depending on how you set this flag:

Setting	Description
Not set	Default value is `COUNT`, and if Compute Engine can't create `COUNT` VMs, the request isn't validated and no VMs are created.
Set to `1`	Compute Engine creates as many VMs as possible, up to `COUNT`.
Greater than `1` and less than `COUNT`	Compute Engine attempts to create at least `MIN_COUNT` VMs up to a maximum of `COUNT` VMs. The request succeeds if Compute Engine creates at least `MIN_COUNT` VMs.

API

Use the instances.bulkInsert method with the required parameters to create multiple VMs in a region.

POST https://www.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/instances/bulkInsert

{
  ...
  "namePattern": "NAME_PATTERN",
  "perInstanceProperties": {
    "PREDEFINED_NAME_1": {},
    "PREDEFINED_NAME_2": {},
    ...
  },
  "count": COUNT,
  "minCount": MIN_COUNT,
  ...
}

Replace the following:

PROJECT_ID: the project ID.
REGION: the region to create the VMs in.
NAME_PATTERN: the name pattern for the VMs. Specify either this or perInstanceProperties. Use a sequence of hash (#) characters for Compute Engine to replace with a sequence of numbers. For example, using vm-# for the name pattern generates VMs with names vm-1, vm-2, and so on, up to the number of VMs specified by --count, which must be less than or equal to the number of VMs that the name pattern allows.

When using a name pattern, Compute Engine tries to avoid name conflicts by checking the names of existing VMs created from previous requests. Additionally, there might be name conflicts if you are using Global DNS. To avoid name conflicts from using Global DNS, switch to Zonal DNS. If you cannot switch to Zonal DNS, avoid using the same name pattern across different regions.
PREDEFINED_NAME_1, PREDEFINED_NAME_2, ...: a list of predefined names for the VMs to create. Specify either this or namePattern. If using this flag and specifying COUNT, COUNT must equal the number of names provided.
COUNT: the number of VMs to create. This must be less than or equal to the number of VMs allowed by NAME_PATTERN. Or, if using perInstanceProperties, you do not have to specify COUNT, but if you do, it must be equal to the number of names provided.

MIN_COUNT: an optional flag that specifies the minimum number of VMs to create. The following table describes the behavior of the request depending on how you set this flag:

Setting	Description
Not set	Default value is `COUNT`, and if Compute Engine can't create `COUNT` VMs, it rolls back the request.
Set to `1`	Compute Engine creates as many VMs as possible, up to `COUNT`.
Greater than `1` and less than `COUNT`	Compute Engine attempts to create at least `MIN_COUNT` VMs up to a maximum of `COUNT` VMs. The request succeeds if Compute Engine creates at least `MIN_COUNT` VMs.

Creating multiple VMs in a specific zone

The following procedures show how to use the bulk instance API to create multiple VMs in a zone.

gcloud

Use the gcloud compute instances bulk create command with the required flags to create multiple VMs in a zone.

gcloud compute instances bulk create \
  ( --name-pattern="NAME_PATTERN" | --predefined-names=[PREDEFINED_NAMES] )\
  --zone=ZONE \
  --count=COUNT \
  [ --min-count=MIN_COUNT ]

Replace the following:

NAME_PATTERN: the name pattern for the VMs. Use a sequence of hash (#) characters for Compute Engine to replace with a sequence of numbers. For example, using vm-# for the name pattern generates VMs with names vm-1, vm-2, and so on, up to the number of VMs specified by --count, which must be less than or equal to the number of VMs that the name pattern allows.

When using a name pattern, Compute Engine tries to avoid name conflicts by checking the names of existing VMs created from previous requests. Additionally, there might be name conflicts if you are using Global DNS. To avoid name conflicts from using Global DNS, switch to Zonal DNS. If you cannot switch to Zonal DNS, avoid using the same name pattern across different regions.
PREDEFINED_NAMES: a list of predefined names for the VMs to create. If using this flag and specifying COUNT, COUNT must equal the number of names provided.
ZONE: the zone to create the VMs in.
COUNT: the number of VMs to create. This must be less than or equal to the number of VMs allowed by NAME_PATTERN. Or, if using --predefined-names, you do not have to specify COUNT, but if you do, it must be equal to the number of names provided.

MIN_COUNT: an optional flag that specifies the minimum number of VMs to create. The following table describes the behavior of the request depending on how you set this flag:

Setting	Description
Not set	Default value is `COUNT`, and if Compute Engine can't create `COUNT` VMs, it rolls back the request.
Set to `1`	Compute Engine creates as many VMs as possible, up to `COUNT`.
Greater than `1` and less than `COUNT`	Compute Engine attempts to create at least `MIN_COUNT` VMs up to a maximum of `COUNT` VMs. The request succeeds if Compute Engine creates at least `MIN_COUNT` VMs.

API

Use the instances.bulkInsert method with the required parameters to create multiple VMs in a zone.

POST https://www.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/bulkInsert

{
  ...
  "namePattern": "NAME_PATTERN",
  "perInstanceProperties": {
    "PREDEFINED_NAME_1": {},
    "PREDEFINED_NAME_2": {},
    ...
  },
  "count": COUNT,
  "minCount": MIN_COUNT,
  ...
}

Replace the following:

PROJECT_ID: the project ID.
ZONE: the zone to create the VMs in.
NAME_PATTERN: the name pattern for the VMs. Specify either this or perInstanceProperties. Use a sequence of hash (#) characters for Compute Engine to replace with a sequence of numbers. For example, using vm-# for the name pattern generates VMs with names vm-1, vm-2, and so on, up to the number of VMs specified by --count, which must be less than or equal to the number of VMs that the name pattern allows.

When using a name pattern, Compute Engine tries to avoid name conflicts by checking the names of existing VMs created from previous requests. Additionally, there might be name conflicts if you are using Global DNS. To avoid name conflicts from using Global DNS, switch to Zonal DNS. If you cannot switch to Zonal DNS, avoid using the same name pattern across different regions.
PREDEFINED_NAME_1, PREDEFINED_NAME_2, ...: a list of predefined names for the VMs to create. Specify either this or namePattern. If using this flag and specifying COUNT, COUNT must equal the number of names provided.
COUNT: the number of VMs to create. This must be less than or equal to the number of VMs allowed by NAME_PATTERN. Or, if using perInstanceProperties, you do not have to specify COUNT, but if you do, it must be equal to the number of names provided.

MIN_COUNT: an optional flag that specifies the minimum number of VMs to create. The following table describes the behavior of the request depending on how you set this flag:

Setting	Description
Not set	Default value is `COUNT`, and if Compute Engine can't create `COUNT` VMs, it rolls back the request.
Set to `1`	Compute Engine creates as many VMs as possible, up to `COUNT`.
Greater than `1` and less than `COUNT`	Compute Engine attempts to create at least `MIN_COUNT` VMs up to a maximum of `COUNT` VMs. The request succeeds if Compute Engine creates at least `MIN_COUNT` VMs.

Checking the status of VMs created with the bulk instance API

After using the bulk instance API to create multiple VMs, use the following procedures to check the status of the request to the bulk instance API or to check the status of a single VM that was part of the request to the bulk instance API.

Checking the status of a bulk instance request

When checking the status of a request to the bulk instance API, the bulk Operation only returns DONE if Compute Engine either successfully creates the minimum number of specified VMs or rolls back the request.

For information about how to check the status of a request to the bulk instance API, see Handling API responses.

Checking the status of a single VM

The bulk instance API creates Operations for each VM, which contains the VM name and the creation status.

The following procedure shows how to check the status of a single VM that Compute Engine created as part of a group of VMs based on a request sent to the bulk instance API.

gcloud

From the Operation returned by the request to the bulk instance API, get the value of the operationGroupId property.
Use the operationGroupId property as a filter with the gcloud compute operations list command to search across all operations and all zones in the project for VMs associated with the regional or zonal request to the bulk instance API:
```
gcloud compute operations list \
  --filter=(operationGroupId=OPERATION_GROUP_ID)
```
Get the rest of the VM's properties by doing any of the following:
- From the list of operations, the targetLink represents the path of the VM. Use the gcloud compute instances describe method with this path as the name of the VM to get the VM's full properties:
```
gcloud compute instances describe VM_NAME
```
- Use the gcloud compute instances list method with a filter that includes the names of the VMs from the list of operations:
```
gcloud compute instances list VM_NAME \
  --filter=(name=VM_NAME_1) OR (name=VM_NAME_2)
```
- Use the gcloud compute instances list command to get the properties of VMs from across all zones and regions, and filter by either a label that is unique to the instances or by their names:
```
gcloud compute instances list \
  --filter=(name=VM_NAME_1) OR (name=VM_NAME_2)
```

API

From the Operation returned by the request to the bulk instance API, get the value of the operationGroupId property.
Use the operationGroupId property as a filter to get the list of VM operations associated with the regional or zonal request to the bulk instance API:
- If you sent a regional request to the bulk instance API, use the globalOperations.aggregatedList method to search across all operations and all zones in the project:
```
GET https://compute.googleapis.com/compute/v1/projects/PROJECT_NAME/aggregated/operations?filter=(operationGroupId=OPERATION_GROUP_ID)
```
- If you sent a zonal request to the bulk instance API, use the zoneOperations.get method to list the operations in that zone:
```
GET https://compute.googleapis.com/compute/v1/projects/PROJECT_NAME/zones/ZONE/instances/bulkInsert?filter=(operationGroupId=OPERATION_GROUP_ID)
```
Get the rest of the VM's properties by doing any of the following:
- From the list of operations, the targetLink represents the path of the VM. Use the instances.get method with this path as the name of the VM to get the VM's full properties:
```
GET https://compute.googleapis.com/compute/v1/projects/PROJECT_NAME/zones/ZONE/instances/VM_NAME
```
- Use the instances.get method with a filter that includes the names of the VMs from the list of operations:
```
GET https://compute.googleapis.com/compute/v1/projects/PROJECT_NAME/zones/ZONE/instances?filter=(name=VM_NAME_1) OR (name=VM_NAME_2)
```
- Use the instances.aggregatedList method to get the properties of VMs from across all zones and regions, and filter by either a label that is unique to the instances or by their names:
```
GET https://compute.googleapis.com/compute/v1/projects/PROJECT_NAME/aggregated/instances?filter=(name=VM_NAME_1) OR (name=VM_NAME_2)
```

Pseudocode examples

The following pseudocode examples show how to customize requests to the bulk instance API.

Creating multiple VMs in various zones in a region

By default, when you use the bulk instance API to create multiple VMs, Compute Engine places all of them in the same zone. The following procedure describes pseudocode that shows how to create up to 1000 VMs that can be in different zones in a region.

Specify the number of VMs to create, and initialize a counter to keep track of the number of VMs as they are created.
```
nTarget = 1000
nCreated = 0
```
Initialize a data structure with the available zones within a region. Use the zones.list method to get a list of zones available to your project.
```
zones = list of zones in region
```
Iterate through the list of zones. On each iteration call the bulk API with minCount equal to 1 and update the count of the number of total created VMs. With minCount equal to 1, Compute Engine creates as many VMs as possible, up to the nTarget.
```
for each zone in zones:
  call bulk API: zone=zone, minCount=1, count=(nTarget - nCreated)
  nCreated += (# of VMs created)
  if (vmsCreated == targetN)
    break
```

Creating multiple VMs in a zone in any region

The following pseudocode describes how to use the bulk instance API to create up to 1000 VMs in a zone within the specified region. When attempting to create multiple VMs in one region out of a designated set of regions, the request first checks for capacity. If there is not enough capacity, the request fails immediately and tries again with the next region in the set.

Specify the number of VMs to create within a zone.
```
nTarget = 1000
```

Designate the regions to attempt to create the VMs in.

acceptableRegions = ["us-central1", "us-east1", "us-west1"]

Iterate through the regions, and attempt to create the VMs in each region until successful.

for region in acceptableRegions:
  call bulk API: region=region, count=nTarget
  if request succeeds and the operation succeeds:
    break

Creating multiple VMs in a zone on a machine type

The following pseudocode describes how to use the bulk instance API to create multiple VMs in a zone on one of the specified machine types. When attempting to create multiple VMs all of the same machine type, the request first checks for availability of those machine types. If there is not enough of the machine type available, the request fails immediately and tries again with the next machine type.

Specify the number of VMs to create and the region to create them in.
```
nTarget = 1000
region = "us-central1"
```
Specify the machine families to attempt to create the VMs on.
```
acceptableMachineFamilies = ["n2","c2","e2","n1"]
```

Iterate through the set of machine types and attempt to create the VMs on the machine type until successful.

for family in acceptableMachineFamilies:
  call bulk APIs: region=region, count=nTarget, machineFamily=family
  if request succeeds and the operation succeeds:
    break

Creating more than 1000 VMs in a zone

When using the bulk instance API to create VMs, you can only create 1000 VMs with each request. The following pseudocode describes how to create more than 1000 VMs in a zone by issuing multiple requests.

Specify the number of VMs to create, a counter to keep track of the total number of created VMs, the region to create the VMs in, and a variable to store the zone that Compute Engine creates the VMs in.
```
nTarget = 10000
nCreated = 0
region = "us-central1"
targetZone = ""
```
Issue an initial request to create 1000 VMs, save the zone returned by the request, and update the counter of the number of VMs created.
```
call bulk API: region=region, count=1000
targetZone = zone chosen by bulk API
nCreated += # of VMs created
```
Continue issuing requests to create up to 1000 VMs at a time in the zone until Compute Engine creates the specified number of VMs.
```
while(nTarget - nCreated > 0):
  call bulk API: zone=targetZone, count=1000
  nCreated += # of VMs created
```

Viewing detailed information about multiple VMs

The following procedure describes pseudocode that you can use to get the details of instances created by using predefined names with the bulk instance API:

Specify the number of VMs to create, the zone to create them in, and a data structure to store the names in.
```
nTarget = 1000
targetZone = "us-central-1a"
names = []
```
Generate the patterned names for the VMs and add the names to the data structure.
```
for n in range(0,1000):
  names.push("instance-%d".format(n))
```

Create the VMs, and use perInstanceProperties to specify the names.

call bulk API(zone=targetZone, count=nTarget, names=perInstanceProperties)

Get the details of the VMs by using the instances.list method with a filter for the names of VMs to return the details about.
```
instances.list(filter=(name = "instance-1") OR (name = "instance-2") ...)
```

Limitations

Consult the following list of limitations before using the instance bulk API:

For each project, the limit on the number of concurrent running operations with the bulk instance API is 10.
For each request, the limit on the number of VMs that you can create is 1000. For an example of how to issue multiple requests to create more than 1000 VMs, see the pseudocode in this document.
All VM properties, except their names, must be identical.
You cannot use any VM properties that are mutually exclusive between VMs, which include but are not limited to the following:
- Custom hostnames
- Static external IPs
- Static internal IPs
You cannot use the instance bulk API to create VMs that use the following:
- Disks that are protected by customer-managed encryption keys
- Disks that are protected by customer-supplied encryption keys
- Machine images
- Sole-tenant node affinity labels
If you are using Global DNS there might be name conflicts because the zone is not included in the fully qualified domain name (FQDN). To avoid this, use Zonal DNS. If you cannot switch to Zonal DNS, avoid using the same name pattern across different regions.
Rarely, after Compute Engine validates a request to the instance bulk API, the request might fail, for example, if Compute Engine can't create the VMs because another request consumed the available quota. In this case, Compute Engine rolls back the request and deletes any VMs that it already created.

Audit logs

Compute Engine logs information about the bulk instance API to the Admin Activity audit log at the beginning of the request when and at the end of the request.

Compute Engine also creates separate audit logs for each VM. You can find the audit log for a single VM by matching the value of protoPayload.resourceName with the VM name generated by the naming pattern.

What's next

Learn how to connect to Linux VMs.
Learn how to connect to Windows Server VMs.