This page describes configuring a virtual machine (VM) instance with multiple network interfaces.
By default, every instance in a VPC network has a single network interface. Use these instructions to create additional network interfaces. Each interface is attached to a different VPC network, giving that instance access to different VPC networks in Google Cloud. You cannot attach multiple network interfaces to the same VPC network.
If you do not require additional network interfaces, follow the instructions in Creating and Starting an Instance.
For more information about multiple network interfaces and how they work, see Multiple Network Interfaces.
Specifications
Requirements
You can only configure a network interface when you create an instance.
Each network interface configured in a single instance must be attached to a different VPC network, and each interface must belong to a subnet whose IP range does not overlap with the subnets of any other interfaces. The attached network can be a standalone VPC network or a Shared VPC network.
The additional VPC networks that the multiple interfaces will attach to must exist before you create the instance. See Using VPC Networks for instructions on creating additional VPC networks.
You cannot delete a network interface without deleting the instance.
Each instance must have at least one network interface. The maximum number of network interfaces per instance is 8, depending on the instance's machine type.
nic0
is required and must be configured at instance creation.nic1
tonic7
are optional, but must also be configured at instance creation.- For more information, see Maximum number of interfaces.
Every interface can optionally have an external IP address.
Google Cloud's DHCP server sends a default route (RFC 3442, "classless static route") only to the default NIC,
nic0
. If a default route is needed on another NIC, you must configure it in the instance. While changing the default route in the instance, use the serial console in order to avoid losing connectivity to the VM.All VM interfaces must connect to networks in the same project as the VM or to a Shared VPC shared with the project.
Limitations
- You cannot add or remove network interfaces from an existing VM.
- IP forwarding is enabled at the VM level and cannot be applied to individual interfaces.
Permissions, IAM, and instances with multiple network interfaces
To create an instance with multiple network interfaces, you must have one of the following roles:
- Project owner or editor role
compute.instanceAdmin.v1
role
Creating and deleting instances and instance templates with multiple
interfaces in a a project not using Shared VPC) environment: A user
with the OWNER, EDITOR, or compute.instanceAdmin.v1
role at the project level
can create an instance with multiple interfaces associated with VPC networks and
subnets that are part of that same project.
Creating and deleting instances and instance templates with multiple
interfaces in Shared VPC environments:
A user with the
OWNER, EDITOR, or compute.instanceAdmin.v1
role at the project level can
create an instance with multiple interfaces. If any of the interfaces is
associated with a subnet in a Shared VPC host project, you must also
have the compute.networkUser
role at the Shared VPC host project
level, or at the Shared VPC subnet level.
To learn more about permissions, read the Compute Engine IAM documentation
Using multiple network interfaces with different VPC network types
- Multiple network interfaces are not supported with legacy networks.
- When you configure multiple network interfaces, attach each interface to an auto mode VPC network or custom mode VPC network.
Network interface IP address allocation
- You must configure each network interface with a primary internal IP
address that is allocated from the primary IP address range of a subnet.
- The primary internal address allocated to each of the interfaces in a single instance must be different.
- You can choose to configure each of the virtual interfaces (NIC) with a unique external IP address. External addresses can be ephemeral or reserved.
Maximum number of network interfaces
- The maximum number of virtual interfaces allowed per instance is 8.
- The number of interfaces allowed in an instance is dependent on the instance's machine type and the number of vCPUs.
In general, the following rules apply for n1-standard-x, n1-highmem-x, n1-highcpu-x, f1-micro, g1-small and custom VMs:
- 2 network interfaces for VM <= 2vCPU
- 1 network interface per vCPU for VM > 2vCPUs, with a cap of max 8 network interfaces per VM
Use the following table to determine how many network interfaces can be attached to an instance:
Type of instance | Number of vCPU | Number of virtual NICs |
---|---|---|
n1-standard-x, n1-highmem-x, n1-highcpu-x, n1-highmem-x, n1-highcpu-x | As specified by the value of "x". | 1 NIC per value of "x", with minimum of 2 NICs and a maximum of 8 NICs |
f1-micro | 0.2 vCPU | 2 NICs |
g1-small | 0.5 vCPU | 2 NICs |
Custom Instances | Configurable from 1 vCPU to 64 vCPU | 1 NIC per vCPU, with minimum of 2 NICs and a maximum of 8 NICs |
Creating virtual machine instances with multiple network interfaces
Creating an instance with a single interface is unchanged. For general instructions on creating instances, see Creating and Starting an Instance.
You can add multiple network interfaces when you create an instance:
- On the Console, you add network interfaces on the Create an Instance page, in the Networking -> Network interfaces section.
- On the gcloud command line tool, use the
instances create
command. Include the--network-interface
flag for each interface, followed by any appropriate networking keys, such as([network | subnet], private-network-ip, address)
.
The first interface is always created as nic0
and it is always the
default interface. This is important for some other aspects of Google Cloud networking.
For example, Google Cloud load balancers (except for Internal TCP/UDP
Load Balancing) only distribute traffic to
nic0
.
Console
- Go to the Create an instance page in the Google Cloud Console.
Go to Create an instance - Complete the sections for the instance name, zone, machine type, and other basic aspects of the instance.
- Complete fields on the Management and Disks tabs.
- On the Networking tab, click Add network interface.
- Choose a network.
- If there are multiple subnets in the VPC network, choose a subnet.
- To assign a custom internal IP address to the interface, on the Internal IP drop-down menu, choose Custom and then type in the IP address.
- To indicate that you do not want an external IP address, on the External IP drop-down menu, choose None.
- To assign a static external IP address, on the External IP drop-down menu, choose New static IP, type in the Name and Description and click Reserve.
- To enable IP forwarding, choose On on the IP forwarding drop-down menu.
- To add more network interfaces, click Add network interface and follow steps 5-10 above.
gcloud
Create network interfaces on a new instance by using the instances create
command. Include the --network-interface
flag for each interface,
followed by any appropriate networking keys, such as
[network,subnet],private-network-ip,address)
.
This snippet only illustrates the --network-interface
flag, one of
many possible parameters you can specify when creating an instance. For
other flags, see the gcloud
reference for the
instances create
command.
See the Maximum number of network interfaces table
to see what machine types support the number of network interfaces you need.
gcloud compute instances create INSTANCE_NAME \ --zone [ZONE] \ [--network-interface [network=NETWORK,subnet=SUBNET], [address=EXT_IP_NAME | no-address], [private-network-ip=INT_NETWORK_IP] ...]
Replace the placeholders with valid values:
ZONE
: the zone where the instance is created.NETWORK
: the network where the interface will attach.SUBNET
: the subnet where the interface will attach.EXT_IP_NAME
: assigns the named external IP address to the interface. You must have previously reserved an external address. Specify "no-address" instead ofaddress=EXT_IP_NAME
if you don't want the interface to have an external IP address. Specifyaddress=''
if you want the interface to receive an ephemeral external IP address.INT_NETWORK_IP
: the internal IP address you want the interface to have in the target subnet. Omit if you just want any valid address assigned.
API
Create a VM instance with multiple network interfaces.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances { "networkInterfaces": [ { "subnetwork": "SUBNET_URL" }, for each interface, specify a network... ], other instance settings... }
Replace the placeholders with valid values:
- SUBNET_URL is the URL of the subnet where the network interface will be located.
- PROJECT_ID is the ID of the project that will contain the instance.
- ZONE the zone that will contain the instance.
For more information, refer to the
instances.insert
method.
Example
Sample command for creating an instance with three network interfaces:
gcloud compute instances create vm1 --machine-type=n1-standard-4 \ --network-interface '' \ --network-interface subnet=net1-subnet-a,private-network-ip=10.128.0.2,address=my-external-address \ --network-interface subnet=net2-subnet-b,private-network-ip=10.129.0.2,no-address
The interfaces are created as follows:
nic0
with all-default settings. Associated with default VPC network, with an automatically allocated internal IP address and an ephemeral external IP addressnic1
belongs to net1-subnet-a, with an internal IP address of 10.128.0.2 and a static external IP address.nic2
belongs to net2-subnet-b, with an internal IP address of 10.129.0.2 and no external IP address
For a complete description of the gcloud compute instances create
command and the --network-interface
flag, read the documentation for the
command.
You can use the IP address from the NIC that you have added to set up DNS forwarding. To learn more about configuring Cloud DNS forwarding zones, see Forwarding zones.
Configuring multiple network interfaces for instance groups
All of the instances within a managed instance group are restricted to a single subnet. All of the instances within an unmanaged instance group are restricted to a single subnet when they were used for load balancing. See the VPC networks and load balancing documentation for more information.
Because of these restrictions, there is only one interface per instance. Google Cloud performs validations based on the subnet associated with the only interface in the instance.
When you configure multiple interfaces for instances in a managed or unmanaged
instance group, Google Cloud continues to validate these conditions
for the default nic0
interface.
Creating an instance template with multiple interfaces
The process for creating an instance template that uses a single interface per
instance is unchanged and is described in the gcloud reference for the
instance-templates create
command.
To configure multiple network interfaces for an instance template, set the
--network-interface
flag for each network interface you want to create:
gcloud compute instance-templates create NAME [--description DESCRIPTION] \ [--network-interface [network NETWORK; default="default" | subnet SUBNET][address ADDRESS| no-address] \ [--network-interface [network NETWORK; default="default" | subnet SUBNET][address ADDRESS | no-address] \ .....
You can assign each interface the following networking keys: network, subnet,
address
.
For example, if you run the following command, each instance will have three network interfaces:
gcloud compute instance-templates create template-1 \ --network-interface subnet=net0-subnet-a \ --network-interface subnet=net1-subnet-b,no-address \ --network-interface subnet=net2-subnet-c,no-address \ --region us-central1`
The interfaces are as follows:
nic0
belongs to net0-subnet-a and has an automatically allocated external IP addressnic1
belongs to net1-subnet-b and does not have an external IP addressnic2
belongs to net2-subnet-c and does not have an external IP address
For complete information on the flags and syntax, see
instance-templates create
command.
Configuring Policy Routing
For Google supported images, when you need a secondary network interface
(an interface other than nic0
) to communicate with any IP address not local
to the primary subnet range of that secondary interface's associated subnet,
you need to configure policy routing to ensure that egress packets will leave
through the correct interface. In such cases, you must configure a separate
routing table for each network interface using policy routing.
Linux
Follow these steps to configure policy routing for a Linux-based instance with multiple interfaces:
Connect to an instance configured with multiple network interfaces:
gcloud compute ssh multinic-vm
Configure policy routing with ifconfig for nic1. The example below assumes that Google Cloud has assigned the internal IP address 192.168.0.2 to
nic1
and the subnet default gateway is 192.168.0.1.sudo ifconfig eth1 192.168.0.2 netmask 255.255.255.255 broadcast 192.168.0.2 mtu 1430 echo "1 rt1" | sudo tee -a /etc/iproute2/rt_tables sudo ip route add 192.168.0.1 src 192.168.0.2 dev eth1 table rt1 sudo ip route add default via 192.168.0.1 dev eth1 table rt1 sudo ip rule add from 192.168.0.2/32 table rt1 sudo ip rule add to 192.168.0.2/32 table rt1
Repeat the commands in step 2 for additional interfaces on the instance (nic2, nic3.... nic7).
Ubuntu 18.04 or higher
With the switch from ifupdown to Netplan as the default network configuration utility starting with Ubuntu 18.04 LTS, the ifconfig package no longer comes pre-installed. Additionally, changes you make to Netplan's network configuration files will not persist over Compute Engine VM reboots.
There are two workarounds you can use to configure routing tables for each network interface in Ubuntu 18.04 or greater:
Create a startup script which sets a routing table for each network interface at VM startup. Learn more about how to create startup scripts.
Using a startup script will not work if you are mounting a file share in your VM because they only run after partitions are mounted. This would lead to your file share always being mounted on the default interface before the creation of the routing table by your startup script.
To mount a file share on a VM through a secondary network, namely an interface other than
nic0
, you must configure the client VM's routing policy to ensure that the file share mounts through the correct network interface. To do this, modify the following files:- In
/etc/default/instance_configs.cfg
, set thesetup
flag forNetworkInterfaces
tofalse
:[NetworkInterfaces] dhclient_script = /sbin/google-dhclient-script dhcp_command = ip_forwarding = true setup = false
In
/etc/network/interfaces
, add the following lines to the interface of interest:auto eth1 iface eth1 inet dhcp up ip route add filestore-reserved-address-range via default-gateway-of-nic-to-filestore
where:
filestore-reserved-address-range
is the reserved address range for use by the VPC instance.default-gateway-of-nic-to-filestore
is the default gateway IP address of the NIC connected to the VPC network that is shared with the VPC instance.
For more information on mounting a file share on a VM instance, see Mounting file shares on Compute Engine.
- In
Troubleshooting
I am not able to create VM with multiple interfaces
You may see one of the following error messages:
Invalid value for field 'resource': ''. Too many network interfaces. The maximum number of network interfaces allowed for this machine type is.
If you get this error message, you are trying to create more interfaces than the maximum interfaces supported by your instance machine type. See the maximum number of interfaces table.
Networks must be distinct for NICs attached to a VM.
If you get this message, you are trying to create more than one interface in the same network. Each network interface must be attached to a different VPC network.
Subnetwork CIDR ranges must be non-overlapping for NICs attached to a VM.
If you get this message, the CIDR ranges associated with your VM's interfaces are overlapping. These CIDR ranges include all primary ranges of subnets associated with VM interfaces as well as secondary ranges used for alias IP ranges. Each interface belongs to a subnet, each in a different VPC network, that must not overlap with subnets of other interfaces. For example, if you are trying to create your instance in region
us-west1
, you can check subnet primary CIDR ranges using the following command or the Google Cloud Console.gcloud compute networks subnets list --regions us-west1 NAME REGION NETWORK RANGE default us-west1 default 10.138.0.0/20 overlapping-subnet us-west1 test-network 10.138.8.0/24
To check secondary subnet CIDR ranges, use the following command or the Google Cloud Console.
gcloud compute networks subnets describe overlapping-subnet --region us-west1
... ipCidrRange: 10.128.8.0/24 ... secondaryIpRanges: - ipCidrRange: 10.138.8.0/24 rangeName: conflicting-range
Multiple network interfaces are not supported on legacy networks.
If you get this message, you are trying to create your instance in a legacy network. Instances with multiple interfaces are not supported with legacy networks. You can check if a network is legacy network using the following command or the Google Cloud Console. The Mode field indicates the type of network.
gcloud compute networks list NAME MODE IPV4_RANGE GATEWAY_IPV4 default auto legacy-network legacy 10.240.0.0/16 10.240.0.1 test-network custom
Required 'compute.instances.create' permission for 'projects/PROJECT_ID/zones/ZONE/instances/test-inst'
If you get this message, the account you are logged in with does not have IAM permissions necessary for creating an instance. Refer to IAM Permissions for detailed information on the roles required for creating instances. You can check if the IAM policy associated with your project grants you any of the following roles:
OWNER
,EDITOR
orcompute.instanceAdmin.v1
. For creating instances in Shared VPC you would additionally need thecompute.networkUser
role. In the below example, the accountemail2@gmail.com
doesn’t have sufficient IAM permissions to create an instance. For more detailed instructions see the Granting, Changing, and Revoking Access to Resources IAM guide.gcloud projects get-iam-policy PROJECT_ID bindings: - members: - user:email1@gmail.com role: roles/owner - members: - serviceAccount:our-project-123@appspot.gserviceaccount.com - serviceAccount:123456789012-compute@developer.gserviceaccount.com role: roles/editor - members: - user:email2@gmail.com role: roles/viewer etag: BwUjMhXbSPU= version: 1
You can ask your project owner or editor to either grant you a role of
OWNER
,EDITOR
, orcompute.instanceAdmin.v1
. You would needcompute.networkUser
role as well if you are associating any interface with a subnet belonging to Shared VPC.gcloud projects set-iam_policy --member user:email2@gmail.com --role roles/editor
I am not able to connect to secondary interface's internal IP
Check firewall rules to see that they allow connectivity to the VM secondary interface. You can check by looking at the firewall rules for the network that is attached to the secondary interface by either visiting Google Cloud Console and clicking the appropriate VPC network or by using the following
gcloud
command.gcloud compute firewall-rules list --filter='network:NETWORK_NAME'
Check if you are trying to connect to a secondary interface from an Internet address or from outside of the secondary interface's network. You can only connect to an interface's internal IP from within its network. If you need to reach that interface from outside the network, you can assign an external IP address to your secondary interface.
Check if you are trying to connect to the secondary interface's internal IP from outside the subnet the secondary interface is attached to, either from another subnet of the same network or from a peered network. Multiple Network Interfaces per instance explains the interplay between VPC Peering and VM instances with multiple interfaces. In order to reach secondary interfaces from outside the interface's subnet, you might have to configure routes on the VM. See DHCP behavior with multiple network interfaces for details on how DHCP programs default routes in the VM.
I am not able to connect to secondary interface using external IP
The DHCP server programs a default route only on the primary network interface of the VM. If you want to connect to the secondary interface using an external IP, there are two options. If you only need to connect outside the network on the secondary network interface, you can set a default route on that network interface. Otherwise, you can use Configuring Policy Routing to configure a separate routing table using source-based policy routing in your VM.
I am having connectivity issues when using a netmask that is not /32
By default, the instance metadata server only responds to ARP requests for the default gateway.
To configure interfaces with a netmask other than /32
, you should create an
image using the flag --guest-os-features MULTI_IP_SUBNET
and use it to create
your instance. For example, if you are using a debian-9
based image, you can
create an image using following command:
gcloud compute images create debian-9-multi-ip-subnet \ --source-disk debian-9-disk \ --source-disk-zone us-west1-a \ --guest-os-features MULTI_IP_SUBNET
To view the guest features configured on the image, run the gcloud compute
images describe
command on the guest image.
gcloud compute images describe debian-9-multi-ip-subnet
For more information on creating custom images, see Creating, Deleting, and Deprecating Custom Images.
Troubleshooting using serial console
It is often a good idea to enable serial console on the VM to debug configuration-related issues. You can enable serial console for interactive debugging by following steps in interacting with serial console.