Multiple network interfaces

This page provides an overview of multiple network interfaces in a virtual machine (VM) instance, including how they work and sample configurations. For information about creating configurations that use multiple interfaces, see Create VMs with multiple network interfaces.

VMs with multiple network interface controllers are referred to as multi-NIC VMs.

Google Cloud Virtual Private Cloud (VPC) networks are by default isolated private networking domains. Networks have a global scope and contain regional subnets. VM instances within a VPC network can communicate among themselves by using internal IP addresses as long as firewall rules permit. However, no internal IP address communication is allowed between networks, unless you set up mechanisms such as VPC Network Peering or Cloud VPN.

Every instance in a VPC network has a default network interface. When you configure a network interface, you select a VPC network and a subnet within that VPC network to connect the interface to. You can create additional network interfaces attached to your VMs, but each interface must attach to a different VPC network. Multiple network interfaces let you create configurations in which an instance connects directly to several VPC networks.

Typically, you might require multiple interfaces if you want to configure an instance as a network appliance that does load balancing, Intrusion Detection and Prevention (IDS/IPS), Web Application Firewall (WAF), or WAN optimization between networks. Multiple network interfaces are also useful when applications running in an instance require traffic separation, such as separation of data plane traffic from management plane traffic.

Each interface on a VM is affected by the MTU of the attached network. For more information about interface MTU, see Maximum transmission unit.

Use cases

Use multiple network interfaces when an individual instance needs access to more than one VPC network, but you don't want to connect both networks directly.

• Network and security function: Multiple network interfaces enable virtualized network appliance functions such as load balancers, network address translation (NAT) servers, and proxy servers that are configured with multiple network interfaces. For more details, see Example 1: Networking and security virtual appliances.

• Perimeter isolation (also known as DMZ isolation): An important best practice in tiered networking architectures is to isolate public-facing services from an internal network and its services. Use multiple network interfaces to create configurations where there are separate network interfaces on the instance, one of them accepting public-facing traffic and another handling backend private traffic that has more restrictive access controls.

  Any resources that can be reached from the internet should be separated from your internal network and its services. This drastically limits the scope and damage that a security breach can cause. For example, you can place a second network interface on each web server that connects to a mid-tier network where an application server resides. The application server can also be dual-homed to a backend network where the database server resides. Each dual-homed instance receives and processes requests on the frontend, initiates a connection to the backend, and then sends requests to the servers on the backend network.

  By configuring separate interfaces, one public-facing and another private- facing, you can apply separate firewall rules and access controls to each interface separately and enforce security functions in communications from the public to private domain. For more information, see Example 2: Using third-party appliances in a Shared VPC network scenario.

Specifications

• You can only configure a network interface when you create an instance.

• If you create an instance with multiple network interfaces, each interface must connect to a different VPC network. The networks can be standalone VPC networks or Shared VPC networks.

  However, VPC networks created with an RDMA network profile let you attach RDMA NICs to the same network, where each NIC is attached to a different subnet.

• A network interface must connect to a subnet that has an IP address range that does not overlap with the IP address range of another interface on the same instance.

• If you want to create a VM that is attached to subnets in multiple VPC networks, all of the networks and subnets must exist before you create the instance. For more information about creating networks and subnets, see Create and manage VPC networks.

• You can't delete a network interface without deleting the instance that it is attached to.

• Each instance must have at least one network interface: nic0. The maximum number of interfaces varies depending on the instance's machine type.

• You can configure VM interfaces with IPv4 addresses only, both IPv4 and IPv6 addresses (dual-stack), or IPv6 addresses only (Preview). The stack type of an interface determines the IP addresses that it can have and the type of subnets to which it can connect. For more information, see Stack types.

• Google Cloud's DHCP server sends a default route (RFC 3442, "classless static route") only to the default network interface, nic0. If a default route is needed on another network interface, you must configure policy routing on the instance. For an example of how to do this, see the following tutorial: Configure routing for an additional interface.

• All VM network interfaces must connect to subnets in networks that are in the same project as the VM or to a Shared VPC network that is shared with the project.

Stack types

When you create a network interface, you specify a stack type. The stack type of an interface determines the IP addresses that it can have and the type of subnets to which it can connect. For more information, see the following table:

Interface stack type	IP addresses	Compatible subnet types
IPv4-only (single-stack)	An internal IPv4 address (required) An external IPv4 address (optional)	IPv4-only and dual-stack subnets
IPv4 and IPv6 (dual-stack)	An internal IPv4 address (required) An external IPv4 address (optional) A /96 IPv6 address range, either internal or external, but not both (required)	Dual-stack subnets
IPv6-only (single-stack) (Preview)	A /96 IPv6 address range, either internal or external, but not both (required)	IPv6-only (Preview) and dual-stack subnets

Network interface IP address allocation

For interfaces with IPv4 addresses:

• You must configure each network interface with a primary internal IPv4 address that is allocated from the primary IPv4 address range of a subnet. The primary internal address allocated to each of the interfaces in a single instance must be different.
• Optional: You can configure each of the network interfaces with a unique external IPv4 address. External addresses can be ephemeral or reserved.

For interfaces with IPv6 addresses:

• You must assign a /96 IPv6 address range from the IPv6 address range of a subnet; the first IPv6 address in the range (/128) is configured on the interface. For more information, see IPv6 address assignment.
• The /96 IPv6 address range can either be internal or external, depending on the IPv6 access type of the subnet.

Maximum number of network interfaces

For most machine types, the number of virtual network interfaces scales with the number of vCPUs with a minimum of 2 and maximum of 10.

Use the following table to determine how many network interfaces can be attached to an instance:

Number of vCPU	Number of vNICs
2 or less	2
4	up to 4
6	up to 6
8 or more	up to 8
10 or more	up to 10

Limitations

• You cannot add or remove network interfaces from an existing VM.

• You can't use the Google Cloud console to create multi-NIC configurations for instance templates or instance groups. Use the gcloud CLI or the API instead.

• To understand whether you can change the stack type of a VM's interface, see the following:

  • You can change the stack type of an existing interface from IPv4-only to dual-stack, or from dual-stack to IPv4-only.

  • IPv6-only interfaces (Preview) are only supported when creating new VMs. You can't change the stack type of an existing IPv4-only or dual-stack interface to IPv6-only.

  • Additionally, you can't change the stack type of an IPv6-only interface to dual-stack or IPv4-only.

• An IPv6-only interface (Preview) can only have a single internal or external address range based on the access type of its connected subnet. To create an IPv6-only instance with both an internal and external IP address, you must configure it with two interfaces.

• IP forwarding is enabled at the VM level and cannot be applied to individual interfaces.

• You can't configure multiple network interfaces for Compute Engine bare metal instances, which have a maximum of one vNIC.

• The maximum number of vNICs for the A3 machine series is different than the maximum for most machine types. For more information, see The A3 machine series.

Configuration examples

This section examines several common examples of how to use multiple network interfaces.

Example 1: Networking and security virtual appliances

Networking and security virtual appliances, such as web application firewalls (WAF), security application-level firewalls, and WAN accelerators, are usually configured with multiple virtual interfaces. Each of the multiple interfaces is configured with its own internal IP address and, optionally, with its own external IP address.

Figure 1 describes an example configuration of an application-level firewall that controls traffic from the internet to a VPC network. The application-level firewall is implemented in Compute Engine VMs.

In this example, the default route of the appliance VM has been configured to use nic1.

Figure 1. An instance with a VM appliance has three network interfaces. Each interface is connected to a subnet that is in a different VPC network (click to enlarge).

Provision and configure instances for example 1

The following assumes that subnet0, subnet1, and subnet2 already exist, with non-overlapping ranges.

To create the VM and network interfaces in this example, use the following command:

gcloud compute instances create vm-appliance \
    --network-interface subnet=subnet0,no-address \
    --network-interface subnet=subnet1 \
    --network-interface subnet=subnet2,no-address \
    --machine-type n1-standard-4

This command creates an instance with three network interfaces:

• nic0 is attached to subnet0 and has no external IP address.
• nic1 is attached to subnet1 and has an ephemeral external IP address.
• nic2 is attached to subnet2 and has no external IP address.

Example 2: Using third-party appliances in a Shared VPC network scenario

This setup is useful when you want to share a single set of centralized third-party appliances for workloads or applications that are hosted in different projects. In figure 2, there are four distinct applications—App1, App2, App3, and App4—that are hosted in different service projects. You need them to be protected for all internet ingress, and you need egress traffic to be inspected and filtered in a third-party appliance that is centrally located in the Shared VPC host project.

Figure 2. An instance in a Shared VPC host project hosts a VM appliance. The instance has a network interface for each of four service projects and another interface for the network-perimeter VPC network (click to enlarge).

Provision and configure the VM and network interfaces for example 2

To create the VM and network interfaces in this example, use the following command:

gcloud compute instances create VM-appliance \
    --network-interface subnet=subnet-perimeter,address='reserved-address' \
    --network-interface subnet=subnet-1,no-address \
    --network-interface subnet=subnet-2,no-address \
    --network-interface subnet=subnet-3,no-address \
    --network-interface subnet=subnet-4,no-address \
    --machine-type=n1-standard-4

This creates an instance with five network interfaces:

• nic0 is attached to subnet-perimeter, which is part of network-perimeter, with a static address reserved-address.
• nic1 is attached to subnet-1, which is part of network-1, with no external IP address.
• nic2 is attached to subnet-2, which is part of network-2, with no external IP address.
• nic3 is attached to subnet-3, which is part of network-3, with no external IP address.
• nic4 is attached to subnet-4, which is part of network-4, with no external IP address.

Additional operational details

Multiple network interfaces in a Shared VPC environment

Shared VPC enables you to share VPC networks across projects in your Google Cloud organization.

Shared VPC lets you create instances associated with a Shared VPC network that is hosted in a centralized Shared VPC host project. For information about configuring Shared VPC networks, see Provisioning Shared VPC.

Note: Connecting to a Shared VPC network using interfaces other than nic0 is supported for the following configurations:

• VM instances: supported in the Google Cloud console, the gcloud CLI, and the API.
• Instance templates: supported in the gcloud CLI and the API.
• Managed instance groups: supported in the gcloud CLI and the API.

To create instances with one or more interfaces associated with Shared VPC networks, you must have the Compute Network User role (roles/compute.networkUser) in the Shared VPC host project.

DNS resolution with multiple network interfaces

When an internal DNS query is made with the instance hostname, it resolves to the primary interface (nic0) of the instance. If the nic0 interface of the instance is attached to a subnet in a VPC network that is different from the VPC network of the instance issuing the internal DNS query, the query fails.

Private Compute Engine DNS records are not generated per interface.

DHCP behavior with multiple network interfaces

In a default multiple interface configuration, the OS is configured to use DHCP. The DHCP and ARP behavior of each of the multiple interfaces is the same as the DHCP and ARP in an instance with a single interface.

In a multiple interface instance that uses DHCP, every interface gets a route for the subnet that it is in. In addition, the instance gets a single default route that is associated with the primary interface eth0. Unless manually configured otherwise, any traffic leaving an instance for any destination other than a directly connected subnet will leave the instance using the default route on eth0.

The behavior is the same for interfaces with IPv6 addresses. The interface gets a route for the IPv6 subnet range that it is in, as well as a single IPv6 default route.

In this sample, the primary interface eth0 gets the default route (default via 10.138.0.1 dev eth0), and both interfaces eth0 and eth1 get routes for their respective subnets.

instance-1:~$ ip route
default via 10.138.0.1 dev eth0
10.137.0.0/20 via 10.137.0.1 dev eth1
10.137.0.1 dev eth1 scope link
10.138.0.0/20 via 10.138.0.1 dev eth0
10.138.0.1 dev eth0 scope link

For more information, see the following tutorial: Configure routing for an additional interface.

Custom static routes and multiple network interfaces

When a VM instance has multiple interfaces and a network tag, the network tag might not impact all of the VM's interfaces. A VM's network tag impacts an interface if the interface is in a VPC network that contains a static route with a matching tag.

For example:

1. A VM has two interfaces: nic0 and nic1. The nic0 interface is in vpc-net-a. The nic1 interface is in vpc-net-b. The VM has a network tag called vpn-ok. The tag is an attribute on the instance, not on a specific interface.
2. The vpc-net-a network has a custom static route with a tag called vpn-ok.
3. The vpc-net-b network has a custom static route with a tag called vpn-123.

These numbered steps correspond to figure 3:

Figure 3. The custom static route in vpc-net-a affects nic0 because they have a tag in common, while the custom static route in vpc-net-b doesn't affect nic1 (click to enlarge).

In the case of the vpc-net-a network, because it has a route with a tag in common with the VM, the VM's vpn-ok tag applies to the VM's nic0 interface in vpc-net-a. In contrast, because the vpc-net-b network doesn't have a static route with the vpn-ok tag, the VM's vpn-ok network tag is ignored on the VM's nic1 interface.

Tags in routes in instances with multiple network interfaces

If you choose to use tags with routes, note that tags are applied at the instance level and, as such, tags apply to all interfaces of a virtual machine instance. If this is not desirable, make sure that the tags applied to the routes are unique to each VPC network.

Load balancers and multiple network interfaces

Except for Internal TCP/UDP Load Balancing, all Google Cloud load balancers only distribute traffic to the first interface (nic0) of a backend instance.

Firewall rules and multiple network interfaces

Each VPC network has its own set of firewall rules. If an instance's interface is in a particular VPC network, that network's firewall rules apply to that interface.

For example, suppose a VM instance has two interfaces:

• nic0 in VPC network network-1
• nic1 in VPC network network-2

Firewall rules that you create for the network-1 network apply to nic0. Firewall rules that you create for the network-2 network apply to nic1.

For more information, see VPC firewall rules.

Firewalls in instances with multiple network interfaces

• Ingress firewall rules can use either network tags or service accounts to identify sources, targets (destinations), or both.

• Egress firewall rules can use either network tags or service accounts to identify targets (sources).

For more information, see source and target filtering by service account.

Network tags and service accounts identify instances, not specific interfaces. Keep in mind that firewall rules are associated with a single VPC network, and each interface of a multi-NIC instance must be in a subnet that is in a unique VPC network.

The following example demonstrates how you can effectively use source tags for ingress allow firewall rules. The vm1 instance has two network interfaces:

• nic0 in network-1
• nic1 in network-2

Suppose you need to allow the following traffic from vm1:

• SSH traffic from vm1 to any instance in network-1
• HTTP and HTTPS traffic from vm1 to any instance in network-2

To accomplish this, you can do the following:

1. Assign two network tags to vm1: vm1-network1 and vm1-network2

2. Create an ingress allow firewall rule in network-1 with the following components to allow SSH traffic from vm1 to all VMs in network-1:

   • Action: allow
   • Direction: ingress
   • Sources: VMs with tag vm1-network1
   • Targets: All instances in the VPC network
   • Protocols and ports: tcp:22

3. Create an ingress allow firewall rule in network-2 with the following components to allow HTTP and HTTPS traffic from vm1 to all VMs in network-2:

   • Action: allow
   • Direction: ingress
   • Sources: VMs with tag vm1-network2
   • Targets: All instances in the VPC network
   • Protocols and ports: tcp:80,443

Figure 4 shows this firewall configuration example:

Figure 4. Firewall rule 1 and firewall rule 2 each have a source tag that is associated with VM1. Firewall rule 1, which is in network-1, only affects nic0 of VM1 because they are both in network-1. Firewall rule 2 only affects nic1 of VM1 because they also share a network (click to enlarge).

What's next

• Read Create VMs with multiple network interfaces.