Migrating VMware VMs using Layer 2 networks

This page describes how to to stretch a Layer 2 network from your on-premises environment to your private cloud by using a Layer 2 VPN (L2VPN).

L2VPN based stretching of Layer 2 networks can work with or without NSX-based networks in your on-premises VMware environment. If you don't have NSX-based networks for workloads on-premises, use a standalone NSX Edge Services Gateway.

Deployment scenario

To stretch your on-premises network using L2VPN, you must configure an L2VPN server (destination NSX-T Tier0 router) and an L2VPN client (source standalone client).

In this deployment scenario, you connect your private cloud to your on-premises environment by using a site-to-site VPN tunnel. The VPN tunnel allows on-premises management and vMotion subnets to communicate with the private cloud management and vMotion subnets. This arrangement enables Cross vCenter vMotion (xVC-vMotion). You deploy an NSX-T Tier0 router as an L2VPN server in the private cloud.

You deploy an NSX standalone Edge in your on-premises environment as an L2VPN client and then pair it with the L2VPN server. Create a GRE tunnel endpoint on each side and configure it to stretch the on-premises Layer 2 network to your private cloud.

For more information about migration using L2VPN, see the VMware documentation on Virtual private networks.

Prerequisites

Verify that the following prerequisites are in place before deploying and configuring the solution:

A site-to-site VPN connects your on-premises environment and your private cloud data center.
The on-premises vSphere version is 6.7U1+ or 6.5P03+.
The on-premises vSphere license is at the Enterprise Plus level (for vSphere Distributed Switch).
You have identified the workload Layer 2 network to be stretched to your private cloud.
You have identified a Layer 2 network in your on-premises environment for deploying your L2VPN client appliance.
You have created a private cloud.
The version of the standalone NSX-T Edge appliance is compatible with the NSX-T Manager version (NSX-T 2.3.0) used in your private cloud environment.
A trunk port group is already created in the on-premises vCenter.
A public IP address is reserved to use for the NSX-T standalone client uplink IP address, and a 1:1 NAT is in place for translation between the two addresses.
DNS forwarding is set on the on-premises DNS servers so that the domain points to the private cloud DNS servers.
RTT latency is less than or equal to 150 ms, which is required for vMotion to work across the two sites.

Limitations and considerations

The following table lists supported vSphere versions and network adaptor types.

vSphere version	Source vSwitch type	Virtual NIC driver	Target vSwitch type	Supported?
All	DVS	All	DVS	Yes
vSphere 6.7UI or higher, 6.5P03 or higher	DVS	VMXNET3	N-VDS	Yes
vSphere 6.7UI or higher, 6.5P03 or higher	DVS	E1000	N-VDS	Not supported per VMware
vSphere 6.7UI or 6.5P03, NSX-V or versions below NSX-T2.2, 6.5P03 or higher	All	All	N-VDS	Not supported per VMware

As of the VMware NSX-T 2.3 release:

The logical switch on the private cloud side that is stretched to on-premises over L2VPN can't be routed at the same time. The stretched logical switch can't be connected to a logical router.
L2VPN and route-based IPSEC VPNs can only be configured using API calls.

For more information, see the VMware documentation on Virtual private networks.

Sample L2VPN deployment

The following tables provide example specifications for L2VPN deployment.

On-premises network where the standalone ESG (L2VPN client) is deployed

Item	Value
Network name	MGMT_NET_VLAN469
VLAN	469
CIDR	10.250.0.0/24
Standalone Edge appliance IP address	10.250.0.111
Standalone Edge appliance NAT IP address	192.227.85.167

On-premises network to be stretched

Item	Value
VLAN	472
CIDR	10.250.3.0/24

Private cloud IP schema for NSX-T Tier0 Router (L2VPN server)

Item	Value
Loopback interface	192.168.254.254/32
Tunnel interface	5.5.5.1/29
Logical switch (stretched)	Stretch_LS
Loopback interface (NAT IP address)	104.40.21.81

Private cloud network to be mapped to the stretched network

Item	Value
VLAN	712
CIDR	10.200.15.0/24

Get the logical router ID needed for L2VPN

The following steps show how to fetch the logical-router ID of a Tier0 DR logical router instance for the IPsec and L2VPN services. Use the logical-router ID later, when implementing the L2VPN.

Log in to NSX-T Manager and select Networking > Routers > Provider-LR > Overview. For High Availability Mode, select Active-Standby. This action opens a pop-up window that shows the Edge VM on which the Tier0 router is currently active.
Select Fabric > Nodes > Edges. Make a note of the management IP address of the active Edge VM (Edge VM1) identified in the previous step.
Open an SSH session to the management IP address of the Edge VM. Run the get-logical-router command.
If you don't see an entry DR-Provider-LR, complete the following steps:
- Create two overlay-backed logical switches. One logical switch is stretched to on-premises where the migrated workloads reside. For more information, see Creating a logical switch.
- Attach the second logical switch to the Tier1 router with a link local IP address or any non-overlapping subnet from on-premises or your private cloud. For more information, see Adding a downlink port on a Tier-1 logical router.
- Run the get logical-router command again on the SSH session of the Edge VM. The output shows that the UUID of the DR-Provider-LR logical router is displayed. Make a note of the UUID, which is required when you configure the L2VPN.

Get the logical-switch ID needed for L2VPN

Log in to NSX-T Manager.
Select Networking > Switching > Switches > Overview.
Make a note of the UUID of the stretch logical switch, which is required when you configure the L2VPN.

Routing and security considerations for L2VPN

To establish an IPsec route-based VPN between the NSX-T Tier0 router and the NSX standalone Edge client, the loopback interface of the NSX-T Tier0 router must be able to communicate with the public IP address of the NSX standalone client on-premises over UDP 500/4500.

Allow UDP 500/4500 for IPsec

Allocate a public IP address for the NSX-T Tier0 loopback interface in the Google Cloud VMware Engine portal.
Create a firewall table with stateful rules that allow UDP 500/ 4500 inbound traffic, and attach the firewall table to the NSX-T HostTransport subnet.

Advertise the loopback interface IP to the underlay network

Create a null route for the loopback interface network.
- Log in to NSX-T Manager and select Networking > Routing > Routers > Provider-LR > Routing > Static routes.
- Click Add.
- For Network, enter the loopback interface IP address.
- For Next Hops, click Add and specify Null for the next hop. Keep the default of 1 for Admin distance.
Create an IP prefix list.
- Log in to NSX-T Manager and select Networking > Routing > Routers > Provider-LR > Routing > IP prefix lists.
- Click Add.
- Enter a name to identify the list.
- For Prefixes, click Add twice.
- In the first line, enter 0.0.0.0/0 for Network and Deny for Action.
- In the second line, select Any for Network and Permit for Action.
Attach the IP prefix list to both BGP neighbors (TOR). Attaching the IP prefix list to the BGP neighbor prevents the default route from being advertised in BGP to the TOR switches. However, any other route that includes the null route advertises the loopback interface IP address to the TOR switches.
Log in to NSX-T Manager and select Networking > Routing > Routers > Provider-LR > Routing > BGP > Neighbors. Select the first neighbor.
- Click Edit > Address families.
- For the IPv4 family, edit the Out filter column and select the IP prefix list that you created. Click Save. Repeat this step for the second neighbor.
Redistribute the null static route into BGP. To advertise the loopback interface route to the underlay, you must redistribute the null static route into BGP.
- Log in to NSX-T Manager and select Networking > Routing > Routers > Provider-LR > Routing > Route redistribution > Neighbors.
- Select Provider-LR-route_redistribution and click Edit.
- Select the Static checkbox and click Save

Configure a route-based VPN on the NSX-T Tier0 router

Use the following template to provide all the details for configuring a route based VPN on the NSX-T Tier0 router. The UUIDs in each POST call are required in subsequent POST calls.

The IP addresses for the loopback and tunnel interfaces for L2VPN must be unique and must not overlap with the on-premises or private cloud networks. The loopback interface network must always be /32.

Loopback interface ip : 192.168.254.254/32
Tunnel interface subnet : 5.5.5.0/29
Logical-router ID : UUID of Tier0 DR logical router obtained in section "Steps to fetch Logical-Router ID needed for L2VPN"
Logical-switch ID(Stretch) : UUID of Stretch Logical Switch obtained earlier
IPSec Service ID :
IKE profile ID :
DPD profile ID :
Tunnel Profile ID :
Local-endpoint ID :
Peer end-point ID :
IPSec VPN session ID (route-based) :
L2VPN service ID :
L2VPN session ID :
Logical-Port ID :
Peer Code :

For all of the following API calls, replace the IP address with your NSX-T Manager IP address. You can run all these API calls by using curl commands.

Enable the IPSec VPN service on the logical router

POST   https://192.168.110.201/api/v1/vpn/ipsec/services/
{
"resource_type": "IPSecVPNService",
"description": "Manage VPN service",
"display_name": "IPSec VPN service",
"logical_router_id": "Logical-router ID",
"ike_log_level": "INFO",
"enabled": true
}

Create profiles: IKE

POST https://192.168.110.201/api/v1/vpn/ipsec/ike-profiles

{
"resource_type": "IPSecVPNIKEProfile",
"description": "IKEProfile for siteA",
"display_name": "IKEProfile siteA",
"encryption_algorithms": ["AES_128"],
"ike_version": "IKE_V2",
"digest_algorithms": ["SHA2_256"],
"sa_life_time":21600,
"dh_groups": ["GROUP14"]
}

Create profiles: DPD

POST  https://192.168.110.201/api/v1/vpn/ipsec/dpd-profiles

{
"resource_type": "IPSecVPNDPDProfile",
"display_name": "nsx-default-dpd-profile",
"enabled": true
}

Create profiles: Tunnel

POST  https://192.168.110.201/api/v1/vpn/ipsec/tunnel-profiles

{
"resource_type": "IPSecVPNTunnelProfile",
"display_name": "nsx-default-tunnel-profile",
"enable_perfect_forward_secrecy": true,
"encryption_algorithms": ["AES_GCM_128"],
"digest_algorithms": [],
"sa_life_time":3600,
"dh_groups": ["GROUP14"],
"encapsulation_mode": "TUNNEL_MODE",
"transform_protocol": "ESP",
"df_policy": "COPY"
}

Create a local endpoint

{
"resource_type": "IPSecVPNLocalEndpoint",
"description": "Local endpoint",
"display_name": "Local endpoint",
"local_id": "Public IP of Loopback interface",
"ipsec_vpn_service_id": {
"target_id": "IPSec VPN service ID"},
"local_address": "IP of Loopback interface",
"trust_ca_ids": [],
"trust_crl_ids": []
}

Create a peer endpoint

POST https://192.168.110.201/api/v1/vpn/ipsec/peer-endpoints

{
"resource_type": "IPSecVPNPeerEndpoint",
"description": "Peer endpoint for site B",
"display_name": "Peer endpoint for site B",
"connection_initiation_mode": "INITIATOR",
"authentication_mode": "PSK",
"ipsec_tunnel_profile_id": "IPSec Tunnel profile ID",
"dpd_profile_id": "DPD profile ID",
"psk":"nsx",
"ike_profile_id": "IKE profile ID",
"peer_address": "Public IP of Standalone client",
"peer_id": "Public IP of Standalone client"
}

Create a route-based VPN session

POST :  https://192.168.110.201/api/v1/vpn/ipsec/sessions

{
"resource_type": "RouteBasedIPSecVPNSession",
"peer_endpoint_id": "Peer Endpoint ID",
"ipsec_vpn_service_id": "IPSec VPN service ID",
"local_endpoint_id": "Local Endpoint ID",
"enabled": true,
"tunnel_ports": [
{
"ip_subnets": [
{
"ip_addresses": [
 "5.5.5.1"
],
"prefix_length": 24
}
  ]
}
]
}

Configure L2VPN on NSX-T Tier0 router

Provide the following information after every POST call. The IDs are required in subsequent POST calls.

L2VPN Service ID:
L2VPN Session ID:
Logical Port ID:

Create the L2VPN service

The output of the following GET command will be empty because the configuration is not yet complete.

GET : https://192.168.110.201/api/v1/vpn/l2vpn/services

For the following POST command, the logical router ID is the UUID of the Tier0 DR logical router obtained earlier.

POST : https://192.168.110.201/api/v1/vpn/l2vpn/services

{
"logical_router_id": "Logical Router ID",
"enable_full_mesh" : true
}

Create the L2VPN session

For the following POST command, the L2VPN service ID is the ID that you just obtained, and the IPsec VPN session ID is the ID obtained in the previous section.

POST: https://192.168.110.201/api/v1/vpn/l2vpn/sessions

{
"l2vpn_service_id" : "L2VPN service ID",
"transport_tunnels" : [
{
"target_id" : "IPSec VPN session ID"
}
]
}

These calls create a GRE tunnel endpoint. To check the status, run the following command.

edge-2> get tunnel-port
Tunnel      : 44648dae-8566-5bc9-a065-b1c4e5c3e03f
IFUID       : 328
LOCAL       : 169.254.64.1
REMOTE      : 169.254.64.2
ENCAP       : GRE

Tunnel      : cf950ca1-5cf8-5438-9b1a-d2c8c8e7229b
IFUID       : 318
LOCAL       : 192.168.140.155
REMOTE      : 192.168.140.152
ENCAP       : GENEVE

Tunnel      : 63639321-87c5-529e-8a61-92c1939799b2
IFUID       : 304
LOCAL       : 192.168.140.155
REMOTE      : 192.168.140.156
ENCAP       : GENEVE

Create a logical port with the tunnel ID specified

   POST https://192.168.110.201/api/v1/logical-ports/

{
"resource_type": "LogicalPort",
"display_name": "Extend logicalSwitch, port for service",
"logical_switch_id": "Logical switch ID",
"admin_state" : "UP",
"attachment": {
"attachment_type":"L2VPN_SESSION",
"id":"L2VPN session ID",
"context" : {
"resource_type" : "L2VpnAttachmentContext",
    "tunnel_id" : 10
}
    }
        }

Obtain the peer code for L2VPN on the NSX-T side

Obtain the peer code of the NSX-T endpoint. The peer code is required when configuring the remote endpoint. The L2VPN can be obtained from the previous section. For more information, see the NSX-T 2.3 API.

GET https://192.168.110.201/api/v1/vpn/l2vpn/sessions/session-id/peer-codes

Deploy the NSX-T standalone client (on-premises)

Before deploying, verify that your on-premises firewall rules allow inbound and outbound UDP 500/4500 traffic from or to the public IP address that was reserved earlier for the NSX-T T0 router loopback interface.

Download the standalone edge client OVF and extract the files from the downloaded bundle into a folder.
Go to the folder with all the extracted files.
Select all the vmdks files and click Next.
- NSX-l2t-client-large.mf and NSX-l2t-client-large.ovf for large appliance size.
- NSX-l2t-client-Xlarge.mf and NSX-l2t-client-Xlarge.ovf for extra large size appliance size.
Enter a name for the NSX-T standalone client and click Next.
Click Next as needed to reach the datastore settings.
Select the appropriate datastore for the NSX-T standalone client and click Next.
Select the correct port groups for the Trunk (Trunk PG), Public (Uplink PG), and HA (Uplink PG) interfaces for the NSX-T standalone client. Click Next.
Provide the following information in the Customize template screen and click Next.
- Expand L2T:
  - Peer address. Enter the IP address reserved on the Google Cloud VMware Engine portal for the NSX-T Tier0 loopback interface.
  - Peer code. Paste the peer code obtained from the L2VPN server deployment.
  - Sub interfaces VLAN (tunnel ID). Enter the VLAN ID to be stretched. In parentheses (), enter the tunnel ID that was previously configured.
- Expand uplink interface:
  - DNS IP address. Enter the on-premises DNS IP address.
  - Default gateway. Enter the gateway of the VLAN to act as the default gateway for this client.
  - IP address. Enter the uplink IP address of the standalone client.
  - Prefix length. Enter the prefix length of the uplink VLAN/subnet.
  - CLI admin/enable/root user password. Set the password for the admin/enable/root account.
Review the settings and click Finish.

Configure an on-premises sink port

If one of the VPN sites doesn't have NSX deployed, you can configure an L2VPN by deploying a standalone NSX edge at that site. For an edge services gateway appliance to function as an L2VPN client, deploy a standalone edge using an OVF file on a host that isn't managed by NSX.

If a standalone edge trunk vNIC is connected to a vSphere distributed switch, either promiscuous mode or a sink port is required for L2VPN function. Using promiscuous mode can cause duplicate pings and duplicate responses. For this reason, use sink port mode in the L2VPN standalone NSX edge configuration. For more information, see Configuring a sink port.

IPsec VPN and L2VPN verification

Use the following commands to verify IPsec and L2VPN sessions from NSX-T standalone edge.

nsx-l2t-edge> show service ipsec
-----------------------------------------------------------------------
vShield Edge IPSec Service Status:
IPSec Server is running.
AESNI is enabled.
Total Sites: 1, 1 UP, 0 Down
Total Tunnels: 1, 1 UP, 0 Down
----------------------------------
Site:  10.250.0.111_0.0.0.0/0-104.40.21.81_0.0.0.0/0
Channel: PeerIp: 104.40.21.81    LocalIP: 10.250.0.111  Version: IKEv2  Status: UP
Tunnel: PeerSubnet: 0.0.0.0/0    LocalSubnet: 0.0.0.0/0   Status: UP
----------------------------------
nsx-l2t-edge> show service l2vpn
L2 VPN is running
----------------------------------------
L2 VPN type: Client/Spoke

SITENAME                       IPSECSTATUS          VTI                  GRE
1ecb00fb-a538-4740-b788-c9049e8cb6c6 UP                   vti-100              l2t-1

Use the following commands to verify IPsec and L2VPN sessions from the NSX-T Tier0 router.

edge-2> get ipsecvpn session
Total Number of Sessions: 1

IKE Session ID : 3
UUID           : 1ecb00fb-a538-4740-b788-c9049e8cb6c6
Type           : Route

Local IP       : 192.168.254.254      Peer IP        : 192.227.85.167
Local ID       : 104.40.21.81         Peer ID        : 192.227.85.167
Session Status : Up

Policy Rules
    VTI UUID       : 4bf96e3b-e50b-49cc-a16e-43a6390e3d53
    ToRule ID      : 560874406            FromRule ID    : 2708358054
    Local Subnet   : 0.0.0.0/0            Peer Subnet    : 0.0.0.0/0
    Tunnel Status  : Up
edge-2> get l2vpn session
Session       : f7147137-5dd0-47fe-9e53-fdc2b687b160
Tunnel        : b026095b-98c8-5932-96ff-dda922ffe316
IPSEC Session : 1ecb00fb-a538-4740-b788-c9049e8cb6c6
Status        : UP

Use the following commands to verify the sink port on the ESXi host where the NSX-T standalone client VM resides in the on-premises environment.

[root@esxi02:~] esxcfg-vswitch -l |grep NSX
  53                  1           NSXT-client-large.eth2
  225                1           NSXT-client-large.eth1
  52                  1           NSXT-client-large.eth0
[root@esxi02:~] net-dvs -l | grep "port\ [0-9]\|SINK\|com.vmware.common.alias"
                com.vmware.common.alias = csmlab2DS ,   propType = CONFIG
        port 24:
        port 25:
        port 26:
        port 27:
        port 13:
        port 19:
        port 169:
        port 54:
        port 110:
        port 6:
        port 107:
        port 4:
        port 199:
        port 168:
        port 201:
        port 0:
        port 49:
        port 53:
        port 225:
                com.vmware.etherswitch.port.extraEthFRP =   SINK
        port 52:

What's next

Set up VPN gateways on the Google Cloud VMware Engine network.
Learn more about VMware on your private cloud.