Distributed Cloud Edge networking features

This page describes the networking features of Google Distributed Cloud Edge, including subnetworks, BGP peering sessions, and load balancing.

The procedures on this page only apply to Distributed Cloud Edge Racks, except load balancing, which applies to both Distributed Cloud Edge Racks and Distributed Cloud Edge networking Servers.

Enable the Distributed Cloud Edge Network API

Before you can configure Distributed Cloud Edge networking, you must enable the Distributed Cloud Edge Network API. To do so, complete the steps in this section. By default, Distributed Cloud Edge Servers ship with the Distributed Cloud Edge Network API already enabled.

Console

1. In the Google Cloud console, go to the Distributed Cloud Edge Network API page.

   Enable the API

2. Click Enable.

gcloud

Use the following command:

gcloud services enable edgenetwork.googleapis.com

Configure Distributed Cloud Edge networking

This section describes how to configure the Distributed Cloud Edge networking components.

The following limitations apply to Distributed Cloud Edge Servers:

• You can only configure subnetworks, and
• Subnetworks only support VLAN IDs; CIDR-based subnetworks are not supported.

A typical network configuration for Distributed Cloud Edge consists of the following steps:

1. Optional: Initialize the network configuration of the target zone, if necessary.

2. Create a network.

3. Create one or more subnetworks within the network.

4. Establish northbound BGP peering sessions with your PE routers by using the corresponding interconnect attachments.

5. Establish southbound BGP peering sessions with the Pods that run your workloads by using the corresponding subnetworks.

6. Optional: Establish loopback BGP peering sessions for high availability.

7. Test your configuration.

8. Connect your Pods to the network.

Optional: Initialize the network configuration of the Distributed Cloud Edge zone

You must initialize the network configuration of your Distributed Cloud Edge zone in the following cases:

• Immediately after your Distributed Cloud Edge hardware has been installed on your premises.
• You upgraded to Distributed Cloud Edge version 1.3.0 or later on an existing Distributed Cloud Edge deployment but did not participate in the private preview of the Distributed Cloud Edge Network API.

Initializing the network configuration of a zone creates a default router named default and a default network named default. It also configures the default router to peer with all of the interconnects that you requested when you ordered the Distributed Cloud Edge hardware by creating corresponding interconnect attachments. This configuration provides your Distributed Cloud Edge deployment with basic uplink connectivity to your local network.

Initializing the network configuration of a zone is a one-time procedure. For complete instructions, see Initialize the network configuration of a zone.

Create a network

To create a new network, follow the instructions in Create a network. You must also create at least one subnetwork within the network to allow Distributed Cloud Edge nodes to connect to the network.

Create one or more subnetworks

To create a subnetwork, follow the instructions in Create a subnetwork. You must create at least one subnetwork in your network to allow nodes to access the network. The VLAN corresponding to each subnetwork that you create is automatically available to all nodes in the zone.

For Distributed Cloud Edge Servers, you can only configure subnetworks using VLAN IDs. CIDR-based subnetworks are not supported.

Establish northbound BGP peering sessions

When you create a network and its corresponding subnetworks, they are local to their Distributed Cloud Edge zone. To enable outbound connectivity, you must establish at least one northbound BGP peering session between the network and your peering edge routers.

To establish a northbound BGP peering session, do the following:

1. List the interconnects available in your zone and then select the target interconnect for this peering session.

2. Create one or more interconnect attachments on the selected interconnect. Interconnect attachments link the router that you create in the next step with the selected interconnect.

3. Create a router. This router routes traffic between the interconnect and your network by using the interconnect attachments that you created in the previous step.

4. Add an interface to the router for each interconnect attachment that you created earlier in this procedure. For each interface, use the IP address of the corresponding top-of-rack (ToR) switch in your Distributed Cloud Edge rack. For instructions, see Establish a northbound peering session.

5. Add a peer for each interface that you created on the router in the previous step.

Establish southbound BGP peering sessions

To enable inbound connectivity to your workloads from your local network, you must establish one or more southbound BGP peering sessions between your peering edge routers and the subnetwork to which your Pods belong. The gateway IP address for each subnetwork is the IP address of the corresponding ToR switch in your Distributed Cloud Edge rack.

To establish a southbound BGP peering session, do the following:

1. Add an interface to the router in the target network for each subnetwork that you want to provision with inbound connectivity. For instructions, see Establish a southbound peering session.

2. Add a peer for each interface that you created on the router in the previous step.

Optional: Establish loopback BGP peering sessions

To enable high-availability connectivity between your workloads and your local network, you can establish a loopback BGP peering session between the target Pod and both ToR switches in your Distributed Cloud Edge rack. A loopback peering session establishes two independent peering sessions for the Pod, one with each ToR switch.

To establish a loopback BGP peering session, do the following:

1. Add a loopback interface to the router in the target network. For instructions, see Establish a loopback peering session.

2. Add a peer for the loopback interface.

Test your configuration

To test your configuration of the network components that you created, do the following:

1. Check the operational status of the network.

2. Check the provisioning status of each subnetwork.

3. Check the operational status of the interconnects.

4. Check the operational status of the interconnect attachments.

5. Check the operational status of the router.

Connect your Pods to the network

To connect your Pods to the network and configure advanced network functions, follow the instructions in Network Function operator.

Load balancing

Distributed Cloud Edge ships with a bundled network load balancing solution based on MetalLB in Layer 2 mode. You can use this solution to expose services that run in your Distributed Cloud Edge zone to the outside world by using virtual IP addresses (VIPs) as follows:

1. Your network administrator plans the network topology and specifies the required virtual IPv4 address subnetwork when ordering Distributed Cloud Edge. Google configures your Distributed Cloud Edge hardware accordingly before delivery. Keep the following in mind:
   • This VIP subnetwork is shared among all Kubernetes clusters that run within your Distributed Cloud Edge zone.
   • A route for the requested VIP subnetwork is advertised through the BGP sessions between the Distributed Cloud Edge zone and your local network.
   • The first (network ID), second (default gateway), and last (broadcast address) addresses in the subnetwork are reserved for core system functionality. Do not assign those addresses to your MetalLB configurations' address pools.
   • Each cluster must use a separate VIP range that falls within the configured VIP subnetwork.
2. When you create a cluster in your Distributed Cloud Edge zone, your cluster administrator specifies the Pod and ClusterIP Service address pools by using CIDR notation. Your network administrator provides the appropriate LoadBalancer VIP subnetwork to your cluster administrator.

3. After the cluster is created, the cluster administrator configures the corresponding VIP pools. For remote control plane clusters, you must edit the metallb-config ConfigMap in the metallb-system namespace by using the kubectl edit or kubectl replace command. Do not use the kubectl apply command because Distributed Cloud Edge overwrites your changes if you do.

   The following example illustrates such a configuration:

   # metallb-config.yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
     namespace: metallb-system
     name: metallb-config
   data:
     config: |
       address-pools:
       - name: default
         protocol: layer2
         addresses:
         - 192.168.1.2-192.168.1.254
   

   For local control plane clusters, you must specify the VIP pools by using the --external-lb-ipv4-address-pools flag when you create the cluster. For more information, see Survivability mode.

4. The cluster administrator creates the appropriate Kubernetes LoadBalancer Services.

Distributed Cloud Edge nodes in a single node pool share a common Layer 2 domain and are therefore also MetalLB load balancer nodes. Distributed Cloud Edge control plane nodes that run on Google Cloud do not function as load balancer nodes.

Distributed Cloud Edge ingress

In addition to load balancing, Distributed Cloud Edge also supports Kubernetes Ingress resources. A Kubernetes Ingress resource controls the flow of HTTP(S) traffic to Kubernetes Services that run on your Distributed Cloud Edge clusters. The following example illustrates a typical Ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            name: my-service
            port:
              number: 80
        path: /foo
        pathType: Prefix

When configured, network traffic flows through the istio-ingress Service, which by default is assigned a random IP address from the VIP pools specified in your MetalLB configuration. You can select a specific IP address or a virtual IP address from the MetalLB configuration by using the loadBalancerIP field in the istio-ingress Service definition. For example:

apiVersion: v1
kind: Service
metadata:
  labels:
    istio: ingress-gke-system
    release: istio
  name: istio-ingress
  namespace: gke-system
spec:
  loadBalancerIP: <targetLoadBalancerIPaddress>

This functionality is not available on Distributed Cloud Edge Servers.

Disable the default Distributed Cloud Edge Ingress resource

Preview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

By default, when you create a Distributed Cloud Edge cluster, Distributed Cloud Edge automatically configures the istio-ingress Service for the cluster. You have the option to create a Distributed Cloud Edge cluster without the istio-ingress Service. To do so, complete the following steps:

gcloud

1. Create a YAML configuration file named SystemsAddonConfig.yaml with the following contents:

   systemAddonsConfig:
    ingress:
      disabled: true
   

2. Pass the SystemsAddonConfig.yaml file by using the --system-addons-config flag in your cluster creation command. You must use the gcloud alpha version to use this feature. For example:

   gcloud alpha edge-cloud container clusters create MyGDCECluster1 --location us-west1 \
       --system-addons-config=SystemsAddonConfig.yaml
   

   For more information about creating a Distributed Cloud Edge cluster, see Create a cluster.

API

1. Add the following JSON content to the JSON payload in your cluster creation request:

   "systemAddonConfig" {
      "ingress" {
              "disabled": true
      }
   }
   

2. Submit the cluster creation request as described in Create a cluster.

SCTP support

Preview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

Distributed Cloud Edge supports the Stream Control Transmission Protocol (SCTP) on the primary network interface for both internal and external networking. SCTP support includes the NodePort, LoadBalancer, and ClusterIP Service types. Pods can use SCTP to communicate with other Pods and external resources. The following example illustrates how to configure IPERF as a ClusterIP Service by using SCTP:

apiVersion: v1
kind: Pod
metadata:
  name: iperf3-sctp-server-client
  labels:
    app.kubernetes.io/name: iperf3-sctp-server-client
spec:
  containers:
  - name: iperf3-sctp-server
    args: ['-s', '-p 31390']
    ports:
      - containerPort: 31390
        protocol: SCTP
        name: server-sctp
  - name: iperf3-sctp-client
    ...

---

apiVersion: v1
kind: Service
metadata:
  name: iperf3-sctp-svc
spec:
  selector:
    app.kubernetes.io/name: iperf3-sctp-server-client
  ports:
    - port: 31390
      protocol: SCTP
      targetPort: server-sctp

This functionality is not available on Distributed Cloud Edge Servers.

SCTP kernel modules

Starting with version 1.5.0, Distributed Cloud Edge configures the sctp Edge OS kernel module as loadable. This allows you to load your own SCTP protocol stacks in the kernel user space.

Additionally, Distributed Cloud Edge loads the following modules into the kernel by default:

Module name	Config name
fou	CONFIG_NET_FOU
nf_conntrack_proto_gre	CONFIG_NF_CT_PROTO_GRE
nf_conntrack_proto_sctp	CONFIG_NF_CT_PROTO_SCTP
inotify	CONFIG_INOTIFY_USER
xt_redirect	CONFIG_NETFILTER_XT_TARGET_REDIRECT
xt_u32	CONFIG_NETFILTER_XT_MATCH_U32
xt_multiport	CONFIG_NETFILTER_XT_MATCH_MULTIPORT
xt_statistic	CONFIG_NETFILTER_XT_MATCH_STATISTIC
xt_owner	CONFIG_NETFILTER_XT_MATCH_OWNER
xt_conntrack	CONFIG_NETFILTER_XT_MATCH_CONNTRACK
xt_mark	CONFIG_NETFILTER_XT_MARK
ip6table_mangle	CONFIG_IP6_NF_MANGLE
ip6_tables	CONFIG_IP6_NF_IPTABLES
ip6table_filter	CONFIG_IP6_NF_FILTER
ip6t_reject	CONFIG_IP6_NF_TARGET_REJECT
iptable_mangle	CONFIG_IP_NF_MANGLE
ip_tables	CONFIG_IP_NF_IPTABLES
iptable_filter	CONFIG_IP_NF_FILTER

ClusterDNS resource

Preview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

Distributed Cloud Edge supports the Google Distributed Cloud ClusterDNS resource for configuring upstream name servers for specific domains by using the spec.domains section. For more information about configuring this resource, see spec.domains.

This functionality is not available on Distributed Cloud Edge Servers.

What's next

• Pod and Service network address allocation