Network Function operator

Distributed Cloud Edge ships with a specialized Network Function Kubernetes operator. This operator implements a set of CRDs that allow Distributed Cloud Edge to execute high-performance workloads.

The Network Function operator allows you do the following:

  • Poll for existing network devices on a Node.
  • Query the IP address and physical link state for each network device on a Node.
  • Provision additional network interfaces on a Node.
  • Configure low-level system features on the Node's physical machine required to support high-performance workloads.
  • Use single-root input/output virtualization (SR-IOV) on PCI Express network interfaces to virtualize them into multiple virtual interfaces. You can then configure your Distributed Cloud Edge workloads to use those virtual network interfaces.

Distributed Cloud Edge's support for SR-IOV is based on the following open source projects:

Configure the Distributed Cloud Edge ToR switches for Network Function operator resources

The Network Function operator resources described on this page might require you to reconfigure the Distributed Cloud Edge Top-of-Rack switches. Please contact Support to file a ticket for this purpose.

Network Function operator resources

The Distributed Cloud Edge Network Function operator implements the following Kubernetes CRDs:

  • Network: Defines a virtual network that Pods can use to communicate with internal and external resources.
  • NetworkInterface: Defines a network interface that Pods can use to communicate with internal and external resources. This interface is referenced in the corresponding Network resource.
  • NetworkInterfaceState: Enables the discovery of network interface states and querying a network interface for link state and IP address.
  • CustomNetworkInterfaceConfig: Enables the configuration of additional network interfaces on a Node.
  • NodeSystemConfigUpdate: Enables the configuration of low-level system features such as kernel options and Kubelet flags.
  • SriovNetworkNodePolicy: Selects a group of SR-IOV virtualized network interfaces and instantiates the group as a Kubernetes resource. You can use this resource in a NetworkAttachmentDefinition resource.
  • SriovNetworkNodeState: Allows you to query the provisioning state of the SriovNetworkNodePolicy resource on a Distributed Cloud Edge Node.
  • NetworkAttachmentDefinition: Allows you to attach Distributed Cloud Edge Pods to one or more logical or physical networks on your Distributed Cloud Edge Node.

The Network Function operator also allows you to define secondary network interfaces that do not use SR-IOV virtual functions.

Network resource

The Network resource defines a virtual network within the Distributed Cloud Edge rack that Pods within your Distributed Cloud Edge cluster can use to communicate with internal and external resources.

The Network resource provides the following configurable parameters for the network interface exposed as writable fields:

  • spec.interfaces.networkName: name of the virtual network to which this interface is assigned.
  • spec.interfaces.type: specifies the network transport layer for this network. The only valid value is L2. You must also specify a nodeInterfaceMatcher.interfaceName value.
  • spec.interfaces.nodeInterfaceMatcher.interfaceName: Name of the physical network interface on the target Distributed Cloud Edge node to use with this network. This interface must be a VLAN interface instantiated through a CustomNetworkInterfaceConfig resource.
  • spec.interfaces.gateway4: IP address of the network gateway for this network.

The following example illustrates the structure of the resource:

apiVersion: networking.gke.io/v1alpha1
kind: Network
metadata:
  name: vlan200-network
spec:
  type: L2
  nodeInterfaceMatcher:
    interfaceName: gdcenet0.200
  gateway4: 10.53.0.1

NetworkInterface resource

The NetworkInterface resource defines the configuration of a single network interface that is available to a specific Pod. It allows you to specify a static IP address, a MAC address, and a descriptive name that uniquely identifies the network interface.

The NetworkInterface resource provides the following configurable parameters for the network interface exposed as writable fields:

  • spec.interfaces.ipAddresses: IP addresses to assign to the target interface.
  • spec.interfaces.macAddress: MAC address to assign to the target interface.
  • spec.interfaces.networkName: name of the Network resource that references this NetworkInterface resource.
  • spec.interfaces.interfaceName: a descriptive name that uniquely identifies the target interface.

The NetworkInterface resource provides the following read-only general status fields:

  • status.interfaces.ipAddresses: IP addresses to assign to the target interface.
  • status.interfaces.macAddress: MAC address to assign to the target interface.

The following example illustrates the structure of the resource:

apiVersion: networking.gke.io/v1alpha1
kind: NetworkInterface
metadata:
 name: multinet-vlan200
 namespace: default
spec:
 networkName: vlan200-network
 ipAddresses:
  - 10.53.0.2/24
 macAddress: 86:fe:36:83:56:1

NetworkInterfaceState resource

The NetworkInterfaceState resource is a read-only resource that allows you to discover physical network interfaces on the Node and collect runtime statistics on the network traffic flowing through those interfaces. Distributed Cloud Edge creates a NetworkInterfaceState resource for each Node in a Cluster.

The default configuration of Distributed Cloud Edge Machines includes a bonded network interface on the Rack Select Network Daughter Card (rNDC) named gdcenet0. This interface bonds the eno1np0 and eno2np1 network interfaces. Each of those is connected to one Distributed Cloud Edge ToR switch, respectively.

The NetworkInterfaceState resource provides the following categories of network interface information exposed as read-only status fields.

General information:

  • status.interfaces.ifname: Name of the target network interface.
  • status.lastReportTime: Time and date of the last status report for the target interface.

IP configuration information:

  • status.interfaces.interfaceinfo.address: IP address assigned to the target interface.
  • status.interfaces.interfaceinfo.dns: IP address of the DNS server assigned to the target interface.
  • status.interfaces.interfaceinfo.gateway: IP address of the network gateway serving the target interface.
  • status.interfaces.interfaceinfo.prefixlen: Length of the IP prefix.

Hardware information:

  • status.interfaces.linkinfo.broadcast: Broadcast MAC address of the target interface.
  • status.interfaces.linkinfo.businfo PCIe device path in bus:slot.function format.
  • status.interfaces.linkinfo.flags: Interface flags. For example: BROADCAST.
  • status.interfaces.linkinfo.macAddress: Unicast MAC address of the target interface.
  • status.interfaces.linkinfo.mtu: MTU value for the target interface.

Reception statistics:

  • status.interfaces.statistics.rx.bytes: Total bytes received by the target interface.
  • status.interfaces.statistics.rx.dropped: Total packets dropped by the target interface.
  • status.interfaces.statistics.rx.errors: Total packet receive errors for the target interface.
  • status.interfaces.statistics.rx.multicast: Total multicast packets received by the target interface.
  • status.interfaces.statistics.rx.overErrors: Total packet receive over errors for the target interface.
  • status.interfaces.statistics.rx.packets: Total packets received by the target interface.

Transmission statistics:

  • status.interfaces.statistics.tx.bytes: Total bytes transmitted by the target interface.
  • status.interfaces.statistics.tx.carrierErrors: Total carrier errors encountered by the target interface.
  • status.interfaces.statistics.tx.collisions: Total packet collisions encountered by the target interface.
  • status.interfaces.statistics.tx.dropped: Total packets dropped by the target interface.
  • status.interfaces.statistics.tx.errors: Total transmission errors for the target interface.
  • status.interfaces.statistics.tx.packets: Total packets transmitted by the target interface.

The following example illustrates the structure of the resource:

apiVersion: networking.gke.io/v1
kind: NetworkInterfaceState
metadata:
  name: MyNode1
nodeName: MyNode1
status:
  interfaces:
  - ifname: eno1np0
    linkinfo:
      businfo: 0000:1a:00.0
      flags: up|broadcast|multicast
      macAddress: ba:16:03:9e:9c:87
      mtu: 9000
    statistics:
      rx:
        bytes: 1098522811
        errors: 2
        multicast: 190926
        packets: 4988200
      tx:
        bytes: 62157709961
        packets: 169847139
  - ifname: eno2np1
    linkinfo:
      businfo: 0000:1a:00.1
      flags: up|broadcast|multicast
      macAddress: ba:16:03:9e:9c:87
      mtu: 9000
    statistics:
      rx:
        bytes: 33061895405
        multicast: 110203
        packets: 110447356
      tx:
        bytes: 2370516278
        packets: 11324730
  - ifname: enp95s0f0np0
    interfaceinfo:
    - address: fe80::63f:72ff:fec4:2bf4
      prefixlen: 64
    linkinfo:
      businfo: 0000:5f:00.0
      flags: up|broadcast|multicast
      macAddress: 04:3f:72:c4:2b:f4
      mtu: 9000
    statistics:
      rx:
        bytes: 37858381
        multicast: 205645
        packets: 205645
      tx:
        bytes: 1207334
        packets: 6542
  - ifname: enp95s0f1np1
    interfaceinfo:
    - address: fe80::63f:72ff:fec4:2bf5
      prefixlen: 64
    linkinfo:
      businfo: 0000:5f:00.1
      flags: up|broadcast|multicast
      macAddress: 04:3f:72:c4:2b:f5
      mtu: 9000
    statistics:
      rx:
        bytes: 37852406
        multicast: 205607
        packets: 205607
      tx:
        bytes: 1207872
        packets: 6545
  - ifname: enp134s0f0np0
    interfaceinfo:
    - address: fe80::63f:72ff:fec4:2b6c
      prefixlen: 64
    linkinfo:
      businfo: 0000:86:00.0
      flags: up|broadcast|multicast
      macAddress: 04:3f:72:c4:2b:6c
      mtu: 9000
    statistics:
      rx:
        bytes: 37988773
        multicast: 205584
        packets: 205584
      tx:
        bytes: 1212385
        packets: 6546
  - ifname: enp134s0f1np1
    interfaceinfo:
    - address: fe80::63f:72ff:fec4:2b6d
      prefixlen: 64
    linkinfo:
      businfo: 0000:86:00.1
      flags: up|broadcast|multicast
      macAddress: 04:3f:72:c4:2b:6d
      mtu: 9000
    statistics:
      rx:
        bytes: 37980702
        multicast: 205548
        packets: 205548
      tx:
        bytes: 1212297
        packets: 6548
  - ifname: gdcenet0
    interfaceinfo:
    - address: 208.117.254.36
      prefixlen: 28
    - address: fe80::b816:3ff:fe9e:9c87
      prefixlen: 64
    linkinfo:
      flags: up|broadcast|multicast
      macAddress: ba:16:03:9e:9c:87
      mtu: 9000
    statistics:
      rx:
        bytes: 34160422968
        errors: 2
        multicast: 301129
        packets: 115435591
      tx:
        bytes: 64528301111
        packets: 181171964
     .. <remaining interfaces omitted>
   lastReportTime: "2022-03-30T07:35:44Z"

CustomNetworkInterfaceConfig resource

The CustomNetworkInterfaceConfig resource allows you to add custom network interfaces to a specific Node on the root network namespace. The resource supports only VLAN interfaces. When instantiating this resource, you must specify the name of the target Node in the nodeName field.

To take advantage of the redundancy provided by the twin ToR switches in the Distributed Cloud Edge rack, you must configure any additional VLAN interfaces on top of the gdcenet0 interface described in NetworkInterfaceState resource. Keep the folowing in mind:

  • Do not use VLANs below 100. Those VLANs are reserved for Distributed Cloud Edge management traffic.
  • The Network Function operator can not configure the eno1np0, eno2np1, and gdcenet0 network interfaces.

Distributed Cloud Edge does not provide a controller to coordinate Distributed Cloud Edge Machine reboots. You must apply this resource to one Node at a time so that the parent Cluster remains up while the configuration changes take effect.

The CustomNetworkInterfaceConfig resource provides the following configurable parameters for the network interface exposed as writable fields:

  • spec.interfaces.interfaceinfo.address: IP address to assign to the target interface.
  • spec.interfaces.mtu: MTU value of the target interface.
  • spec.interfaces.type: Interface type. VLAN is the only valid value.
  • spec.interfaces.vlan.id: Specifies the VLAN ID of the target interface.
  • spec.interfaces.vlan.masterInterface: Specifies the parent interface with which the target interface is bonded. The only valid value is gdcenet0.

The CustomNetworkInterfaceConfig resource provides the following network interface information exposed as read-only status fields:

  • status.interfaces.interfaceinfo.address: IP address assigned to the target interface.
  • status.interfaces.mtu: MTU value of the target interface.
  • status.interfaces.type: Interface type.VLAN is the only valid value.
  • status.interfaces.vlan.id: Returns the VLAN ID of the target interface.
  • status.interfaces.vlan.masterInterface: Returns the interface with which the target interface is bonded.

The CustomNetworkInterfaceConfig resource provides the following read-only general status fields:

  • status.lastReportTime: Most recent time status was reported for the target interface.
  • status.conditions.lastTransitionTime: Most recent time the condition of the interface has changed.
  • status.conditions.observedGeneration: Denotes the .metadata.generation value on which the initial condition was based.
  • status.conditions.message: An informative message describing the change of the interface's condition.
  • status.conditions.reason: A programmatic identifier denoting the reason for the last change of the interface's condition.
  • status.conditions.status: Status descriptor of the condition. Valid values are True, False, and Unknown.
  • status.conditions.type: Condition type in camelCase.

The following example illustrates the structure of the resource:

apiVersion: networking.gke.io/v1
kind: CustomNetworkInterfaceConfig
metadata:
   name: myCustomNetworkInterfaceConfig
   namespace: nf-operator
spec:
  nodeName: MyNode1
  interfaces:
    - ifname: gdcenet0.500
      type: VLAN
      vlan:
        id: 500
        masterInterface: gdcenet0
    - ifname: gdcenet0.600
      type: VLAN
      vlan:
        id: 600
        masterInterface: gdcenet0
    - ifname: gdcenet0.700
      type: VLAN
      vlan:
        id: 700
        masterInterface: gdcenet0

Upgrade CustomNetworkInterfaceConfig resources from Distributed Cloud Edge 1.2.2 to 1.3.0

If you are upgrading from Distributed Cloud Edge 1.2.2 to Distributed Cloud Edge 1.3.0 or later, you must complete the steps in this section. If you are upgrading from Distributed Cloud Edge 1.3.0 to a later version, skip this section and complete the steps in the next section.

For each Distributed Cloud Edge cluster on which you have deployed one or more CustomNetworkInterfaceConfig resources, you must complete the following steps to upgrade the cluster to Distributed Cloud Edge version 1.3.0:

  1. Back up each deployed CustomNetworkInterfaceConfig resource in each namespace on the cluster. For example:

    kubectl get customnetworkinterfaceconfigs.networking.gke.io -A -o yaml > CustomNetworkInterfaceConfigsBackup.yaml

  2. Delete each deployed CustomNetworkInterfaceConfig resource by applying an empty resource definition in its place. For example:

    apiVersion: networking.gke.io/v1
kind: CustomNetworkInterfaceConfig
metadata:
  name: RESOURCE_NAME
  namespace: NAMESPACE
spec:
  nodeName: NODE_NAME

    Replace the following:

    • RESOURCE_NAME is the name of the target CustomNetworkInterfaceConfig resource.
    • NAMESPACE is the name of the target namespace.
    • NODE_NAME is the name of the target Distributed Cloud Edge node.

    When you apply the empty resource definition, the corresponding Distributed Cloud Edge node reboots.

  3. Upgrade the Distributed Cloud Edge cluster to version 1.3.0.

  4. Re-apply the backed up CustomNetworkInterfaceConfig resources onto their respective namespaces.

Upgrade CustomNetworkInterfaceConfig resources from Distributed Cloud Edge 1.3.0 to 1.4.0

Distributed Cloud Edge version 1.4.0 replaces the bond0 interface with a new interface named gdcenet0. The gdcenet0 interface allows you to use the host management network interface card (NIC) in each Distributed Cloud Edge machine in your rack for your workloads, while keeping the Distributed Cloud Edge management and control plane network traffic completely separated. To take advantage of this functionality, complete the steps in this section to reconfigure your CustomNetworkInterfaceConfig resources, then follow the instructions in Configure Distributed Cloud Edge networking to provision the appropriate networks and subnetworks.

For each Distributed Cloud Edge cluster on which you have deployed one or more CustomNetworkInterfaceConfig resources, the following migration rules apply:

  • For each new CustomNetworkInterfaceConfig resource, use gdcenet0 instead of bond0 as the value of the masterInterface field. If you apply a resource that uses bond0 or an empty value for this field, Distributed Cloud Edge replaces the value with gdcenet0, then stores and applies the resource to the cluster.
  • For each existing CustomNetworkInterfaceConfig resource, replace bond0 with gdcenet0 as the value of the masterInterface field and re-apply the resource to the cluster to restore full network connectivity to the affected Pods.

NodeSystemConfigUpdate resource

The NodeSystemConfigUpdate resource allows you to make changes to the Node's operating system configuration as well as modify Kubelet flags. Changes other than sysctl changes require a Node reboot. Distributed Cloud Edge does not provide a controller to coordinate Distributed Cloud Edge Machine reboots. You must apply this resource to one Node at a time so that the parent Cluster remains up while the configuration changes take effect.

When instantiating this resource, you must specify the name of the target Node in the nodeName field.

The NodeSystemConfigUpdate resource provides the following configuration fields specific to Distributed Cloud Edge:

  • spec.containerRuntimeDNSConfig.ip: Specifies a list of IP addresses for private image registries.
  • spec.containerRuntimeDNSConfig: Specifies a list of custom DNS entries used by the Container Runtime Environment on each Distributed Cloud Edge node. Each entry consists of an ip and domain value pairs.
  • spec.kubeletConfig.cpuManagerPolicy: Specifies the Kubernetes CPUManager policy. Valid values are None and Static.
  • spec.kubeletConfig.topologyManagerPolicy: Specifies the Kubernetes TopologyManager policy. Valid values are None, BestEffort, Restricted, and SingleNumaMode.
  • spec.osConfig.hugePagesConfig: Specifies the huge page configuration per NUMA node. Valid values are 2MB and 1GB. The number of huge pages requested is evenly distributed across both NUMA nodes in the system. For example, if you allocate 16 1GB hugepages then each Node receives a pre-allocation of 8GB.
  • spec.osConfig.isolatedCpusPerSocket: Specifies the number of isolated CPUs per socket. Required if cpuManagerPolicy is set to Static.
  • spec.osConfig.cpuIsolationPolicy: Specifies the CPU isolation policy. Default policy only isolates systemd tasks from CPUs reserved for workloads. Kernel policy marks the CPUs as isolcpus and sets the rcu_nocb, nohz_full, and rcu_nocb_poll flags on each CPU.

  • spec.sysctls.NodeLevel: Specifies the sysctls parameters that you can configure globally on a Node using the Network Function operator. The configurable parameters are:

    • fs.inotify.max_user_instances
    • fs.inotify.max_user_watches
    • kernel.sched_rt_runtime_us
    • kernel.core_pattern
    • net.ipv4.tcp_wmem
    • net.ipv4.tcp_rmem
    • net.ipv4.tcp_slow_start_after_idle
    • net.ipv4.udp_rmem_min
    • net.ipv4.udp_wmem_min
    • net.core.rmem_max
    • net.core.wmem_max
    • net.core.rmem_default
    • net.core.wmem_default
    • net.netfilter.nf_conntrack_tcp_timeout_unacknowledged
    • net.netfilter.nf_conntrack_tcp_timeout_max_retrans
    • net.sctp.auth_enable
    • net.sctp.sctp_mem
    • net.ipv4.udp_mem
    • net.ipv4.tcp_mem
    • net.ipv4.tcp_slow_start_after_idle
    • net.sctp.auth_enable
    • vm.max_map_count

    You can also scope both safe and unsafe sysctls parameters to a specific Pod or namespace using the tuning Container Networking Interface (CNI) plug-in.

The NodeSystemConfigUpdate resource provides the following read-only general status fields:

  • status.lastReportTime: Most recent time status was reported for the target interface.
  • status.conditions.lastTransitionTime: Most recent tame the condition of the interface has changed.
  • status.conditions.observedGeneration: Denotes the .metadata.generation value on which the initial condition was based.
  • status.conditions.message: An informative message describing the change of the interface's condition.
  • status.conditions.reason: A programmatic identifier denoting the reason for the last change of the interface's condition.
  • status.conditions.status: Status descriptor of the condition. Valid values are True", False, and Unknown.
  • status.conditions.type: Condition type in camelCase.

The following example illustrates the structure of the resource:

apiVersion: networking.gke.io/v1
kind: NodeSystemConfigUpdate
metadata:
  name: nodeconfig-MyNode1
  namespace: default
spec:
  nodeName: MyNode1
  sysctls:
    nodeLevel:
      "net.ipv4.udp_mem" : "12348035 16464042 24696060"
  kubeletConfig:
    topologyManagerPolicy: BestEffort
    cpuManagerPolicy: Static
  osConfig:
    hugePagesConfig:
      "TWO_MB": 0
      "ONE_GB": 16
    isolatedCpusPerSocket:
      "0": 10
      "1": 10

SriovNetworkNodePolicy resource

The SriovNetworkNodePolicy resource allows you to allocate a group of SR-IOV virtual functions (VFs) on a Distributed Cloud Edge physical machine and instantiate that group as a Kubernetes resource. You can then use this resource in a NetworkAttachmentDefinition resource.

You can select each target VF by its PCIe vendor and device ID, its PCIe device addresses, or by its Linux enumerated device name. The SR-IOV Network Operator configures each physical network interface to provision the target VFs. This includes updating the network interface firmware, configuring the Linux kernel driver, and rebooting the Distributed Cloud Edge machine, if necessary.

To discover the network interfaces available on your Node, you can look up the NetworkInterfaceState resources on that Node in the nf-operator namespace.

The following example illustrates the structure of the resource:

apiVersion: sriovnetwork.k8s.cni.cncf.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: mlnx6-p2-sriov-en2
  namespace: sriov-network-operator
spec:
  deviceType: netdevice
  isRdma: true
  mtu: 9000
  nicSelector:
    pfNames:
    - enp134s0f1np1
  nodeSelector:
    edgecontainer.googleapis.com/network-sriov.capable: "true"
  numVfs: 31
  priority: 99
  resourceName: mlnx6_p2_sriov_en2

The preceding example creates a maximum of 31 VFs from the second port on the network interface named enp134s0f1np1 with an MTU value of 9000 (the maximum allowed value). Use the Node selector label edgecontainer.googleapis.com/network-sriov.capable, which is present on all Distributed Cloud Edge Nodes capable of SR-IOV.

For information on using this resource, see SriovNetworkNodeState.

SriovNetworkNodeState resource

The SriovNetworkNodeState read-only resource allows you to query the provisioning state of the SriovNetworkNodePolicy resource on a Distributed Cloud Edge Node. It returns the complete configuration of the SriovNetworkNodePolicy resource on the Node as well as a list of active VFs on the Node. The status.syncStatus field indicates whether all SriovNetworkNodePolicy resources defined for the Node have been properly applied.

The following example illustrates the structure of the resource:

apiVersion: sriovnetwork.k8s.cni.cncf.io/v1
kind: SriovNetworkNodeState
metadata:
  name: MyNode1
  namespace: sriov-network-operator
spec:
  dpConfigVersion: "1969684"
  interfaces:
  - mtu: 9000
    name: enp134s0f1np1
    numVfs: 31
    pciAddress: 0000:86:00.1
    vfGroups:
    - deviceType: netdevice
      mtu: 9000
      policyName: mlnx6-p2-sriov-en2
      resourceName: mlnx6_p2_sriov_en2
      vfRange: 0-30
status:

Status:
  Interfaces:
    Device ID:    1015
    Driver:       mlx5_core
    Link Speed:   25000 Mb/s
    Link Type:    ETH
    Mac:          ba:16:03:9e:9c:87
    Mtu:          9000
    Name:         eno1np0
    Pci Address:  0000:1a:00.0
    Vendor:       15b3
    Device ID:    1015
    Driver:       mlx5_core
    Link Speed:   25000 Mb/s
    Link Type:    ETH
    Mac:          ba:16:03:9e:9c:87
    Mtu:          9000
    Name:         eno2np1
    Pci Address:  0000:1a:00.1
    Vendor:       15b3
    Vfs:
  - Vfs:
    - deviceID: 101e
      driver: mlx5_core
      mac: c2:80:29:b5:63:55
      mtu: 9000
      name: enp134s0f1v0
      pciAddress: 0000:86:04.1
      vendor: 15b3
      vfID: 0
    - deviceID: 101e
      driver: mlx5_core
      mac: 7e:36:0c:82:d4:20
      mtu: 9000
      name: enp134s0f1v1
      pciAddress: 0000:86:04.2
      vendor: 15b3
      vfID: 1
      .. <omitted 29 other VFs here>
  syncStatus: Succeeded

For information on using this resource, see SriovNetworkNodeState.

NetworkAttachmentDefinition resource

The NetworkAttachmentDefinition resource allows you to attach Distributed Cloud Edge Pods to one or more logical or physical networks on your Distributed Cloud Edge Node. It leverages the Multus-CNI framework and the SRIOV-CNI plug-in plug-in.

To do so, use an annotation to reference the name of the appropriate SriovNetworkNodePolicy resource. You must do the following when creating this annotation:

  • Use the key k8s.v1.cni.cncf.io/resourceName.
  • Use the prefix gke.io/ in its value, followed by the name of the target SriovNetworkNodePolicy resource.

The following example illustrates the structure of the resource:

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-net1
  namespace: mynamespace
  annotations:
    k8s.v1.cni.cncf.io/resourceName: gke.io/mlnx6_p2_sriov_en2
spec:
  config: '{
  "type": "sriov",
  "cniVersion": "0.3.1",
  "name": "sriov-network",
  "ipam": {
    "type": "host-local",
    "subnet": "10.56.217.0/24",
    "routes": [{
      "dst": "0.0.0.0/0"
    }],
    "gateway": "10.56.217.1"
  }
}'

Upgrade NetworkAttachmentDefinition resources to Distributed Cloud Edge 1.4.0

Distributed Cloud Edge version 1.4.0 replaces the bond0 interface with a new interface named gdcenet0. The gdcenet0 interface allows you to use the host management network interface card (NIC) in each Distributed Cloud Edge machine in your rack for your workloads, while keeping the Distributed Cloud Edge management and control plane network traffic completely separated. To take advantage of this functionality, complete the steps in this section to reconfigure your NetworkAttachmentDefinition resources, then follow the instructions in Configure Distributed Cloud Edge networking to provision the appropriate networks and subnetworks.

For each Distributed Cloud Edge cluster on which you have deployed one or more NetworkAttachmentDefinition resources, the following migration rules apply:

  • For each new NetworkAttachmentDefinition resource, use gdcenet0 instead of bond0 as the value of the master field. If you apply a resource that uses bond0 or an empty value for this field, Distributed Cloud Edge replaces the value with gdcenet0, then stores and applies the resource to the cluster.
  • For each existing NetworkAttachmentDefinition resource, replace bond0 with gdcenet0 as the value of the master field and re-apply the resource to the cluster to restore full network connectivity to the affected Pods.

For information on using this resource, see NetworkAttachmentDefinition.

Configure a secondary interface on a Pod using SR-IOV VFs

After you have configured a SriovNetworkNodePolicy resource and a corresponding NetworkAttachmentDefinition resource, you can configure a secondary network interface on a Distributed Cloud Edge Pod using SR-IOV virtual functions.

To do so, add an annotation to your Distributed Cloud Edge Pod definition as follows:

  • Key: k8s.v1.cni.cncf.io/networks
  • Value: nameSpace/<NetworkAttachmentDefinition1,nameSpace/NetworkAttachmentDefinition2...

The following example illustrates this annotation:

apiVersion: v1
kind: Pod
metadata:
  name: sriovpod
  annotations:
    k8s.v1.cni.cncf.io/networks: mynamespace/sriov-net1
spec:
  containers:
  - name: sleeppodsriov
    command: ["sh", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine
    securityContext:
      capabilities:
        add:
          - NET_ADMIN

Configure a secondary interface on a Pod using the MacVLAN driver

Distributed Cloud Edge also supports creating a secondary network interface on a Pod using the MacVLAN driver. Only the gdcenet0 interface supports this configuration and only on Pods running containerized workloads. To configure an interface to use the MacVLAN driver, first configure a NetworkAttachmentDefinition resource as shown in the following example:

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-b400-1
spec:
  config: '{
  "type": "macvlan",
  "master": "gdcenet0.400",
  "ipam": {
    "type": "static",
    "addresses": [
      {
        "address": "192.168.100.20/27",
        "gateway": "192.168.100.1"
      }
    ]
  ...
  }
}'

Next, add an annotation to your Distributed Cloud Edge Pod definition as follows:

apiVersion: v1
kind: Pod
metadata:
  name: macvlan-testpod1
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-b400-1

Configure a secondary interface on a Pod using Distributed Cloud Edge multi-networking

Distributed Cloud Edge supports creating a secondary network interface on a Pod using its multi-network feature. To do so, complete the following steps:

  1. Configure a Network resource. For example:

    apiVersion: networking.gke.io/v1alpha1
kind: Network
metadata:
  name: vlan200-network
spec:
  type: L2
  nodeInterfaceMatcher:
    interfaceName: vlan200-interface
  gateway4: 10.53.0.1

  2. Configure a NetworkInterface resource. For example:

    apiVersion: networking.gke.io/v1alpha1
kind: NetworkInterface
metadata:
 name: multinet-vlan200
 namespace: default
spec:
 networkName: vlan200-network
 ipAddresses:
  - 10.53.0.2/24
 macAddress: 86:fe:36:83:56:1

  3. Add an annotation to your Distributed Cloud Edge Pod definition as follows:

    apiVersion: v1
kind: Pod
metadata:
  name: myPod
  annotations:
    networking.gke.io/interfaces: [{"interfaceName":"eth1","interface":"multinet-vlan200"}]
    networking.gke.io/default-interface: eth1
...