Using Google Virtual NIC


This page provides instructions for enabling Google Virtual NIC (gVNIC) in a Google Kubernetes Engine (GKE) cluster.

Overview

Google Virtual NIC (gVNIC) is a virtual network interface. Installing the gVNIC driver allows for more efficient traffic transmission across the Google network infrastructure. gVNIC is an alternative to the virtIO-based ethernet driver.

gVNIC is required to support higher network bandwidths such as the 50-100 Gbps speeds that can be used for distributed workloads on VMs that have attached GPUs. Also, gVNIC is required when working with some VM shapes that are meant for optimal performance when using VMs.

Limitations

Requirements

  • GKE nodes must use a Container-Optimized OS node image.
  • GKE node version 1.15.11-gke.15 or later.

Before you begin

Before you start, make sure you have performed the following tasks:

  • Ensure that you have enabled the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • Ensure that you have installed the Cloud SDK.
  • Set up default gcloud command-line tool settings for your project by using one of the following methods:
    • Use gcloud init, if you want to be walked through setting project defaults.
    • Use gcloud config, to individually set your project ID, zone, and region.

    gcloud init

    1. Run gcloud init and follow the directions:

      gcloud init

      If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

      gcloud init --console-only
    2. Follow the instructions to authorize the gcloud tool to use your Google Cloud account.
    3. Create a new configuration or select an existing one.
    4. Choose a Google Cloud project.
    5. Choose a default Compute Engine zone.
    6. Choose a default Compute Engine region.

    gcloud config

    1. Set your default project ID:
      gcloud config set project PROJECT_ID
    2. Set your default Compute Engine region (for example, us-central1):
      gcloud config set compute/region COMPUTE_REGION
    3. Set your default Compute Engine zone (for example, us-central1-c):
      gcloud config set compute/zone COMPUTE_ZONE
    4. Update gcloud to the latest version:
      gcloud components update

    By setting default locations, you can avoid errors in gcloud tool like the following: One of [--zone, --region] must be supplied: Please specify location.

Creating a cluster with default node pools that uses gVNIC

You can enable gVNIC in a new cluster on GPU nodes.

To create a cluster with gVNIC on GPU nodes, use the gcloud container cluster create command and specify the --enable-gvnic flag as in the following command:

gcloud container cluster create CLUSTER_NAME \
    --accelerator type=ACCELERATOR_TYPE,count=ACCELERATOR_COUNT \
    --machine-type=MACHINE_TYPE \
    --enable-gvnic

Replace the following:

  • CLUSTER_NAME: the name of the new cluster.
  • ACCELERATOR_TYPE: the type of GPUs you want to attach to the nodes in the cluster.
  • ACCELERATOR_COUNT: the number of GPUs per node.
  • MACHINE_TYPE: the type of machine you want to use. Note that gVNIC is not supported on memory-optimized machine types.

Example: Cluster with GPU nodes

In the following example, we create a cluster that has three nodes using the n1-standard-1 machine type, with each node using one nvidia-tesla-v100 GPU, and with gVNIC enabled:

gcloud container clusters create gpu-gvnic-cluster \
    --accelerator type=nvidia-tesla-v100,count=1 \
    --machine-type= n1-standard-1 \
    --enable-gvnic

Creating a node pool with gVNIC enabled

To create a node pool with gVNIC enabled in an existing cluster, use the gcloud container node-pools create command and specify the --enable-gvnic flag as in the following command:

gcloud container node-pools create NODEPOOL_NAME \
    --cluster=CLUSTER_NAME \
    --enable-gvnic

Replace the following:

  • NODEPOOL_NAME: the name of the new node pool that you want to create.
  • CLUSTER_NAME: the name of the existing cluster.

Example: New node pool with GPU nodes

In the following example, we create a node pool that has three nodes using the n1-standard-1 machine type, with each node using one nvidia-tesla-v100 GPU, and with gVNIC enabled:

gcloud container node-pools create gvnic-pool \
    --cluster=gpu-gvnic-cluster \
    --accelerator type=nvidia-tesla-v100,count=1 \
    --machine-type= n1-standard-1 \
    --enable-gvnic

Update an existing node pool to enable gVNIC

You can enable gVNIC on an existing node pool by using the gcloud container node-pools update command and specifying the --enable-gvnic flag as in the following command:

gcloud container node-pools update NODEPOOL_NAME \
    --cluster=CLUSTER_NAME \
    --enable-gvnic

Replace the following:

  • NODEPOOL_NAME: the name of the existing node pool that you want to update.
  • CLUSTER_NAME: the name of the existing cluster.

Disabling gVNIC

You can disable gVNIC on an existing node pool by using the gcloud container node-pools update command and specifying the --no-enable-gvnic flag as in the following command:

gcloud container node-pools update NODEPOOL_NAME \
    --cluster=CLUSTER_NAME \
    --no-enable-gvnic

Replace the following:

  • NODEPOOL_NAME: the name of the existing node pool that you want to update.
  • CLUSTER_NAME: the name of the existing cluster.

Verifying that gVNIC is enabled

To verify that gVNIC is enabled, perform the following tasks:

  1. Get the names of the nodes in the cluster:

    kubectl describe nodes
    
  2. SSH into the node that you want to verify:

    gcloud compute ssh NODE_NAME
    
  3. Inside the node, run one of the following commands:

    • Check by listing the module:

      lsmod | grep gve
      

      If gVNIC is successfully installed, the following output displays:

      gve                    45056  0
      
    • Check by listing the driver:

      ls /sys/devices/pci0000\:00/0000:00:04.0/driver/module/drivers
      

      If gVNIC is successfully installed, the following output displays:

      pci:gvnic
      
    • Check by using the Linux ethtool:

      /usr/sbin/ethtool -i eth0
      

      If gVNIC is successfully installed, the following output displays:

      driver: gve
      

Troubleshooting

To troubleshoot gVNIC, see Troubleshooting Google Virtual NIC.

What's next