Edit on GitHub
Report issue
Page history

Calculating network throughput

Author(s): @dcavalheiro ,   Published: 2019-08-07


This document describes how to calculate network throughput, both within GCP and to your on-premises or third-party cloud locations. This document includes information on how to analyze results, explanations of variables that can affect network performance, and troubleshooting tips.

Common use cases

Choosing and configuring your applications and machine types for optimal network throughput is essential for balancing costs and networking performance.

Here are the common use cases for a network throughput assessment:

Know your tools

When calculating network throughput, it's important to use well-tested, well-documented tools:

  • iPerf3: A commonly used network testing tool that can create TCP/UDP data streams (single-thread or multi-thread) and measure the throughput of the network that carries them.
  • traceroute: A computer network diagnostic tool that measures and displays the routes that packets take across a network. The traceroute tool records the route's history as the round-trip times of packets received from each successive host in a route. The sum of the mean times in each hop is an approximate measure of the total time spent to establish the TCP connection.
  • tcpdump: A command-line packet analyzer that captures packet details and TCP/IP communications for more advanced troubleshooting. tcpdump is compatible with other tools, such as Wireshark.
  • TCP throughput calculator: A calculator on the SWITCH Foundation website that measures theoretical network limits based on the TCP window and RTT.
  • Perfkit Benchmarker: A tool that contains a set of benchmarks to measure and compare cloud offerings. The benchmarks use defaults to reflect what most users will see.
  • gcping: A command-line tool that provides median latencies to GCP regions.

Round-trip time, latency, and distances

When analyzing network performance, one of the most common mistakes is to measure data transfer speed without considering the characteristics of the network or the actual distance between network endpoints. Network performance analysis is highly dependent on factors such as latency and distance. You need to consider these real-world characteristics, and not rely on simple assumptions.

For example, the speed of light traveling in a vacuum is around 300,000 km/s, but the speed of light through the fiber-optic cable that makes up most of the internet is approximatey 200,000 km/s.

Another important consideration is that the distance between two virtual machines (VMs) isn't normally a straight line, but is instead a complex maze of connections, increasing the total distance traveled. This is analogous to the fact that the distance between New York City and Los Angeles is almost 3,944 km in a straight line, but driving on roads increases this value to at least 4,466 km.

Because of the bandwidth-delay product, even connections with large bandwidth capabilities will perform poorly when tested via TCP, since their TCP window may not allow for full link utilization.

Choosing the closest GCP location to your on-premises location is the best approach to reduce latency.

You can use the gcping command-line tool to determine median latencies from any location to various GCP regions.

Egress per virtual machine limit (all network interfaces)

To choose the best egress bandwidth cap for your application, refer to the per-instance VPC resource limits for the maximum egress data rate.

Single-thread and multi-thread connection modes

Another important concept is the use of single or parallel threads when sending data over a network connection. For example, a machine's CPU can process network requests over a single or over multiple processor threads. Multi-connection, multi-thread, or parallel-thread mode gives you maximum potential speed by using multiple CPU threads in parallel to send network packets.

For example, if you sequentially download a single file from start to end, you use a single thread, but when you download many smaller files, or break a large file into small chunks and download them at the same time, each thread is responsible for each file or chunk.

it is possible to achieve values close to the maximum egress data rate using multi-thread mode. However, because the maximum egress data rate is a limit, the data rate will not go over the maximum egress data rate, even under ideal conditions.

Single connection mode is best for testing over a VPN (non HA) or simulating the download of a single file. Expect it to have lower transfer speeds than multi-thread mode.

Measuring throughput with iPerf3

Follow this procedure to measure throughput from the perspective of a single VM.

Choose a large machine type

To perform throughput tests, we recommend that you use a large machine type, such as n1-standard-8. This machine type provides a maximum egress throughput limit of 16 Gbps, so the per-VM egress throughput will not interfere with the tests.

Install tcpdump and iPerf3

Install tcpdump and iPerf3 on a Linux VM machine.

To install both pieces of software on a Debian or Ubuntu system, run the following commands:

sudo apt-get update
sudo apt-get install tcpdump
sudo apt-get install iperf3

Run an iPerf3 server

The iPerf3 server accepts connections on both the VM's internal address and (if configured) its external IP address. By default, the server listens on port 5201 for both TCP and UDP traffic. To change the port, use the -p flag.

iperf3 -s -p [PORT_NUMBER]

The -s flag tells iPerf3 to run as a server.

Allow the server and client communication through the GCP firewall

Create a firewall rule to the iPerf3 server to allow ingress TCP on the selected port, as described in Creating firewall rules.

Run an iPerf3 client

Run iPerf3 with the -c (client) flag and specify the destination IP address of the iPerf3 server. By default, iPerf3 uses TCP, unless you specify the -u (UDP) flag for the client. No such flag is needed server-side for UDP; however, a firewall rule allowing incoming UDP traffic to the server is required.

If you run the server on a custom port, you'll need to specify that same port using the -p (port) flag. If you omit the port flag, the client assumes that the destination port is 5201.

Use the -P (parallel threads) flag to specify a number of simultaneous threads, and use the -t (time) flag to specify the duration of the test, in seconds.

iperf3 -c [VM_IP] -P [THREADS] -p [PORT_NUMBER] -t [DURATION] -R

Capturing packets

It's sometimes helpful to get packet captures on the iPerf3 client or server. The following example command shows you how to capture full packets for a given destination port range from an eth0 interface, saving a file in the working directory called mycap.pcap.

sudo /usr/sbin/tcpdump -s 100 -i eth0 dst port [PORT_NUMBER] -w mycap.pcap

You'll have to modify this command to suit your use case; for example, the interface name isn't always eth0 on all distributions, and you'll need to specifically choose the right interface for a multiple network interface VM.

Measuring VPC network throughput with PerfKit Benchmarker

To measure network throughput performance from a given machine type in a specific zone, use PerfKit Benchmarker. This tool is very valuable because it creates an instance and measures its performance, without the need to install the tools on an existing VM.

Replace the placeholders in the commands below with the following:

  • [MACHINE_TYPE]: The machine type that you want to test (for example, n1-standard-32).
  • [ZONE]: The zone to create the instance in.
  • [NUMBER_OF_VCPUS]: The number of vCPUs of the instance (for example, 32 for the n1-standard-32 machine type).

Measure single thread performance:

./pkb.py --cloud=GCP --machine_type=[MACHINE_TYPE]
--benchmarks=iperf --ip_addresses=INTERNAL --zones=[ZONE]

Measure multi-thread performance:

./pkb.py --cloud=GCP --machine_type=,[MACHINE_TYPE]
--benchmarks=iperf --ip_addresses=INTERNAL --zones=[ZONE]
--iperf_sending_thread_count=[NUMBER_OF_VCPUS]

Speed test

To measure network data transfer speed from a VM to the global internet, you can choose from many available tools, including the following:

Troubleshooting

This section includes tips for investigating and troubleshooting some common issues that may occur when measuring network throughput.

iPerf3 server and client are not able to establish a connection

  1. Verify that the firewall rules allow ingress and egress traffic to and from the VM on your VPC.
  2. If using multiple VPCs, the firewall must allow traffic on the selected port for all VPCs.
  3. Selecting the nic0 of the instance will open a console page with the breakdown of the firewall rules and route that affect this specific instance, which is a very valuable source of information.

TCP window size and RTT not optimized

If you are not able to realize your full connection bandwidth over a TCP connection, the cause may be TCP send and receive windows. For an explanation of this behavior, see this video on the bandwidth delay problem. You can use the TCP throughput calculator to understand the impact of window size on your connection bandwidth.

To fix this issue, you may increase the TCP send and receive windows:

  • SO_RCVBUF is influenced by increasing net.core.rmem_default and net.core.rmem_max.
  • SO_SNDBUF is influenced by increasing net.core.wmem_default and net.core.wmem_max.

Submit a Tutorial

Share step-by-step guides

SUBMIT A TUTORIAL

Request a Tutorial

Ask for community help

SUBMIT A REQUEST

GCP Tutorials

Tutorials published by GCP

VIEW TUTORIALS

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.