Connectivity Tests is a diagnostics tool that lets you check connectivity between network endpoints. It analyzes your configuration and, in some cases, performs live data plane analysis between the endpoints. An endpoint is a source or destination of network traffic, such as a VM, Google Kubernetes Engine (GKE) cluster, load balancer forwarding rule, or an IP address on the internet.
To analyze network configurations, Connectivity Tests simulates the expected forwarding path of a packet through your Virtual Private Cloud (VPC) network, Cloud VPN tunnels, or VLAN attachments. Connectivity Tests can also simulate the expected inbound forwarding path to resources in your VPC network.
For some connectivity scenarios, Connectivity Tests also performs live data plane analysis. This feature sends packets over the data plane to validate connectivity and provides baseline diagnostics of latency and packet loss. If the route is supported for the feature, each test that you run includes a live data plane analysis result.
The API for Connectivity Tests is the Network Management API. For more information, see the API documentation.
Why use Connectivity Tests?
Connectivity Tests can help you troubleshoot the following network connectivity issues:
- Unintended inconsistent configurations
- Obsolete configurations caused by network configuration changes or migrations
- Configuration errors for a variety of network services and functions
When testing Google-managed services, Connectivity Tests can also help you determine whether there is an issue in your VPC network or in the Google-owned VPC network used for the service resources.
How Connectivity Tests analyzes configurations
When analyzing network configurations, Connectivity Tests uses an abstract state machine to model how a VPC network should process packets. Google Cloud processes a packet in several logical steps.
Because of the variety of VPC network services and features that the configuration analysis supports, a test packet traversing a VPC network configuration can take many possible paths.
The following diagram shows a model for how the configuration analysis simulates trace traffic between two Compute Engine virtual machine (VM) instances—one on the left and another on the right.
Depending on your Google Cloud network and resource configurations, this traffic might go through a Cloud VPN tunnel, a Google Cloud load balancer, or a peered VPC network before reaching the destination VM instance.
The bounded number of steps between discrete states until a packet has been delivered or dropped is modeled as a finite state machine. This finite state machine can be in exactly one of many finite states at any one time and might have multiple successor states.
For example, when Connectivity Tests matches several routes according to route precedence, Google Cloud can choose a route among several routes based on an unspecified hashing function in the data plane.
If a policy-based route is configured, Connectivity Tests routes the packet to the next hop, which is an internal load balancer.
In the previous case, the Connectivity Tests trace returns all of the possible routes but can't determine the method Google Cloud used to return the routes. This is because that method is internal to Google Cloud and is subject to change.
Google-managed services
Google-managed services, such as Cloud SQL and Google Kubernetes Engine (GKE), allocate resources for customers in projects and VPC networks that Google owns and manages. Customers do not have permission to access these resources.
The Connectivity Tests configuration analysis can still run a test and provide an overall reachability result for Google-managed services, but it does not provide details for the tested resources in the project owned by Google.
The following diagram shows a model for how the configuration analysis simulates trace traffic from a VM instance in a customer VPC network to a Cloud SQL instance in the Google-owned VPC network. In this example, the networks are connected through VPC Network Peering.
Similar to a standard test between two VMs, the logical steps include checking the relevant egress firewall rules and matching the route. When you run a test, Connectivity Tests configuration analysis provides details about these steps. However, for the final logical step of analyzing the configuration in the Google-owned VPC network, the analysis provides only an overall reachability result. Connectivity Tests does not provide details for the resources in the Google-owned project because you do not have permission to view them.
For more information, see the test examples in Test connectivity to and from Google-managed services.
Supported configurations
The Connectivity Tests configuration analysis supports testing the network configurations described in the following sections.
Traffic flows
- VM instances to and from the internet
- VM instance to VM instance
- From Google Cloud to and from on-premises networks
- Between two on-premises networks that are connected through Network Connectivity Center
VPC networking features
You can test connectivity between resources that use the following features:
- VPC Network Peering
- Shared VPC
- Private Google Access
- Alias IP ranges
- Private Service Connect
- Private IP addresses outside of the RFC 1918 address range
- A Compute Engine VM instance with multiple network interfaces
- Custom routes imported from peered VPC networks
- VPC transitive routing
- VPC firewall rules
- Hierarchical firewall policies and global network firewall policies
- Policy-based routes
- Dual-stack instances with both IPv4 and IPv6 addresses, including instances with multiple network interfaces
Google Cloud hybrid networking solutions
The following hybrid networking solutions are supported:
- Cloud VPN
- Cloud Interconnect
- Cloud Router, including dynamic routes that use BGP and static routes
Network Connectivity Center
Network Connectivity Center is supported.
Cloud NAT
Cloud NAT is supported.
Cloud Load Balancing
- Connectivity to the following Google Cloud load balancers is supported: external HTTP(S) load balancers, network load balancers, internal HTTP(S) load balancers, internal TCP/UDP load balancers, external TCP proxy load balancers, and external SSL proxy load balancers.
- Connectivity to load balancer IP addresses is supported.
- Connectivity of Cloud Load Balancing health checks to backends is supported. Backends can be instance groups or network endpoint groups (NEGs).
- Internal TCP/UDP load balancers can be used as next hops.
For Cloud Load Balancing features that are unsupported, see the Unsupported configurations section.
Google Kubernetes Engine (GKE)
- Connectivity to and between GKE
nodes and the GKE control plane is supported.
- When testing the private IP address of a GKE control plane, Connectivity Tests configuration analysis determines whether the packet can be delivered to the control plane. This includes analyzing the configuration within the Google-owned VPC network.
- When testing the public IP address of a GKE control plane, Connectivity Tests configuration analysis determines whether the packet can be sent to the Google-owned VPC network where the control plane runs. This does not include analyzing the configuration within the Google-owned VPC network.
- Connectivity to the GKE service through Cloud Load Balancing is supported.
- Connectivity to a GKE pod in a VPC-native cluster is supported.
For GKE features that are unsupported, see the Unsupported configurations section.
Other Google Cloud products and services
The following additional Google Cloud products or services are supported:
- Cloud SQL instances are supported.
- When testing the private IP address of a Cloud SQL instance, the configuration analysis determines whether the packet can be delivered to the instance. This includes analyzing the configuration within the Google-owned VPC network.
- When testing the public IP address of a Cloud SQL instance, the configuration analysis determines whether the packet can be sent to the Google-owned VPC network where the instance runs. This does not include analyzing the configuration within the Google-owned VPC network.
- Cloud Functions (1st gen) is supported.
- Cloud Run revisions are supported.
- App Engine standard environment is supported.
Unsupported configurations
The Connectivity Tests configuration analysis does not support testing the following network configurations:
- Google Cloud Armor policies are not considered or used when tracing connectivity to an external HTTP(S) load balancer IP address.
- GKE network policy. Connectivity Tests does not incorporate network policies when tracing connectivity to IP addresses within GKE clusters.
- An external HTTP(S) load balancer that uses Cloud Storage buckets is not supported.
- Cloud Functions (2nd gen) is not supported. However, you can test Cloud Functions (2nd gen) by creating a Connectivity Test for the underlying Cloud Run revision. A Cloud Run revision is created every time a Cloud Function is deployed.
- App Engine flexible environment is not supported.
- Cloud Run jobs are not supported. For more information, see Services and jobs: two ways to run your code.
- Resource Manager tags for firewalls are not supported. Connectivity Tests does not analyze secure tags attached to VM instances.
- Regional network firewall policies are not supported.
How Connectivity Tests analyzes the live data plane
The live data plane analysis feature tests connectivity by sending multiple trace packets from the source endpoint to the destination. The live data plane analysis results show you the number of probes sent, the number of probes that successfully reached the destination, and a reachability status. This status is determined based on how many probes were successfully delivered, as described in the following table.
| Status | Number of probes that reached their destination |
|---|---|
| Reachable | At least 95% |
| Unreachable | None |
| Partially reachable | More than 0 and less than 95% |
In addition to showing how many packets were successfully delivered, dynamic verification also shows median and 95th percentile one-way latency information.
Live data plane analysis does not depend on the configuration analysis. Rather, live data plane analysis provides an independent assessment of the connectivity state.
If you see apparent discrepancies between the configuration analysis and live data plane analysis results, see Troubleshoot Connectivity Tests.
Supported configurations
Live data plane analysis supports the following network configurations.
Traffic flows
- Between two VM instances
- Between a VM instance and a Cloud SQL instance
- Between a VM instance and a Google network edge location
- IP protocols: TCP, UDP
VPC networking features
You can dynamically verify connectivity between resources that use the following features:
- VPC Network Peering
- Shared VPC
- Alias IP ranges
- External IP addresses
- Internal IP addresses, private IP addresses outside of the RFC 1918 address range
- Custom routes
- Layer 4 internal TCP/UDP load balancers, for tests between a VM and an internal TCP/UDP load balancer
- Ingress firewall rules, including ingress hierarchical firewall policy rules and ingress VPC firewall rules
- Dual-stack instances with both IPv4 and IPv6 addresses, including instances with multiple network interfaces
Unsupported configurations
All configurations not explicitly listed as supported are not supported. In addition, configurations where connectivity is blocked by egress firewall rules are not supported.
For any given test, if the live data plane analysis feature is not executed,
an N/A or - appears in the Last packet transmission result field.
Considerations
Evaluate the following considerations when deciding whether to use Connectivity Tests.
The configuration analysis that Connectivity Tests performs is based entirely on configuration information for Google Cloud resources and might not represent the actual condition or status of the data plane for a VPC network.
While Connectivity Tests does acquire some dynamic configuration information, such as the Cloud VPN tunnel state and dynamic routes on Cloud Router, it does not access or maintain the health state of Google internal production infrastructure and data plane components.
A status of Packet could be delivered for a Connectivity Test
does not guarantee that traffic can pass through the data plane. The purpose of
the test is to validate configuration issues that can cause traffic to drop.
For supported routes, the live data plane analysis results supplement the configuration analysis results by testing whether transmitted packets arrive at the destination.
Connectivity Tests has no knowledge of networks outside of Google Cloud
Outside networks are defined as the following:
- On-premises networks that reside in your data center or other facility where you operate your hardware devices and software applications.
- Other cloud providers where you run resources.
- A host on the internet that sends traffic to your VPC network.
Connectivity Tests doesn't perform firewall connection tracking
Connection tracking for VPC firewalls stores information about new and established connections and enables allowing or restricting subsequent traffic based on that information.
The Connectivity Tests configuration analysis doesn't support firewall connection tracking because the firewall connection table is located in the data plane for a VM instance and is inaccessible. However, the configuration analysis can simulate connection tracking by allowing a return connection that would normally be denied by an ingress firewall rule, as long as Connectivity Tests initiates the outbound connection.
Live data plane analysis does not support testing firewall connection tracking.
Connectivity Tests can't test VM instances configured to modify forwarding behavior
Connectivity Tests can't test VM instances that have been configured to act in the data plane as routers, firewalls, NAT gateways, VPNs, and so on. This type of configuration makes it difficult to assess the environment running on the VM instance. Additionally, live data plane analysis doesn't support this testing scenario.
Connectivity Tests result times can vary
Getting Connectivity Tests results can take from 30 seconds to up to 10 minutes. The time a test takes is based on the size of your VPC network configuration and the number of Google Cloud resources that you use.
The following table shows response times that you can expect for all users running a test against a sample configuration in a query. This configuration contains VM instances, a Cloud VPN tunnel, and Google Cloud load balancers.
| Project size | Number of Google Cloud resources | Response latency |
|---|---|---|
| Small project | Fewer than 50 | 60 seconds for 95% of queries from all users |
| Medium project | Greater than 50 but fewer than 5000 | 120 seconds for 95% of queries from all users |
| Large project | Greater than 5000 | 600 seconds for 95% of queries from all users |
Live data plane analysis is not intended for continuous monitoring
Live data plane analysis performs one-time verification of network connectivity for diagnostic purposes. For continuous monitoring of connectivity and packet loss, use Performance Dashboard.
Live data plane analysis doesn't test multiple traces
Live data plane analysis is not supported in cases where the route is not deterministic.
What is reachability?
A resource is reachable from another endpoint if network configurations such as firewalls and routes allow traffic to get from one to the other. For example, if the network configuration should allow VM1 to send packets to VM2, then VM2 is said to be reachable from VM1.
Connectivity Tests measures reachability from a particular source to a particular destination. The fact that VM1 can reach VM2 does not necessarily mean that VM3 can reach VM2.
Connectivity Tests measures unidirectional reachability. The fact that VM1 can open a connection to VM2 does not mean that VM2 can open a connection with VM1. Firewall rules might allow traffic in one direction, but not the other.
Connectivity Tests measures reachability for a particular protocol and
destination port. The fact that VM1 can reach VM2 on tcp:443 does not mean it
can reach it on tcp:80.
Connectivity Tests only tests Google Cloud VPC network configurations that might affect packet delivery from the source to the destination. It does not check to see whether a valid server is running on the destination, whether operating system firewall rules might block the traffic, or if security software blocks a packet carrying a virus payload.
The concept of Reachability originated in graph theory. Conceptually, the entire reachability graph of a network contains all of the endpoints as nodes, and directional edges that indicate reachability from source nodes to destination nodes.
Reachability analysis is a more generic term that describes a body of analysis that can be conducted to determine network reachability. One of the use cases from a reachability analysis is a connectivity test. Connectivity, in this case, refers to the state of network connections.
For every step along the network forwarding path, a reachability analysis tests and provides results for the underlying network configuration. For example, Connectivity Tests analyzes the Google Cloud firewall rules and routes that are applied to a simulated test packet.
How Connectivity Tests works
Connectivity Tests includes two main components: a configuration analysis and live data plane analysis. This section describes how these two types of analyses work.
How configuration analysis works
This section describes how Connectivity Tests and its components work.
Connectivity Tests performs a reachability analysis that evaluates the Google Cloud resources in your testing path against an ideal configuration model. It is augmented by the live data plane analysis feature, which sends packets to verify the state of the data plane and provide baseline information for supported configurations. For details about how dynamic verification works, see How live data plane analysis works.
As a network administrator, you have control over many configurations that could impact the analysis result, although some exceptions apply. For example, you do not have control over VPC networks that host Google-managed services such as Cloud SQL instances. Additionally, because of permissions restrictions, you might not have control over hierarchical firewall policy rules that affect your network.
When you run a Connectivity Test, you input a specific set of parameters and receive formatted results in the form of a network trace, or query. A Connectivity Test generates more than one trace if a test has multiple possible paths in the network (for example, when the destination endpoint is a Google Cloud load balancer with multiple backends).
- A match means that Connectivity Tests finds a Google Cloud configuration that allows the simulated packet to continue through the testing path.
- No match means that Connectivity Tests can't find a match. Thus, the configuration doesn't exist.
- A denied match means that Connectivity Tests finds a Google Cloud configuration where the simulated test packet must be dropped.
Connectivity Tests components
A Connectivity Test is the top-level component that contains all other testing sub-components needed for the configuration analysis. These components include the following:
- Source and destination endpoints
- Reachability details for the test and its traces, including an overall reachability result determined by the configuration analysis
- One or more traces that each contain one or more steps
- A state for each step
Each test has a unique name and each step has a state and Info metadata
associated with it. For example, if a step checks a route, RouteInfo
metadata is included in that step.
The following diagram shows a test from one Compute Engine VM instance to another. For descriptions of the test components, see the sections that follow.
Source and destination endpoints
The Connectivity Tests configuration analysis supports a 5-tuple packet header without the source port. This is because the source port is not used for validating resources in Google Cloud network configurations. Thus, you don't need to provide it when running tests.
The packet header contains the following components:
- A network protocol
- A source endpoint, consisting of one of the following:
- A source IP address
- A VM instance name
- A cluster name for a GKE control plane
- A Cloud SQL instance name
- A destination endpoint, consisting of one of the following and a port number:
- A destination IP address
- A VM instance name
- A cluster name for a GKE control plane
- A Cloud SQL instance name
You can also specify a Google Cloud or non-Google Cloud network type or a combination of a network type and IP address or VM instance name to uniquely identify a network location.
The following networking protocols are supported for VM, IP address, and Google-managed services:
- TCP
- UDP
- ICMP
- ESP
- AH
- SCTP
- IPIP
The following networking protocols are supported by Serverless VPC Access connectors:
- TCP
- UDP
Destination ports for the TCP or UDP protocols are supported. If you don't
specify a port, the default setting is port 80.
Traces, steps, and states
The configuration analysis contains one or more traces. Each trace represents one unique, simulated packet forwarding path in a test.
- Each trace contains multiple ordered steps.
- Each step contains a state related to the Google Cloud configuration that Connectivity Tests checks for that step.
- States are categorized into non-final and final states.
Non-final states
Non-final states represent a configuration check for each Google Cloud resource in the testing path, such as a VM instance, endpoint, firewall rule, route, or Google Cloud load balancer.
There are four non-final states:
- Initial
- Config checking
- Forwarding
- Transition
For more information, see Configuration analysis states.
Final state
Each trace must end with a final state, which is the last step in the trace.
There are four possible final states:
DropAbortForwardDeliver
Each state has a reason associated with it. For more information, see the details for each final state.
Overall reachability result
The configuration analysis also provides an overall reachability result that can
take one of four values: Reachable, Unreachable, Ambiguous, or
Undetermined.
Knowing the overall reachability result can be helpful for setting up monitoring or automation.
For more information, see Overall reachability result.
Spoof checking
Connectivity Tests performs a spoof check when a simulated packet to or from a VM instance uses an IP address not owned by that instance. IP addresses owned by a VM include all VM internal IP addresses and secondary IP addresses.
If the address is an address that appears to originate from external traffic, also called a foreign address, then the IP address fails the spoof check.
Metadata
Each state can have metadata associated with it in the form of an Info field.
For example, InstanceInfo contains details for a VM instance, including
the name and IP address.
The configuration analysis provides metadata for the test itself and metadata for each step in a test.
How live data plane analysis works
The probing mechanism for live data plane analysis does not involve the guest OS and is fully transparent to the user. Probes are injected on behalf of the source endpoint to the network and are dropped just before being delivered to the destination endpoint. Probes are excluded from regular network billing, telemetry metrics, and flow logs.
VPC Service Controls support (Preview)
VPC Service Controls can provide additional security for Connectivity Tests to help mitigate the risk of data exfiltration. Using VPC Service Controls, you can add projects to service perimeters that protect resources and services from requests that originate outside the perimeter.
To learn more about service perimeters, see the Service perimeter details and configuration page of the VPC Service Controls documentation.
What's next
Identify and fix ICMP issues (tutorial)