This document gives troubleshooting guidance for issues related to your connection to Google.
Network Connectivity Gateway
This section gives troubleshooting tips related to Network Connectivity Gateway.
If you need additional assistance, reach out to Cloud Customer Care.Verify that the controller and gateway Pods are running
List the Network Connectivity Gateway Pods:
kubectl --kubeconfig
CLUSTER_KUBECONFIG get pods \ --namespace kube-system | grep ncgReplace
with the path of your cluster kubeconfig file.CLUSTER_KUBECONFIG In the output, verify that you see a controller Pod and a gateway Pod. The name of the controller Pod begins with
ncg-controller
, and the name of the gateway Pod begins withncgd
. For example:ncg-controller-5f5d489b68-r87pp 1/1 Running ncgd-gvl76 1/1 Running
Make a note of the name of the gateway Pod. You will need it for various troubleshooting commands.
Check the gateway Pod logs
Check to see if there are any errors in the logs from the gateway Pod:
kubectl --kubeconfig
CLUSTER_KUBECONFIG logsGATEWAY_POD \ --namespace kube-systemReplace
with the name of your gateway Pod.GATEWAY_POD
Running ncgcli
inside the gateway Pod
The gateway Pod has a command-line tool named ncgcli
that you can use to get
information about the gateway and traffic in the tunnels.
Use the following command to run
ncgcli
inside the gateway Pod:kubectl --kubeconfig
CLUSTER_KUBECONFIG exec -itGATEWAY_POD \ --namespace kube-system --ncgCLI_COMMAND Replace
with ancgCLI_COMMAND ncgcli
command.For example:
kubectl --kubeconfig my-kubeconfig exec ncgd-6hkk2 --namespace kube-system \ -- ncgcli --help
The following sections give more examples of ncgcli
commands.
IPsec tunnel failed to come up
Check the status of the
NetworkConnectivityGateway
custom resource:kubectl --kubeconfig
CLUSTER_KUBECONFIG get NetworkConnectivityGateway \ --namespace kube-system --output yamlIn the output, verify that you see
Status: Healthy
. For example:apiVersion: networking.gke.io/v1alpha1 kind: NetworkConnectivityGateway metadata: namespace: kube-system name: default spec: status: CurrNode: worker1-node CreatedTime: 2021-09-07T03:18:15Z LastReportTime: 2021-09-21T23:57:54Z Status: Healthy
Check the status of your
OverlayVPNTunnel
custom resources:kubectl --kubeconfig
CLUSTER_KUBECONFIG get OverlayVPNTunnel \ --namespace kube-system --output yamlCheck the status of your tunnels from inside the gateway Pod:
ncgcli overlay peers show
For more information, see Cloud VPN troubleshooting.
Failed to establish BGP session
Verify that you have used link-local IP addresses, such as 169.254.1.2
and
169.254.2.2
, for the BGP sessions between the Cloud Router and the
gateway Pod in your cluster.
Check the status of your BGP sessions:
kubectl --kubeconfig
CLUSTER_KUBECONFIG get OverlayBGPPeer \ --namespace kube-system --output yamlCheck the status of your BGP sessions from inside the gateway Pod:
ncgcli overlay routes show
For more information about troubleshooting your BGP sessions, see the following:
Traffic forwarding issues even if the tunnel is up
If the traffic forwarding fails, run ncgcli
inside the gateway Pod
to check the status of tunnels, routing, and data plane programming.
Ensure that IPsec tunnel to the peer is established:
ncgcli overlay peers show
Confirm that the overlay routes have been exchanged over BGP and resolved to the appropriate IPsec tunnel:
ncgcli overlay routes show
Display routes programmed in the data plane and the output should include overlay routes:
ncgcli fast-path routes show
Adjust maximum transmission unit
The
maximum transmission unit (MTU)
is the size, in bytes, of the largest packet supported by a network layer
protocol, including both headers and data. If there are traffic issues caused by
underlay network MTU limitations, you can configure the MTU on the
OverlayVPNTunnel
custom resource. The default MTU for the tunnels is 1380
bytes.
To adjust MTU for your tunnels:
Check the status of the tunnels:
kubectl --kubeconfig
CLUSTER_KUBECONFIG get OverlayVPNTunnel \ --namespace kube-system --output yamlThe
status
section should look like the following example:status: IfName: ipsec2 LastTransition: "2022-06-16T07:13:28Z" SLA: DiscoveredMTU: 1300 Jitter(ms): 1 Latency(ms): 26 Loss(%): "0.00"
If the
status.SLA.DiscoveredMTU
value is less than the default of1380
, use the following steps to change theSpec.mtu
value in yourOverlayVPNTunel
custom resources to match.Open your
OverlayVPNTunnel
custom resources for editing with thekubectl edit
command:kubectl --kubeconfig
CLUSTER_KUBECONFIG editOVERLAY_VPN_TUNNEL_NAME --namespace kube-system --output yamlTo avoid packet loss, configure the same MTU for the two
OverlayVPNTunnel
custom resources used by Network Connectivity Gateway.Set the
mtu
value to match theDiscoveredMTU
value from the first step:apiVersion: networking.gke.io/v1alpha1 kind: OverlayVPNTunnel metadata: name: vpc2 namespace: kube-system Spec: mtu: 1300 ikeKey: name: ike-key namespace: kube-system localTunnelIP: 169.254.0.2 peer: publicIP: 34.100.10.10 self: publicIP: 70.32.151.5
To apply your changes, save and close your editor.
For more information about MTU and how it affects traffic performance, see MTU considerations.
See traffic flow and application-level statistics
To see traffic flow and statistics, run ncgcli
inside the gateway Pod.
For an instantaneous view of the flows and traffic applications:
ncgcli forwarding flows top
For aggregate statistics at the flow level:
ncgcli forwarding flows aggregates show
For aggregate statistics at the interface level:
ncgcli interfaces statistics show