This page explains how to control egress communication between Pods and
resources outside of the Google Kubernetes Engine (GKE) cluster using fully
qualified domain names (FQDN). The custom resource that you use to configure
FQDNs is the
FQDN network policy is a paid feature, but payment will not be required at this time until FQDN network policy becomes a generally available offering.
Before you begin
Before you start, make sure you have performed the following tasks:
- Enable the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- If you want to use the Google Cloud CLI for this task,
install and then
gcloud CLI. If you previously installed the gcloud CLI, get the latest
version by running
gcloud components update.
Requirements and limitations
FQDNNetworkPolicy resources have the following requirements and limitations:
- You must have a GKE cluster running one of the following
- 1.26.4-gke.500 or later
- 1.27.1-gke.400 or later
- Your cluster must use GKE Dataplane V2.
- You must use one of the DNS providers in your GKE cluster, kube-dns or Cloud DNS. Custom kube-dns or Core DNS deployments are not supported.
- Google Cloud CLI version 434.0.0 or later.
- Windows node pools are not supported.
- Anthos Service Mesh is not supported.
- Network Policy Logging for
FQDNNetworkPolicyresources is not supported.
- If you have hard-coded IP addresses in your application, use the
IPBlockfield of Kubernetes Network Policy instead of a
- Results returned by non-cluster DNS nameservers such as alternate nameservers
resolv.confare not considered valid to be programmed in the allowlist in the GKE data plane.
- The maximum number of IPv4 and IPv6 IP addresses that a
FQDNNetworkPolicycan resolve to is 50.
- You cannot allow traffic to a ClusterIP or Headless Service as an egress
destination in a
FQDNNetworkPolicybecause GKE translates the Service virtual IP address (VIP) to backend Pod IP addresses before evaluating Network Policy rules. Instead, use a Kubernetes label-based Network Policy.
- There is a maximum quota of 100 IP addresses per hostname.
Enable FQDN Network Policy
You can enable FQDN Network Policy on a new or an existing cluster.
Enable FQDN Network Policy in a new cluster
Create your cluster using the
gcloud beta container clusters create CLUSTER_NAME \ --enable-fqdn-network-policy
CLUSTER_NAME with the name of your cluster.
Enable FQDN Network Policy in an existing cluster
For both Autopilot and Standard clusters, update the cluster using the
gcloud beta container clusters update CLUSTER_NAME \ --enable-fqdn-network-policy
CLUSTER_NAMEwith the name of your cluster.
For Standard clusters only, restart the GKE Dataplane V2
kubectl rollout restart ds -n kube-system anetd
Save the following manifest as
apiVersion: networking.gke.io/v1alpha1 kind: FQDNNetworkPolicy metadata: name: allow-out-fqdnnp spec: podSelector: matchLabels: app: curl-client egress: - matches: - pattern: "*.yourdomain.com" - name: "www.google.com" ports: - protocol: "TCP" port: 443
This manifest has the following properties:
name: www.google.com: the fully qualified domain name. IP addresses provided by the nameserver associated with www.google.com are allowed. You must specify either
pattern, or both.
pattern: "*.yourdomain.com": IP addresses provided by nameservers matching this pattern are allowed. You can use the following regular expressions for the pattern key:
^([a-zA-Z0-9*]([-a-zA-Z0-9_*]*[a-zA-Z0-9*])*\.?)*$. Match criteria are additive. You can use multiple
patternfields. You must specify either
pattern, or both.
port: 443: specifies a protocol and port. If a Pod tries to establish a connection to IP addresses using this protocol and port combination, the name resolution works, but the data plane blocks the outbound connection. This field is optional.
Verify that the network policy is selecting your workloads:
kubectl describe fqdnnp
The output is similar to the following:
Name: allow-out-fqdnnp Labels: <none> Annotations: <none> API Version: networking.gke.io/v1alpha1 Kind: FQDNNetworkPolicy Metadata: ... Spec: Egress: Matches: Pattern: *.yourdomain.com Name: www.google.com Ports: Port: 443 Protocol: TCP Pod Selector: Match Labels: App: curl-client Events: <none>
You can delete a
FQDNNetworkPolicy using the
kubectl delete fqdnnp command:
kubectl delete fqdnnp FQDN_POLICY_NAME
FQDN_POLICY_NAME with the name of your
GKE deletes the rules from policy enforcement, but existing connections remain active until they close following the conntrack standard protocol guidelines.
How FQDN network policies work
FQDNNetworkPolicy that selects workloads does not affect the ability
of workloads to make DNS requests. Commands such as
dig work on
any domains without being affected by the policy. However, subsequent requests
to the IP address backing domains not in the allowist would be dropped.
For example, if a
FQDNNetworkPolicy allows egress to
DNS requests for all domains are allowed but traffic sent to an IP address
twitter.com is dropped.
FQDNNetworkPolicy honors the TTL provided by a DNS record. If a Pod attempts
to contact an expired IP address after the TTL of the DNS record has elapsed,
new connections are rejected. Long lived connections whose duration exceeds the
TTL of the DNS record should not experience traffic disruption while conntrack
considers the connection still active.
FQDNNetworkPolicy and NetworkPolicy
When both a
FQDNNetworkPolicy and a
NetworkPolicy apply to the same Pod,
meaning the Pod's labels match what is configured in the policies, egress
traffic is allowed as long as it matches one of the policies. There is no
hierarchy between egress NetworkPolicies specifying IP addresses or
label-selectors and FQDNNetworkPolicies.
protocol: ALL causes policy to be ignored
If you create a
FQDNNetworkPolicy which specifies
protocol: ALL in the
ports section, GKE does not enforce the policy. This issue
occurs because of an issue with parsing the policy. Specifying
does not cause this issue.
As a workaround, if you do not specify a
protocol in the
ports entry, the
rule matches all protocols by default. Removing the
protocol: ALL bypasses the
parsing issue and GKE enforces the
If the FQDN object in the FQDN Network Policy includes a domain that has CNAMEs in the DNS record, you must configure your FQDN Network Policy with all domain names that your Pod can query directly, including all potential aliases, in order to ensure a reliable FQDN Network Policy behavior.
If your Pod queries
example.com is what you should write in
the rule. Even if you get back a chain of aliases from your upstream DNS servers
220.127.116.11), the FQDN Network Policy
will still allow your traffic through.
- Read the Kubernetes documentation about network policies
- Use security insights to explore other ways to harden your infrastructure