Manage flow logs

Use flow logs to save network traffic metadata, including 5-tuples, in Google Distributed Cloud (GDC) air-gapped to the observability stack in the form of queryable logs. Use these logs to:

• Understand traffic flows for a particular service in an organization.
• Understand and identify issues with the network health of a Kubernetes workload.
• Verify Kubernetes network policies.

Flow log is a GDC feature based on the open source Hubble project: https://github.com/cilium/hubble. Configure flow logs using the FlowLog resource from the Networking API.

Before you begin

• You must obtain authorization to manage or view flow logs from the GDC console. To manage flow logs, ask your Organization IAM Admin to grant you the Flow Log Admin (flowlog-admin) role. To only view existing flow logs, ask your Organization IAM Admin to grant you the Flow Log Viewer (flowlog-viewer) role.

  For information about setting role bindings from the GDC console, see Grant access to resources.

• Before querying and viewing flow logs on dashboards, you must obtain access to the monitoring instance. For more information, see Query and view metrics.

Create a flow log

Create a flow log to save network traffic metadata that is filtered by a given set of specifications.

Note: To reduce excess and undesired information in the saved logs, narrow down the filter conditions. This is important when the network traffic volume is large.

Create flow log with protocol and namespace filters

1. Create a file called example-flowlog.yaml with the following contents:

   apiVersion: networking.gdc.goog/v1
   kind: FlowLog
   metadata:
     name: "FLOW_LOG_NAME"
     namespace: "platform"
   spec:
     filters:
       - l4Protocols:
           - tcp
         source:
           namespacePodSelectors:
             - namespace: gpc-system
             - namespace: kube-system
         destination:
           namespacePodSelectors:
             - namespace: kube-system
     lifetime:
       duration: "1h"
   

   Replace FLOW_LOG_NAME with the chosen name for the flow log.

   In this example, the flow log captures all flows for which a TCP connection was started by any pod in the gpc-system or kube-system namespace to any pod in kube-system namespace. It stops recording logs one hour from when the object is created.

2. Create the flow log object:

   kubectl --kubeconfig MANAGEMENT_API_SERVER apply -f example-flowlog.yaml
   

   Replace MANAGEMENT_API_SERVER with the zonal Management API server's kubeconfig path.

   Note: The FlowLog object must be created in the platform namespace.

3. Verify that the FLOW_LOG_NAME value was correctly reconciled, and that the logging process has started by examining the Status field:

   kubectl --kubeconfig MANAGEMENT_API_SERVER describe -n platform FLOW_LOG_NAME
   

   Example status:

   Status:
     Clusters:
       Cluster:  org-1-infra-cluster
       Conditions:
         Last Transition Time:  2024-01-19T01:46:11Z
         Message:
         Observed Generation:   1
         Reason:                ResourcesPropagated
         Status:                True
         Type:                  Propagated
       Name:                    cilium-flowlog-config
       Namespace:               kube-system
       Node:                    org-1-infra
       Cluster:                 user-vm-1-cluster
       Conditions:
         Last Transition Time:  2024-01-19T01:46:11Z
         Message:
         Observed Generation:   1
         Reason:                ResourcesPropagated
         Status:                True
         Type:                  Propagated
       Name:                    cilium-flowlog-config
       Namespace:               kube-system
       Node:                    user-vm-1
       Cluster:                 user-vm-2-cluster
       Conditions:
         Last Transition Time:  2024-01-19T01:46:11Z
         Message:
         Observed Generation:   1
         Reason:                ResourcesPropagated
         Status:                True
         Type:                  Propagated
       Name:                    cilium-flowlog-config
       Namespace:               kube-system
       Node:                    user-vm-2
     Conditions:
       Last Transition Time:  2024-01-18T19:17:53Z
       Message:
       Observed Generation:   1
       Reason:                Active
       Status:                True
       Type:                  Logging
       Last Transition Time:  2024-01-19T01:46:11Z
       Message:
       Observed Generation:   1
       Reason:                ResourcesPropagated
       Status:                True
       Type:                  Propagated
       Last Transition Time:  2024-01-18T19:17:53Z
       Message:
       Observed Generation:   1
       Reason:                Succeeded
       Status:                True
       Type:                  Reconciled
     Start Time:              2024-01-18T19:17:53Z
     End Time:              2024-01-18T20:17:53Z
   

   This example output contains the following conditions:

   • Propagated: the flow log configuration was sent to clusters user-vm-1, user-vm-2, and org-1-infra.
   • Reconciled: the flow log was successfully programmed on clusters user-vm-1, user-vm-2, and org-1-infra.
   • Logging: the flow log is not expired or disabled, has not encountered any error, and may produce logs if the filters match the flows.
   • The Start Time and End Time indicate the job was started at 19:17:53, and expires at 20:17:53.

   For this flow log to be able to successfully produce log entries, all Propagated, Reconciled and Logging conditions must be true.

Modify flow log

To modify an existing flow log, modify the created flow log object definition in the example-flowlog.yaml file and apply it again:

kubectl --kubeconfig MANAGEMENT_API_SERVER apply -f example-flowlog.yaml

View flow logs in monitor dashboard

1. Open the operational log dashboard. For more information, see Query operational logs.

2. Set the identifier drop-down value to network-flow-logs, then click add json parser, and supply the following value to the query: {identifier="network-flow-logs"} | json:

   

3. Click Run query. This shows the saved flow log entries:

   

4. Use the filters to further refine the results. For example, provide the cluster drop-down with a value of user-vm-1 to only look at flows collected from the user-vm-1 cluster:

   

5. Browse flow log entries, expand the entry to look at the details, and add more label filters as needed using the shortcut icon. In this example, we want to further filter flows by its source IP address:

   

6. Click the + shortcut icon to filter flows by a specific metadata field.

   

View results by flow log name

When multiple flow logs are created, an additional label filter can be created to view results from one or more flow logs.

To view results limited to logs produced by FLOW_LOG_NAME, create a label filter with the following values:

• Label set to On. This is case sensitive.
• Operator set to ~=. This uses a regular expression matcher.
• Value set to /FLOW_LOG_NAME/. The FLOW_LOG_NAME value must be surrounded by /.

Filter results to only see logs produced by flowlog1:

Note: To see results from multiple flow logs, join the flow log names using |. For example, "/flowlog1/|/flowlog2/" includes log entries from both flowlog1 and flowlog2.

References

• Documentation on Hubble Dynamic Exporter: https://docs.cilium.io/en/latest/observability/hubble-exporter/#dynamic-exporter-configuration