About VPC Flow Logs records

This page describes the VPC Flow Logs record format, including which base and metadata fields are available. It also explains how you can use log filtering so that only logs that match certain criteria are generated.

Record format

Log records contain base fields, which are the core fields of every log record, and metadata fields that add additional information. Metadata fields may be omitted to save storage costs.

Some log fields are in a multi-field format, with more than one piece of data in a given field. For example, the connection field is of the IpConnection format, which contains the source and destination IP address and port, plus the protocol, in a single field. These multi-field fields are described below the record format table.

Metadata fields have the following limitations:

  • The values for metadata fields are not based on the data plane path; they are approximations and might be missing or incorrect. Unlike metadata fields, the values for base fields are taken directly from packet headers.
  • For the internet_routing_details field, the autonomous system (AS) path might be missing in some cases. For example, when packets are routed within a VPC, AS path information is not included.
Field Field format Field type: Base or optional metadata
connection IpConnection
5-Tuple describing this connection 		Base
reporter string
The side which reported the flow. Can be either SRC or DEST. 		Base
rtt_msec int64
Latency as measured during the time interval, for TCP flows only. The measured latency is the time elapsed between sending a SEQ and receiving a corresponding ACK. The latency result is the sum of the network RTT and any time consumed by the application. 		Base
bytes_sent int64
Amount of bytes sent from the source to the destination 		Base
packets_sent int64
Number of packets sent from the source to the destination 		Base
start_time string
Timestamp (RFC 3339 date string format) of the first observed packet during the aggregated time interval. 		Base
end_time string
Timestamp (RFC 3339 date string format) of the last observed packet during the aggregated time interval 		Base
internet_routing_details InternetRoutingDetails
If the connection is between Google Cloud and the internet, this field is populated with routing details. Available only for egress flows. 		Metadata
src_gke_details GkeDetails
GKE metadata for source endpoints. Only available if the endpoint is GKE. 		Metadata
dest_gke_details GkeDetails
GKE metadata for destination endpoints. Only available if the endpoint is GKE. 		Metadata
src_instance InstanceDetails
If the source of the connection was a VM located on the same VPC, this field is populated with VM instance details. In a Shared VPC configuration, project_id corresponds to the project that owns the instance, usually the service project. 		Metadata
dest_instance InstanceDetails
If the destination of the connection was a VM located on the same VPC, this field is populated with VM instance details. In a Shared VPC configuration, project_id corresponds to the project that owns the instance, usually the service project. 		Metadata
src_location GeographicDetails
If the source of the connection was external to the VPC, this field is populated with available location metadata. 		Metadata
dest_location GeographicDetails
If the destination of the connection was external to the VPC, this field is populated with available location metadata. 		Metadata
src_vpc VpcDetails
If the source of the connection was a VM located on the same VPC, this field is populated with VPC network details. In a Shared VPC configuration, project_id corresponds to that of the host project. 		Metadata
dest_vpc VpcDetails
If the destination of the connection was a VM located on the same VPC, this field is populated with VPC network details. In a Shared VPC configuration, project_id corresponds to that of the host project. 		Metadata

IpConnection field format

Field Type Description
protocol int32 The IANA protocol number
src_ip string Source IP address
dest_ip string Destination IP address
src_port int32 Source port
dest_port int32 Destination port

InternetRoutingDetails field format

Field Type Description
egress_as_path AsPath List of relevant AS paths. If there are multiple AS paths available to the flow, the field might contain more than one AS path.

AsPath field format

Field Type Description
as_details AsDetails List of AS details for all systems in the AS path. The list starts from the first AS that is external to Google Cloud's network and ends with the AS to which the remote IP address belongs.

AsDetails field format

Field Type Description
asn uint32 The autonomous system number (ASN) of the AS.

GkeDetails field format

Field Type Description
cluster ClusterDetails GKE cluster metadata
pod PodDetails GKE Pod metadata, populated when the source or destination of the traffic is a Pod
service ServiceDetails GKE Service metadata, populated in Service endpoints only. The record contains up to two Services. If there are more than two relevant Services, this field contains a single Service with a special MANY_SERVICES marker.

ClusterDetails field format

Field Type Description
cluster_location string Location of the cluster. This can be a zone or a region depending if the cluster is zonal or regional.
cluster_name string GKE cluster name.

PodDetails field format

Field Type Description
pod_name string Name of the Pod
pod_namespace string Namespace of the Pod

ServiceDetails field format

Field Type Description
service_name string Name of the Service. If there are more than two relevant Services, the field is set to a special MANY_SERVICES marker.
service_namespace string Namespace of the Service

Example:

If there are two services, the Service field looks like this:

service: [
 0: {
  service_name: "my-lb-service"
  service_namespace: "default"
 }
 1: {
  service_name: "my-lb-service2"
  service_namespace: "default"
 }
]

If there are more than two services, the Service field looks like this:

service: [
 0: {
  service_name: "MANY_SERVICES"
 }
]

InstanceDetails field format

Field Type Description
project_id string ID of the project containing the VM
region string Region of the VM
vm_name string Instance name of the VM
zone string Zone of the VM

GeographicDetails field format

Field Type Description
asn int32 The ASN of the external network to which this endpoint belongs.
city string City for external endpoints
continent string Continent for external endpoints
country string Country for external endpoints, represented as ISO 3166-1 Alpha-3 country codes
region string Region for external endpoints

VpcDetails field format

Field Type Description
project_id string ID of the project containing the VPC
subnetwork_name string Subnetwork on which the VM is operating
vpc_name string VPC on which the VM is operating

Metadata annotations

Log records contain base fields and metadata fields. The Record format section lists which fields are type metadata and which are type base. All base fields are always included. You can customize which metadata fields you keep.

  • If you select all metadata, all metadata fields in the VPC Flow Logs record format are included in the flow logs. When new metadata fields are added to the record format, the flow logs automatically include the new fields.

  • If you select no metadata, this omits all metadata fields.

  • If you select custom metadata, you can specify the metadata fields that you want to include by the parent field, such as src_vpc, or by their full names, such as src_vpc.project_id

    When new metadata fields are added to the record format, the flow logs won't include these fields, unless they are a new field within a parent field that you have specified to include.

    • If you specify custom metadata using parent fields, when new metadata fields are added to the record format within that parent field, the flow logs will automatically include the new fields.

    • If you specify custom metadata using the full name of the field, when new metadata fields are added to the parent field, the flow logs won't include the new fields.

For information on customizing metadata fields, see the Google Cloud CLI or API instructions for enabling VPC flow logging when you create a subnet.

GKE metadata annotations

Flows that have an endpoint in a GKE Cluster can be annotated with GKE metadata annotations, which can include details of the Cluster, Pod, and Service of the endpoint.

GKE Service annotations

Traffic sent to a ClusterIP, NodePort, or LoadBalancer can receive Service annotations. If sent to a NodePort or LoadBalancer, the flow receives the Service annotation on both hops of the connection.

Traffic sent directly to a Pod's Service port is annotated with a Service annotation on the destination endpoint.

Traffic sent to a Pod's Service port where the Pod is backing more than one Service on the same Service port is annotated with multiple Services on the destination endpoint. This is limited to two Services. If there are more than that, the endpoint will be annotated with a special MANY_SERVICES marker.

Pod annotations on internet traffic

Traffic between a Pod and the internet does not receive Pod annotations by default. For packets to the internet, the masquerade agent translates the Pod IP address to the node IP address before VPC Flow Logs sees the packet, so VPC Flow Logs doesn't know anything about the Pod and cannot add Pod annotations.

Because of the masquerade, Pod annotations are only visible if the destinations are within either the default non-masquerade destinations or in a custom nonMasqueradeCIDRs list. If you include internet destinations in a custom nonMasqueradeCIDRs list, you need to provide a way for the internal Pod IP addresses to be translated before they are delivered to the internet. For both private and non-private clusters, you can use Cloud NAT. See GKE interaction for more details.

Log filtering

When you enable VPC Flow Logs, you can set a filter based on both base and metadata fields that only preserves logs that match the filter. All other logs are discarded before being written to Logging, which saves you money and reduces the time needed to find the information you are looking for.

You can filter on any subset of fields listed in Record format, except for the following fields:

  • rtt_msec
  • bytes_sent
  • packets_sent
  • start_time
  • end_time

VPC Flow Logs filtering uses CEL, an embedded expression language for attribute-based logic expressions. Filter expressions for VPC Flow Logs have a limit of 2,048 characters. For more information, see Supported CEL logic operators.

For more information about CEL, see the CEL introduction and the language definition. The generation filter feature supports a limited subset of CEL syntax.

For information about creating a subnet that uses log filtering, see the gcloud CLI or API instructions for Enabling VPC Flow Logs when you create a subnet.

For information about configuring log filtering, see the gcloud CLI or API instructions for Updating VPC Flow Logs parameters.

Example 1: Limit logs collection to a specific VM named my-vm. In this case, only logs where the src_instance field as reported by the source of the traffic is my-vm or the dst_instance field as reported by the destination of the traffic is my-vm are recorded.

gcloud compute networks subnets update my-subnet \
    --logging-filter-expr="(src_instance.vm_name == 'my-vm' && reporter=='SRC') || (dest_instance.vm_name == 'my-vm' && reporter=='DEST')"

Example 2: Limit logs collection to packets whose source IP addresses are in the 10.0.0.0/8 subnet.

gcloud compute networks subnets update my-subnet \
    --logging-filter-expr="inIpRange(connection.src_ip, '10.0.0.0/8')"

Example 3: Limit logs collection to traffic that is external to a VPC.

gcloud compute networks subnets update my-subnet \
    --logging-filter-expr '!(has(src_vpc.vpc_name) && has(dest_vpc.vpc_name))'

Supported CEL logic operators

Expression Supported types Description
true, false Boolean Boolean constants

x == y

x != y

 Boolean, Int, String

Comparison operators

Example: connection.protocol == 6

x && y

x || y

 Boolean

Boolean logic operators

Example: connection.protocol == 6 && src_instance.vm_name == "vm_1"
!x Boolean Negation
1, 2.0, 0, ... Int Constant numeric literals
x + y String String concatenation
"foo", 'foo', ... String Constant string literal
x.lower() String Returns the lowercase value of the string
x.upper() String Returns the uppercase value of the string
x.contains(y) String Returns true if the string contains the specified substring
x.startsWith(y) String Returns true if the string begins with the specified substring
x.endsWith(y) String Returns true if the string ends with the specified substring
inIpRange(X, Y) String

Returns true if X is an IP and Y is an IP range that contains X

Example: inIpRange("1.2.3.1", "1.2.3.0/24")
x.containsFieldValue(y) x: list
y: map(string, string)

Returns true if the list contains an object with fields that match the specified key-value pairs

Example: dest_gke_details.service.containsFieldValue({'service_name': 'service1', 'service_namespace': 'namespace1'})
has(x) String

Returns true if the field is present.

What's next