About VPC Flow Logs records

This page describes the VPC Flow Logs record format, including which base and metadata fields are available. It also explains how you can use log filtering so that only logs that match certain criteria are generated.

Record format

Log records contain base fields, which are the core fields of every log record, and metadata fields that add additional information. Metadata fields may be omitted to save storage costs.

Some log fields are in a multi-field format, with more than one piece of data in a given field. For example, the connection field is of the IpConnection format, which contains the source and destination IP address and port, plus the protocol, in a single field. These multi-field fields are described below the record format table.

The values for metadata fields aren't based on the data plane path; they are approximations and might be missing or incorrect. Unlike metadata fields, the values for base fields are taken directly from packet headers.

Field	Field format	Field type: Base or optional metadata
connection	IpConnection 5-tuple describing the flow.	Base
reporter	string The side which reported the flow. Can be `SRC`, `DEST`, `SRC_GATEWAY`, or `DEST_GATEWAY`.	Base
rtt_msec	int64 Latency as measured during the time interval. This field is available only for TCP traffic reported from VMs. The measured latency is the time elapsed between sending a SEQ and receiving a corresponding ACK. The latency result is the sum of the network RTT and any time consumed by the application.	Base
bytes_sent	int64 Amount of bytes sent from the source to the destination.	Base
packets_sent	int64 Number of packets sent from the source to the destination.	Base
start_time	string Timestamp (RFC 3339 date string format) of the first observed packet during the aggregated time interval.	Base
end_time	string Timestamp (RFC 3339 date string format) of the last observed packet during the aggregated time interval.	Base
Source and destination metadata fields
src_gateway	GatewayDetails If the source of the flow is an on-premises or other cloud endpoint that is connected to Google Cloud through a gateway such as a VLAN attachment for Cloud Interconnect or a Cloud VPN tunnel and either of the following conditions is met, this field is populated with gateway details: The source gateway is the reporter of the flow. The destination of the flow is the reporter, and the VPC network that the source gateway is connected to is in the same Google Cloud project as the project in which VPC Flow Logs reports logs for the destination of the flow.	Metadata
dest_gateway	GatewayDetails If the destination of the flow is an on-premises or other cloud endpoint that is connected to Google Cloud through a gateway such as a VLAN attachment for Cloud Interconnect or a Cloud VPN tunnel and either of the following conditions is met, this field is populated with gateway details: The destination gateway is the reporter of the flow. The source of the flow is the reporter, and the VPC network that the destination gateway is connected to is in the same Google Cloud project as the project in which VPC Flow Logs reports logs for the source of the flow.	Metadata
src_gke_details	GkeDetails If the source of the flow is a Google Kubernetes Engine (GKE) endpoint, this field is populated with GKE endpoint details.	Metadata
dest_gke_details	GkeDetails If the destination of the flow is a GKE endpoint, this field is populated with GKE endpoint details.	Metadata
src_google_service	GoogleServiceDetails If the source of the flow is a Google API, this field is populated with available Google API metadata.	Metadata
dest_google_service	GoogleServiceDetails If the destination of the flow is a Google API, this field is populated with available Google API metadata.	Metadata
src_instance	InstanceDetails If the source of the flow is a VM located in a VPC network and either of the following conditions is met, this field is populated with VM instance details: The source VM is the reporter of the flow. The destination of the flow is the reporter, and the source VM, or the VPC network that the source VM is attached to, is in the same Google Cloud project as the project in which VPC Flow Logs reports logs for the destination of the flow.	Metadata
dest_instance	InstanceDetails If the destination of the flow is a VM located in a VPC network and either of the following conditions is met, this field is populated with VM instance details. The destination VM is the reporter of the flow. The source of the flow is the reporter, and the destination VM, or the VPC network that the destination VM is attached to, is in the same Google Cloud project as the project in which VPC Flow Logs reports logs for the source of the flow.	Metadata
src_location	GeographicDetails If the source of the flow is a public IP address outside of the VPC network, this field is populated with available location metadata.	Metadata
dest_location	GeographicDetails If the destination of the flow is a public IP address outside of the VPC network, this field is populated with available location metadata.	Metadata
src_vpc	VpcDetails If the source of the flow is a VM located in a VPC network and either of the following conditions is met, this field is populated with VPC network details: The source VM is the reporter of the flow. The destination of the flow is the reporter, and the source VM, or the VPC network that the source VM is attached to, is in the same Google Cloud project as the project in which VPC Flow Logs reports logs for the destination of the flow.	Metadata
dest_vpc	VpcDetails If the destination of the flow is a VM located in a VPC network and either of the following conditions is met, this field is populated with VPC network details: The destination VM is the reporter of the flow. The source of the flow is the reporter, and the destination VM, or the VPC network that the destination VM is attached to, is in the same Google Cloud project as the project in which VPC Flow Logs reports logs for the source of the flow.	Metadata
Other metadata fields
internet_routing_details	InternetRoutingDetails If the flow is between Google Cloud and the internet, this field is populated with routing details. Available only for egress flows.	Metadata
load_balancing	LoadBalancingDetails If the flow passes through a load balancer in one of the following configurations, this field is populated with Cloud Load Balancing details: The reporter of the flow is the client of the load balancer, and the load balancer type is `APPLICATION_LOAD_BALANCER`, `PROXY_NETWORK_LOAD_BALANCER`, `PASSTHROUGH_NETWORK_LOAD_BALANCER`, or `PROTOCOL_FORWARDING`. The reporter of the flow is the backend of the load balancer, and the load balancer type is `PASSTHROUGH_NETWORK_LOAD_BALANCER` or `PROTOCOL_FORWARDING`.	Metadata
network_service	NetworkServiceDetails If the Differentiated Services Code Point (DSCP) header is set, this field is populated with network service details.	Metadata
psc	PrivateServiceConnectDetails If the flow passes through Private Service Connect in either of the following configurations, this field is populated with Private Service Connect details: The reporter of the Private Service Connect traffic is a consumer and is using a Private Service Connect endpoint that targets a published service or global Google APIs. The reporter of the Private Service Connect traffic is a producer and is using an internal passthrough Network Load Balancer or internal protocol forwarding.	Metadata

IpConnection field format

Field	Type	Description
protocol	int32	The IANA protocol number
src_ip	string	Source IP address
dest_ip	string	Destination IP address
src_port	int32	Source port
dest_port	int32	Destination port

GatewayDetails field format

Field	Type	Description
project_id	string	Google Cloud project ID of the gateway
location	string	Region of the gateway
name	string	Name of the gateway
type	string	Type of the gateway. Can be `INTERCONNECT_ATTACHMENT` or `VPN_TUNNEL`.
vpc	VpcDetails	VPC network details of the gateway

GkeDetails field format

Field	Type	Description
cluster	ClusterDetails	GKE cluster metadata
pod	PodDetails	GKE Pod metadata, populated when the source or destination of the traffic is a Pod
service	ServiceDetails	GKE Service metadata, populated in Service endpoints only. The record contains up to two Services. If there are more than two relevant Services, this field contains a single Service with a special `MANY_SERVICES` marker.

ClusterDetails field format

Field	Type	Description
cluster_location	string	Location of the cluster. This can be a zone or a region depending if the cluster is zonal or regional.
cluster_name	string	GKE cluster name.

PodDetails field format

Field	Type	Description
pod_name	string	Name of the Pod
pod_namespace	string	Namespace of the Pod
pod_workload	WorkloadDetails	Metadata about the top-level workload resource that controls the Pod

WorkloadDetails field format

Field	Type	Description
workload_name	string	Name of the top-level workload controller
workload_type	string	Type of the top-level workload controller. Can be `DEPLOYMENT`, `REPLICA_SET`, `STATEFUL_SET`, `DAEMON_SET`, `JOB`, `CRON_JOB`, or `REPLICATION_CONTROLLER`.

ServiceDetails field format

Field	Type	Description
service_name	string	Name of the Service. If there are more than two relevant Services, the field is set to a special `MANY_SERVICES` marker.
service_namespace	string	Namespace of the Service

Example:

If there are two services, the Service field looks like this:

service: [
 0: {
  service_name: "my-lb-service"
  service_namespace: "default"
 }
 1: {
  service_name: "my-lb-service2"
  service_namespace: "default"
 }
]

If there are more than two services, the Service field looks like this:

service: [
 0: {
  service_name: "MANY_SERVICES"
 }
]

GoogleServiceDetails field format

Field	Type	Description
type	string	This field is set to `GOOGLE_API`.

InstanceDetails field format

Field	Type	Description
project_id	string	ID of the Google Cloud project that contains the VM resource
region	string	Region of the VM
vm_name	string	Instance name of the VM
zone	string	Zone of the VM

GeographicDetails field format

Field	Type	Description
asn	int32	The ASN of the external network to which this endpoint belongs.
city	string	City for external endpoints
continent	string	Continent for external endpoints
country	string	Country for external endpoints, represented as ISO 3166-1 Alpha-3 country codes
region	string	Region for external endpoints

VpcDetails field format

Field	Type	Description
project_id	string	ID of the Google Cloud project containing the VPC. In a Shared VPC configuration, `project_id` is the ID of the host project.
subnetwork_name	string	Name of the subnet, if applicable
subnetwork_region	string	Region of the subnet, if applicable
vpc_name	string	Name of the network

InternetRoutingDetails field format

Field	Type	Description
egress_as_path	AsPath	List of relevant AS paths. If there are multiple AS paths available to the flow, the field might contain more than one AS path.

AsPath field format

Field	Type	Description
as_details	AsDetails	List of AS details for all systems in the AS path. The list starts from the first AS that is external to Google Cloud's network and ends with the AS to which the remote IP address belongs.

AsDetails field format

Field	Type	Description
asn	uint32	The autonomous system number (ASN) of the AS

LoadBalancingDetails field format

Field	Type	Description
forwarding_rule_project_id	string	Google Cloud project ID of the forwarding rule
reporter	string	Cloud Load Balancing reporter. Can be either `CLIENT` or `BACKEND`. If the reporter of the flow is the client of the load balancer, this field is set to `CLIENT`. If the reporter of the flow is the backend of the load balancer, this field is set to `BACKEND`.
type	string	Load balancer type. Can be `APPLICATION_LOAD_BALANCER`, `PROXY_NETWORK_LOAD_BALANCER`, `PASSTHROUGH_NETWORK_LOAD_BALANCER`, or `PROTOCOL_FORWARDING`.
scheme	string	Load balancer scheme. Can be `EXTERNAL_MANAGED`, `INTERNAL_MANAGED`, `EXTERNAL`, `INTERNAL`, or `INTERNAL_SELF_MANAGED`.
url_map_name	string	Name of the URL map. Populated if the type of the load balancer is `APPLICATION_LOAD_BALANCER`.
forwarding_rule_name	string	Name of the forwarding rule
backend_service_name	string	Name of the backend service. Populated if the reporter is `BACKEND` and load balancer type is `PASSTHROUGH_NETWORK_LOAD_BALANCER`. If the backend group type is `TARGET_POOL`, this field isn't populated.
backend_group_name	string	Name of the backend group. Populated if the reporter is `BACKEND` and load balancer type is `PASSTHROUGH_NETWORK_LOAD_BALANCER`.
backend_group_type	string	Type of the backend group. Can be `INSTANCE_GROUP`, `NETWORK_ENDPOINT_GROUP`, or `TARGET_POOL`. Populated if the reporter is `BACKEND` and the load balancer type is `PASSTHROUGH_NETWORK_LOAD_BALANCER`.
backend_group_location	string	Location of the backend group. Can be a zone or a region depending on whether the scope of the backend group is zonal or regional. Populated if the reporter is `BACKEND` and the load balancer type is `PASSTHROUGH_NETWORK_LOAD_BALANCER`. If the backend group type is `TARGET_POOL`, this field isn't populated.
vpc	VpcDetails	VPC network details of the load balancer

NetworkServiceDetails field format

Field	Type	Description
dscp	int32	If the Differentiated Services field is present in packet headers, this field is populated with the DSCP value.

PrivateServiceConnectDetails field format

Field	Type	Description
reporter	string	Private Service Connect reporter. Can be either `CONSUMER` or `PRODUCER`.
psc_endpoint	PrivateServiceConnectEndpointDetails	Endpoint details. Populated if the reporter is `CONSUMER`.
psc_attachment	PrivateServiceConnectAttachmentDetails	Service attachment details. Populated if the traffic flow includes a Private Service Connect producer.

PrivateServiceConnectEndpointDetails field format

Field	Type	Description
project_id	string	Google Cloud project ID of the Private Service Connect endpoint
region	string	Region of the endpoint. Not populated if the target service type is `GLOBAL_GOOGLE_APIS`.
psc_connection_id	string	Private Service Connect connection ID
target_service_type	string	Target service type. Can be either `GLOBAL_GOOGLE_APIS` or `PUBLISHED_SERVICE`.
vpc	VpcDetails	VPC network details of the Private Service Connect endpoint

PrivateServiceConnectAttachmentDetails field format

Field	Type	Description
project_id	string	Google Cloud project ID of the service attachment
region	string	Region of the service attachment
vpc	VpcDetails	VPC network details of the service attachment

Metadata annotations

Log records contain base fields and metadata fields. The Record format section lists which fields are type metadata and which are type base. All base fields are always included. You can customize which metadata fields you keep.

If you select all metadata, all metadata fields in the VPC Flow Logs record format are included in the flow logs. When new metadata fields are added to the record format, the flow logs automatically include the new fields.
If you select no metadata, this omits all metadata fields.
If you select custom metadata, you can specify the metadata fields that you want to include by the parent field, such as src_vpc, or by their full names, such as src_vpc.project_id

When new metadata fields are added to the record format, they're excluded from the flow logs unless they're within a parent field that you have specified to include.
- If you specify custom metadata using parent fields, when new metadata fields are added to the record format within that parent field, the flow logs will automatically include the new fields.
- If you specify custom metadata using the full name of the field, new metadata fields that are added to the parent field are excluded from the flow logs.

For information about customizing metadata fields, see the Google Cloud CLI or API instructions for enabling VPC flow logging when you create a subnet.

GKE metadata annotations

Flows that have an endpoint in a GKE Cluster can be annotated with GKE metadata annotations, which can include details of the Cluster, Pod, and Service of the endpoint.

GKE Service annotations

Traffic sent to a ClusterIP, NodePort, or LoadBalancer can receive Service annotations. If sent to a NodePort or LoadBalancer, the flow receives the Service annotation on both hops of the connection.

Traffic sent directly to a Pod's Service port is annotated with a Service annotation on the destination endpoint.

Traffic sent to a Pod's Service port where the Pod is backing more than one Service on the same Service port is annotated with multiple Services on the destination endpoint. This is limited to two Services. If there are more than that, the endpoint will be annotated with a special MANY_SERVICES marker.

Pod annotations on internet traffic

Traffic between a Pod and the internet doesn't receive Pod annotations by default. VPC Flow Logs can't add Pod annotations because, for packets to the internet, the masquerade agent translates the Pod IP address to the node IP address before VPC Flow Logs sees the packet.

Because of the masquerade, Pod annotations are only visible if the destinations are within either the default non-masquerade destinations or in a custom nonMasqueradeCIDRs list. If you include internet destinations in a custom nonMasqueradeCIDRs list, you need to provide a way for the internal Pod IP addresses to be translated before they are delivered to the internet. For both private and non-private clusters, you can use Cloud NAT. See GKE interaction for more details.

Log filtering

When you enable VPC Flow Logs, you can set a filter based on both base and metadata fields that only preserves logs that match the filter. All other logs are discarded before being written to Logging, which saves you money and reduces the time needed to find the information you are looking for.

You can filter on any subset of fields listed in Record format, except for the following fields:

rtt_msec
bytes_sent
packets_sent
start_time
end_time

VPC Flow Logs filtering uses CEL, an embedded expression language for attribute-based logic expressions. Filter expressions for VPC Flow Logs have a limit of 2,048 characters. For more information, see Supported CEL logic operators.

For more information about CEL, see the CEL introduction and the language definition. The generation filter feature supports a limited subset of CEL syntax.

For information about creating a subnet that uses log filtering, see the gcloud CLI or API instructions for Enabling VPC Flow Logs when you create a subnet.

For information about configuring log filtering, see the gcloud CLI or API instructions for Updating VPC Flow Logs parameters.

Example 1: Limit logs collection to a specific VM named my-vm. In this case, only logs where the src_instance field as reported by the source of the traffic is my-vm or the dst_instance field as reported by the destination of the traffic is my-vm are recorded.

gcloud compute networks subnets update my-subnet \
    --logging-filter-expr="(src_instance.vm_name == 'my-vm' && reporter=='SRC') || (dest_instance.vm_name == 'my-vm' && reporter=='DEST')"

Example 2: Limit logs collection to packets whose source IP addresses are in the 10.0.0.0/8 subnet.

gcloud compute networks subnets update my-subnet \
    --logging-filter-expr="inIpRange(connection.src_ip, '10.0.0.0/8')"

Example 3: Limit logs collection to traffic that is external to a VPC.

gcloud compute networks subnets update my-subnet \
    --logging-filter-expr '!(has(src_vpc.vpc_name) && has(dest_vpc.vpc_name))'

Supported CEL logic operators

Expression	Supported types	Description
true, false	Boolean	Boolean constants
x == y x != y	Boolean, Int, String	Comparison operators Example: connection.protocol == 6
x && y x \|\| y	Boolean	Boolean logic operators Example: connection.protocol == 6 && src_instance.vm_name == "vm_1"
!x	Boolean	Negation
1, 2.0, 0, ...	Int	Constant numeric literals
x + y	String	String concatenation
"foo", 'foo', ...	String	Constant string literal
x.lower()	String	Returns the lowercase value of the string
x.upper()	String	Returns the uppercase value of the string
x.contains(y)	String	Returns true if the string contains the specified substring
x.startsWith(y)	String	Returns true if the string begins with the specified substring
x.endsWith(y)	String	Returns true if the string ends with the specified substring
inIpRange(X, Y)	String	Returns true if X is an IP and Y is an IP range that contains X Example: inIpRange("1.2.3.1", "1.2.3.0/24")
x.containsFieldValue(y)	x: list y: map(string, string)	Returns true if the list contains an object with fields that match the specified key-value pairs Example: dest_gke_details.service.containsFieldValue({'service_name': 'service1', 'service_namespace': 'namespace1'})
has(x)	String	Returns true if the field is present.