About VPC Flow Logs records

This page describes the VPC Flow Logs record format, including which base and metadata fields are available. It also explains how you can use log filtering so that only logs that match certain criteria are generated.

Record format

Log records contain base fields, which are the core fields of every log record, and metadata fields that add additional information. Metadata fields may be omitted to save storage costs.

Some log fields are in a multi-field format, with more than one piece of data in a given field. For example, the connection field is of the IpConnection format, which contains the source and destination IP address and port, plus the protocol, in a single field. These multi-field fields are described below the record format table.

The values for metadata fields aren't based on the data plane path; they are approximations and might be missing or incorrect. Unlike metadata fields, the values for base fields are taken directly from packet headers.

Field Field format Field type: Base or optional metadata
connection IpConnection
5-tuple describing the flow.
Base
reporter string
The side which reported the flow. Can be SRC, DEST, SRC_GATEWAY, or DEST_GATEWAY.
Base
rtt_msec int64
Latency as measured during the time interval. This field is available only for TCP traffic reported from VMs. The measured latency is the time elapsed between sending a SEQ and receiving a corresponding ACK. The latency result is the sum of the network RTT and any time consumed by the application.
Base
bytes_sent int64
Amount of bytes sent from the source to the destination.
Base
packets_sent int64
Number of packets sent from the source to the destination.
Base
start_time string
Timestamp (RFC 3339 date string format) of the first observed packet during the aggregated time interval.
Base
end_time string
Timestamp (RFC 3339 date string format) of the last observed packet during the aggregated time interval.
Base
Source and destination metadata fields
src_gateway GatewayDetails
If the source of the flow is an on-premises or other cloud endpoint that is connected to Google Cloud through a gateway such as a VLAN attachment for Cloud Interconnect or a Cloud VPN tunnel and either of the following conditions is met, this field is populated with gateway details:
  • The source gateway is the reporter of the flow.
  • The destination of the flow is the reporter, and the VPC network that the source gateway is connected to is in the same Google Cloud project as the project in which VPC Flow Logs reports logs for the destination of the flow.
Metadata
dest_gateway GatewayDetails
If the destination of the flow is an on-premises or other cloud endpoint that is connected to Google Cloud through a gateway such as a VLAN attachment for Cloud Interconnect or a Cloud VPN tunnel and either of the following conditions is met, this field is populated with gateway details:
  • The destination gateway is the reporter of the flow.
  • The source of the flow is the reporter, and the VPC network that the destination gateway is connected to is in the same Google Cloud project as the project in which VPC Flow Logs reports logs for the source of the flow.
Metadata
src_gke_details GkeDetails
If the source of the flow is a Google Kubernetes Engine (GKE) endpoint, this field is populated with GKE endpoint details.
Metadata
dest_gke_details GkeDetails
If the destination of the flow is a GKE endpoint, this field is populated with GKE endpoint details.
Metadata
src_google_service GoogleServiceDetails
If the source of the flow is a Google API, this field is populated with available Google API metadata.
Metadata
dest_google_service GoogleServiceDetails
If the destination of the flow is a Google API, this field is populated with available Google API metadata.
Metadata
src_instance InstanceDetails
If the source of the flow is a VM located in a VPC network and either of the following conditions is met, this field is populated with VM instance details:
  • The source VM is the reporter of the flow.
  • The destination of the flow is the reporter, and the source VM, or the VPC network that the source VM is attached to, is in the same Google Cloud project as the project in which VPC Flow Logs reports logs for the destination of the flow.
Metadata
dest_instance InstanceDetails
If the destination of the flow is a VM located in a VPC network and either of the following conditions is met, this field is populated with VM instance details.
  • The destination VM is the reporter of the flow.
  • The source of the flow is the reporter, and the destination VM, or the VPC network that the destination VM is attached to, is in the same Google Cloud project as the project in which VPC Flow Logs reports logs for the source of the flow.
Metadata
src_location GeographicDetails
If the source of the flow is a public IP address outside of the VPC network, this field is populated with available location metadata.
Metadata
dest_location GeographicDetails
If the destination of the flow is a public IP address outside of the VPC network, this field is populated with available location metadata.
Metadata
src_vpc VpcDetails
If the source of the flow is a VM located in a VPC network and either of the following conditions is met, this field is populated with VPC network details:
  • The source VM is the reporter of the flow.
  • The destination of the flow is the reporter, and the source VM, or the VPC network that the source VM is attached to, is in the same Google Cloud project as the project in which VPC Flow Logs reports logs for the destination of the flow.
Metadata
dest_vpc VpcDetails
If the destination of the flow is a VM located in a VPC network and either of the following conditions is met, this field is populated with VPC network details:
  • The destination VM is the reporter of the flow.
  • The source of the flow is the reporter, and the destination VM, or the VPC network that the destination VM is attached to, is in the same Google Cloud project as the project in which VPC Flow Logs reports logs for the source of the flow.
Metadata
Other metadata fields
internet_routing_details InternetRoutingDetails
If the flow is between Google Cloud and the internet, this field is populated with routing details. Available only for egress flows.
Metadata
load_balancing LoadBalancingDetails
If the flow passes through a load balancer in one of the following configurations, this field is populated with Cloud Load Balancing details:
  • The reporter of the flow is the client of the load balancer, and the load balancer type is APPLICATION_LOAD_BALANCER, PROXY_NETWORK_LOAD_BALANCER, PASSTHROUGH_NETWORK_LOAD_BALANCER, or PROTOCOL_FORWARDING.
  • The reporter of the flow is the backend of the load balancer, and the load balancer type is PASSTHROUGH_NETWORK_LOAD_BALANCER or PROTOCOL_FORWARDING.
Metadata
network_service NetworkServiceDetails
If the Differentiated Services Code Point (DSCP) header is set, this field is populated with network service details.
Metadata
psc PrivateServiceConnectDetails
If the flow passes through Private Service Connect in either of the following configurations, this field is populated with Private Service Connect details:
  • The reporter of the Private Service Connect traffic is a consumer and is using a Private Service Connect endpoint that targets a published service or global Google APIs.
  • The reporter of the Private Service Connect traffic is a producer and is using an internal passthrough Network Load Balancer or internal protocol forwarding.
Metadata

IpConnection field format

Field Type Description
protocol int32 The IANA protocol number
src_ip string Source IP address
dest_ip string Destination IP address
src_port int32 Source port
dest_port int32 Destination port

GatewayDetails field format

Field Type Description
project_id string Google Cloud project ID of the gateway
location string Region of the gateway
name string Name of the gateway
type string Type of the gateway. Can be INTERCONNECT_ATTACHMENT or VPN_TUNNEL.
vpc VpcDetails VPC network details of the gateway

GkeDetails field format

Field Type Description
cluster ClusterDetails GKE cluster metadata
pod PodDetails GKE Pod metadata, populated when the source or destination of the traffic is a Pod
service ServiceDetails GKE Service metadata, populated in Service endpoints only. The record contains up to two Services. If there are more than two relevant Services, this field contains a single Service with a special MANY_SERVICES marker.

ClusterDetails field format

Field Type Description
cluster_location string Location of the cluster. This can be a zone or a region depending if the cluster is zonal or regional.
cluster_name string GKE cluster name.

PodDetails field format

Field Type Description
pod_name string Name of the Pod
pod_namespace string Namespace of the Pod
pod_workload WorkloadDetails Metadata about the top-level workload resource that controls the Pod
WorkloadDetails field format
Field Type Description
workload_name string Name of the top-level workload controller
workload_type string Type of the top-level workload controller. Can be DEPLOYMENT, REPLICA_SET, STATEFUL_SET, DAEMON_SET, JOB, CRON_JOB, or REPLICATION_CONTROLLER.

ServiceDetails field format

Field Type Description
service_name string Name of the Service. If there are more than two relevant Services, the field is set to a special MANY_SERVICES marker.
service_namespace string Namespace of the Service

Example:

If there are two services, the Service field looks like this:

service: [
 0: {
  service_name: "my-lb-service"
  service_namespace: "default"
 }
 1: {
  service_name: "my-lb-service2"
  service_namespace: "default"
 }
]

If there are more than two services, the Service field looks like this:

service: [
 0: {
  service_name: "MANY_SERVICES"
 }
]

GoogleServiceDetails field format

Field Type Description
type string This field is set to GOOGLE_API.

InstanceDetails field format

Field Type Description
project_id string ID of the Google Cloud project that contains the VM resource
region string Region of the VM
vm_name string Instance name of the VM
zone string Zone of the VM

GeographicDetails field format

Field Type Description
asn int32 The ASN of the external network to which this endpoint belongs.
city string City for external endpoints
continent string Continent for external endpoints
country string Country for external endpoints, represented as ISO 3166-1 Alpha-3 country codes
region string Region for external endpoints

VpcDetails field format

Field Type Description
project_id string ID of the Google Cloud project containing the VPC. In a Shared VPC configuration, project_id is the ID of the host project.
subnetwork_name string Name of the subnet, if applicable
subnetwork_region string Region of the subnet, if applicable
vpc_name string Name of the network

InternetRoutingDetails field format

Field Type Description
egress_as_path AsPath List of relevant AS paths. If there are multiple AS paths available to the flow, the field might contain more than one AS path.

AsPath field format

Field Type Description
as_details AsDetails List of AS details for all systems in the AS path. The list starts from the first AS that is external to Google Cloud's network and ends with the AS to which the remote IP address belongs.

AsDetails field format

Field Type Description
asn uint32 The autonomous system number (ASN) of the AS

LoadBalancingDetails field format

Field Type Description
forwarding_rule_project_id string Google Cloud project ID of the forwarding rule
reporter string Cloud Load Balancing reporter. Can be either CLIENT or BACKEND.
  • If the reporter of the flow is the client of the load balancer, this field is set to CLIENT.
  • If the reporter of the flow is the backend of the load balancer, this field is set to BACKEND.
type string Load balancer type. Can be APPLICATION_LOAD_BALANCER, PROXY_NETWORK_LOAD_BALANCER, PASSTHROUGH_NETWORK_LOAD_BALANCER, or PROTOCOL_FORWARDING.
scheme string Load balancer scheme. Can be EXTERNAL_MANAGED, INTERNAL_MANAGED, EXTERNAL, INTERNAL, or INTERNAL_SELF_MANAGED.
url_map_name string Name of the URL map. Populated if the type of the load balancer is APPLICATION_LOAD_BALANCER.
forwarding_rule_name string Name of the forwarding rule
backend_service_name string Name of the backend service. Populated if the reporter is BACKEND and load balancer type is PASSTHROUGH_NETWORK_LOAD_BALANCER. If the backend group type is TARGET_POOL, this field isn't populated.
backend_group_name string Name of the backend group. Populated if the reporter is BACKEND and load balancer type is PASSTHROUGH_NETWORK_LOAD_BALANCER.
backend_group_type string Type of the backend group. Can be INSTANCE_GROUP, NETWORK_ENDPOINT_GROUP, or TARGET_POOL. Populated if the reporter is BACKEND and the load balancer type is PASSTHROUGH_NETWORK_LOAD_BALANCER.
backend_group_location string Location of the backend group. Can be a zone or a region depending on whether the scope of the backend group is zonal or regional. Populated if the reporter is BACKEND and the load balancer type is PASSTHROUGH_NETWORK_LOAD_BALANCER. If the backend group type is TARGET_POOL, this field isn't populated.
vpc VpcDetails VPC network details of the load balancer

NetworkServiceDetails field format

Field Type Description
dscp int32 If the Differentiated Services field is present in packet headers, this field is populated with the DSCP value.

PrivateServiceConnectDetails field format

Field Type Description
reporter string Private Service Connect reporter. Can be either CONSUMER or PRODUCER.
psc_endpoint PrivateServiceConnectEndpointDetails Endpoint details. Populated if the reporter is CONSUMER.
psc_attachment PrivateServiceConnectAttachmentDetails Service attachment details. Populated if the traffic flow includes a Private Service Connect producer.

PrivateServiceConnectEndpointDetails field format

Field Type Description
project_id string Google Cloud project ID of the Private Service Connect endpoint
region string Region of the endpoint. Not populated if the target service type is GLOBAL_GOOGLE_APIS.
psc_connection_id string Private Service Connect connection ID
target_service_type string Target service type. Can be either GLOBAL_GOOGLE_APIS or PUBLISHED_SERVICE.
vpc VpcDetails VPC network details of the Private Service Connect endpoint

PrivateServiceConnectAttachmentDetails field format

Field Type Description
project_id string Google Cloud project ID of the service attachment
region string Region of the service attachment
vpc VpcDetails VPC network details of the service attachment

Metadata annotations

Log records contain base fields and metadata fields. The Record format section lists which fields are type metadata and which are type base. All base fields are always included. You can customize which metadata fields you keep.

  • If you select all metadata, all metadata fields in the VPC Flow Logs record format are included in the flow logs. When new metadata fields are added to the record format, the flow logs automatically include the new fields.

  • If you select no metadata, this omits all metadata fields.

  • If you select custom metadata, you can specify the metadata fields that you want to include by the parent field, such as src_vpc, or by their full names, such as src_vpc.project_id

    When new metadata fields are added to the record format, they're excluded from the flow logs unless they're within a parent field that you have specified to include.

    • If you specify custom metadata using parent fields, when new metadata fields are added to the record format within that parent field, the flow logs will automatically include the new fields.

    • If you specify custom metadata using the full name of the field, new metadata fields that are added to the parent field are excluded from the flow logs.

For information about customizing metadata fields, see the Google Cloud CLI or API instructions for enabling VPC flow logging when you create a subnet.

GKE metadata annotations

Flows that have an endpoint in a GKE Cluster can be annotated with GKE metadata annotations, which can include details of the Cluster, Pod, and Service of the endpoint.

GKE Service annotations

Traffic sent to a ClusterIP, NodePort, or LoadBalancer can receive Service annotations. If sent to a NodePort or LoadBalancer, the flow receives the Service annotation on both hops of the connection.

Traffic sent directly to a Pod's Service port is annotated with a Service annotation on the destination endpoint.

Traffic sent to a Pod's Service port where the Pod is backing more than one Service on the same Service port is annotated with multiple Services on the destination endpoint. This is limited to two Services. If there are more than that, the endpoint will be annotated with a special MANY_SERVICES marker.

Pod annotations on internet traffic

Traffic between a Pod and the internet doesn't receive Pod annotations by default. VPC Flow Logs can't add Pod annotations because, for packets to the internet, the masquerade agent translates the Pod IP address to the node IP address before VPC Flow Logs sees the packet.

Because of the masquerade, Pod annotations are only visible if the destinations are within either the default non-masquerade destinations or in a custom nonMasqueradeCIDRs list. If you include internet destinations in a custom nonMasqueradeCIDRs list, you need to provide a way for the internal Pod IP addresses to be translated before they are delivered to the internet. For both private and non-private clusters, you can use Cloud NAT. See GKE interaction for more details.

Log filtering

When you enable VPC Flow Logs, you can set a filter based on both base and metadata fields that only preserves logs that match the filter. All other logs are discarded before being written to Logging, which saves you money and reduces the time needed to find the information you are looking for.

You can filter on any subset of fields listed in Record format, except for the following fields:

  • rtt_msec
  • bytes_sent
  • packets_sent
  • start_time
  • end_time

VPC Flow Logs filtering uses CEL, an embedded expression language for attribute-based logic expressions. Filter expressions for VPC Flow Logs have a limit of 2,048 characters. For more information, see Supported CEL logic operators.

For more information about CEL, see the CEL introduction and the language definition. The generation filter feature supports a limited subset of CEL syntax.

For information about creating a subnet that uses log filtering, see the gcloud CLI or API instructions for Enabling VPC Flow Logs when you create a subnet.

For information about configuring log filtering, see the gcloud CLI or API instructions for Updating VPC Flow Logs parameters.

Example 1: Limit logs collection to a specific VM named my-vm. In this case, only logs where the src_instance field as reported by the source of the traffic is my-vm or the dst_instance field as reported by the destination of the traffic is my-vm are recorded.

gcloud compute networks subnets update my-subnet \
    --logging-filter-expr="(src_instance.vm_name == 'my-vm' && reporter=='SRC') || (dest_instance.vm_name == 'my-vm' && reporter=='DEST')"

Example 2: Limit logs collection to packets whose source IP addresses are in the 10.0.0.0/8 subnet.

gcloud compute networks subnets update my-subnet \
    --logging-filter-expr="inIpRange(connection.src_ip, '10.0.0.0/8')"

Example 3: Limit logs collection to traffic that is external to a VPC.

gcloud compute networks subnets update my-subnet \
    --logging-filter-expr '!(has(src_vpc.vpc_name) && has(dest_vpc.vpc_name))'

Supported CEL logic operators

Expression Supported types Description
true, false Boolean Boolean constants

x == y

x != y

Boolean, Int, String

Comparison operators

Example: connection.protocol == 6

x && y

x || y

Boolean

Boolean logic operators

Example: connection.protocol == 6 && src_instance.vm_name == "vm_1"

!x Boolean Negation
1, 2.0, 0, ... Int Constant numeric literals
x + y String String concatenation
"foo", 'foo', ... String Constant string literal
x.lower() String Returns the lowercase value of the string
x.upper() String Returns the uppercase value of the string
x.contains(y) String Returns true if the string contains the specified substring
x.startsWith(y) String Returns true if the string begins with the specified substring
x.endsWith(y) String Returns true if the string ends with the specified substring
inIpRange(X, Y) String

Returns true if X is an IP and Y is an IP range that contains X

Example: inIpRange("1.2.3.1", "1.2.3.0/24")

x.containsFieldValue(y) x: list
y: map(string, string)

Returns true if the list contains an object with fields that match the specified key-value pairs

Example: dest_gke_details.service.containsFieldValue({'service_name': 'service1', 'service_namespace': 'namespace1'})

has(x) String

Returns true if the field is present.

What's next