About VPC Flow Logs records
This page describes the VPC Flow Logs record format, including which base and metadata fields are available. It also explains how you can use log filtering so that only logs that match certain criteria are generated.
Record format
Log records contain base fields, which are the core fields of every log record, and metadata fields that add additional information. Metadata fields may be omitted to save storage costs.
Some log fields are in a multi-field format, with more than one piece of data
in a given field. For example, the connection
field is of the IpConnection
format, which contains the source and destination IP address and port, plus the
protocol, in a single field. These multi-field fields are described below the
record format table.
The values for metadata fields aren't based on the data plane path; they are approximations and might be missing or incorrect. Unlike metadata fields, the values for base fields are taken directly from packet headers.
Field | Field format | Field type: Base or optional metadata |
---|---|---|
connection |
IpConnection
5-tuple describing the flow. |
Base |
reporter |
string
The side which reported the flow. Can be SRC ,
DEST , SRC_GATEWAY , or
DEST_GATEWAY .
|
Base |
rtt_msec |
int64
Latency as measured during the time interval. This field is available only for TCP traffic reported from VMs. The measured latency is the time elapsed between sending a SEQ and receiving a corresponding ACK. The latency result is the sum of the network RTT and any time consumed by the application. |
Base |
bytes_sent |
int64
Amount of bytes sent from the source to the destination. |
Base |
packets_sent |
int64
Number of packets sent from the source to the destination. |
Base |
start_time |
string
Timestamp (RFC 3339 date string format) of the first observed packet during the aggregated time interval. |
Base |
end_time |
string
Timestamp (RFC 3339 date string format) of the last observed packet during the aggregated time interval. |
Base |
Source and destination metadata fields | ||
src_gateway |
GatewayDetails
If the source of the flow is an on-premises or other cloud endpoint that is connected to Google Cloud through a gateway such as a VLAN attachment for Cloud Interconnect or a Cloud VPN tunnel and either of the following conditions is met, this field is populated with gateway details:
|
Metadata |
dest_gateway |
GatewayDetails
If the destination of the flow is an on-premises or other cloud endpoint that is connected to Google Cloud through a gateway such as a VLAN attachment for Cloud Interconnect or a Cloud VPN tunnel and either of the following conditions is met, this field is populated with gateway details:
|
Metadata |
src_gke_details |
GkeDetails
If the source of the flow is a Google Kubernetes Engine (GKE) endpoint, this field is populated with GKE endpoint details. |
Metadata |
dest_gke_details |
GkeDetails
If the destination of the flow is a GKE endpoint, this field is populated with GKE endpoint details. |
Metadata |
src_google_service |
GoogleServiceDetails
If the source of the flow is a Google API, this field is populated with available Google API metadata. |
Metadata |
dest_google_service |
GoogleServiceDetails
If the destination of the flow is a Google API, this field is populated with available Google API metadata. |
Metadata |
src_instance |
InstanceDetails
If the source of the flow is a VM located in a VPC network and either of the following conditions is met, this field is populated with VM instance details:
|
Metadata |
dest_instance |
InstanceDetails
If the destination of the flow is a VM located in a VPC network and either of the following conditions is met, this field is populated with VM instance details.
|
Metadata |
src_location |
GeographicDetails
If the source of the flow is a public IP address outside of the VPC network, this field is populated with available location metadata. |
Metadata |
dest_location |
GeographicDetails
If the destination of the flow is a public IP address outside of the VPC network, this field is populated with available location metadata. |
Metadata |
src_vpc |
VpcDetails
If the source of the flow is a VM located in a VPC network and either of the following conditions is met, this field is populated with VPC network details:
|
Metadata |
dest_vpc |
VpcDetails
If the destination of the flow is a VM located in a VPC network and either of the following conditions is met, this field is populated with VPC network details:
|
Metadata |
Other metadata fields | ||
internet_routing_details |
InternetRoutingDetails
If the flow is between Google Cloud and the internet, this field is populated with routing details. Available only for egress flows. |
Metadata |
load_balancing |
LoadBalancingDetails
If the flow passes through a load balancer in one of the following configurations, this field is populated with Cloud Load Balancing details:
|
Metadata |
network_service |
NetworkServiceDetails
If the Differentiated Services Code Point (DSCP) header is set, this field is populated with network service details. |
Metadata |
psc |
PrivateServiceConnectDetails
If the flow passes through Private Service Connect in either of the following configurations, this field is populated with Private Service Connect details:
|
Metadata |
IpConnection field format
Field | Type | Description |
---|---|---|
protocol | int32 | The IANA protocol number |
src_ip | string | Source IP address |
dest_ip | string | Destination IP address |
src_port | int32 | Source port |
dest_port | int32 | Destination port |
GatewayDetails field format
Field | Type | Description |
---|---|---|
project_id | string | Google Cloud project ID of the gateway |
location | string | Region of the gateway |
name | string | Name of the gateway |
type | string |
Type of the gateway. Can be INTERCONNECT_ATTACHMENT or
VPN_TUNNEL .
|
vpc | VpcDetails | VPC network details of the gateway |
GkeDetails field format
Field | Type | Description |
---|---|---|
cluster | ClusterDetails | GKE cluster metadata |
pod | PodDetails | GKE Pod metadata, populated when the source or destination of the traffic is a Pod |
service | ServiceDetails |
GKE Service metadata, populated in Service endpoints
only. The record contains up to two Services. If there are more than two
relevant Services, this field contains a single Service with a special
MANY_SERVICES marker.
|
ClusterDetails field format
Field | Type | Description |
---|---|---|
cluster_location | string | Location of the cluster. This can be a zone or a region depending if the cluster is zonal or regional. |
cluster_name | string | GKE cluster name. |
PodDetails field format
Field | Type | Description |
---|---|---|
pod_name | string | Name of the Pod |
pod_namespace | string | Namespace of the Pod |
pod_workload | WorkloadDetails | Metadata about the top-level workload resource that controls the Pod |
WorkloadDetails field format
Field | Type | Description |
---|---|---|
workload_name | string | Name of the top-level workload controller |
workload_type | string |
Type of the top-level workload controller. Can be
DEPLOYMENT , REPLICA_SET ,
STATEFUL_SET , DAEMON_SET , JOB ,
CRON_JOB , or
REPLICATION_CONTROLLER .
|
ServiceDetails field format
Field | Type | Description |
---|---|---|
service_name | string |
Name of the Service. If there are more than two relevant Services, the
field is set to a special MANY_SERVICES marker.
|
service_namespace | string | Namespace of the Service |
Example:
If there are two services, the Service field looks like this:
service: [ 0: { service_name: "my-lb-service" service_namespace: "default" } 1: { service_name: "my-lb-service2" service_namespace: "default" } ]
If there are more than two services, the Service field looks like this:
service: [ 0: { service_name: "MANY_SERVICES" } ]
GoogleServiceDetails field format
Field | Type | Description |
---|---|---|
type | string | This field is set to GOOGLE_API . |
InstanceDetails field format
Field | Type | Description |
---|---|---|
project_id | string | ID of the Google Cloud project that contains the VM resource |
region | string | Region of the VM |
vm_name | string | Instance name of the VM |
zone | string | Zone of the VM |
GeographicDetails field format
Field | Type | Description |
---|---|---|
asn | int32 | The ASN of the external network to which this endpoint belongs. |
city | string | City for external endpoints |
continent | string | Continent for external endpoints |
country | string | Country for external endpoints, represented as ISO 3166-1 Alpha-3 country codes |
region | string | Region for external endpoints |
VpcDetails field format
Field | Type | Description |
---|---|---|
project_id | string | ID of the Google Cloud project containing the VPC. In a
Shared VPC configuration, project_id is the ID
of the host project. |
subnetwork_name | string | Name of the subnet, if applicable |
subnetwork_region | string | Region of the subnet, if applicable |
vpc_name | string | Name of the network |
InternetRoutingDetails field format
Field | Type | Description |
---|---|---|
egress_as_path | AsPath | List of relevant AS paths. If there are multiple AS paths available to the flow, the field might contain more than one AS path. |
AsPath field format
Field | Type | Description |
---|---|---|
as_details | AsDetails | List of AS details for all systems in the AS path. The list starts from the first AS that is external to Google Cloud's network and ends with the AS to which the remote IP address belongs. |
AsDetails field format
Field | Type | Description |
---|---|---|
asn | uint32 | The autonomous system number (ASN) of the AS |
LoadBalancingDetails field format
Field | Type | Description |
---|---|---|
forwarding_rule_project_id | string | Google Cloud project ID of the forwarding rule |
reporter | string | Cloud Load Balancing reporter. Can be either
CLIENT or BACKEND .
|
type | string | Load balancer type. Can be APPLICATION_LOAD_BALANCER ,
PROXY_NETWORK_LOAD_BALANCER ,
PASSTHROUGH_NETWORK_LOAD_BALANCER ,
or PROTOCOL_FORWARDING . |
scheme | string | Load balancer scheme. Can be EXTERNAL_MANAGED ,
INTERNAL_MANAGED , EXTERNAL ,
INTERNAL , or INTERNAL_SELF_MANAGED . |
url_map_name | string | Name of the URL map. Populated if the type of the
load balancer is APPLICATION_LOAD_BALANCER . |
forwarding_rule_name | string | Name of the forwarding rule |
backend_service_name | string | Name of the backend service. Populated if
the reporter is BACKEND and load balancer type is
PASSTHROUGH_NETWORK_LOAD_BALANCER . If the backend group type
is TARGET_POOL , this field isn't populated. |
backend_group_name | string | Name of the backend group. Populated if
the reporter is BACKEND and load balancer type is
PASSTHROUGH_NETWORK_LOAD_BALANCER . |
backend_group_type | string | Type of the backend group. Can be INSTANCE_GROUP ,
NETWORK_ENDPOINT_GROUP , or TARGET_POOL .
Populated if the reporter is BACKEND and the load balancer
type is PASSTHROUGH_NETWORK_LOAD_BALANCER . |
backend_group_location | string | Location of the backend group. Can be a zone or a region
depending on whether the scope of the backend group is zonal or regional.
Populated if the reporter is BACKEND and the load balancer
type is PASSTHROUGH_NETWORK_LOAD_BALANCER . If the backend
group type is TARGET_POOL , this field isn't populated. |
vpc | VpcDetails | VPC network details of the load balancer |
NetworkServiceDetails field format
Field | Type | Description |
---|---|---|
dscp | int32 | If the Differentiated Services field is present in packet headers, this field is populated with the DSCP value. |
PrivateServiceConnectDetails field format
Field | Type | Description |
---|---|---|
reporter | string | Private Service Connect reporter.
Can be either CONSUMER or
PRODUCER . |
psc_endpoint | PrivateServiceConnectEndpointDetails |
Endpoint details. Populated if the reporter is CONSUMER .
|
psc_attachment | PrivateServiceConnectAttachmentDetails | Service attachment details. Populated if the traffic flow includes a Private Service Connect producer. |
PrivateServiceConnectEndpointDetails field format
Field | Type | Description |
---|---|---|
project_id | string | Google Cloud project ID of the Private Service Connect endpoint |
region | string | Region of the endpoint. Not populated if the target
service type is GLOBAL_GOOGLE_APIS . |
psc_connection_id | string | Private Service Connect connection ID |
target_service_type | string | Target service type. Can be either GLOBAL_GOOGLE_APIS or
PUBLISHED_SERVICE . |
vpc | VpcDetails | VPC network details of the Private Service Connect endpoint |
PrivateServiceConnectAttachmentDetails field format
Field | Type | Description |
---|---|---|
project_id | string | Google Cloud project ID of the service attachment |
region | string | Region of the service attachment |
vpc | VpcDetails | VPC network details of the service attachment |
Metadata annotations
Log records contain base fields and metadata fields. The Record format section lists which fields are type metadata and which are type base. All base fields are always included. You can customize which metadata fields you keep.
If you select all metadata, all metadata fields in the VPC Flow Logs record format are included in the flow logs. When new metadata fields are added to the record format, the flow logs automatically include the new fields.
If you select no metadata, this omits all metadata fields.
If you select custom metadata, you can specify the metadata fields that you want to include by the parent field, such as
src_vpc
, or by their full names, such assrc_vpc.project_id
When new metadata fields are added to the record format, they're excluded from the flow logs unless they're within a parent field that you have specified to include.
If you specify custom metadata using parent fields, when new metadata fields are added to the record format within that parent field, the flow logs will automatically include the new fields.
If you specify custom metadata using the full name of the field, new metadata fields that are added to the parent field are excluded from the flow logs.
For information about customizing metadata fields, see the Google Cloud CLI or API instructions for enabling VPC flow logging when you create a subnet.
GKE metadata annotations
Flows that have an endpoint in a GKE Cluster can be annotated with GKE metadata annotations, which can include details of the Cluster, Pod, and Service of the endpoint.
GKE Service annotations
Traffic sent to a ClusterIP, NodePort, or LoadBalancer can receive Service annotations. If sent to a NodePort or LoadBalancer, the flow receives the Service annotation on both hops of the connection.
Traffic sent directly to a Pod's Service port is annotated with a Service annotation on the destination endpoint.
Traffic sent to a Pod's Service port where the Pod is backing more
than one Service on the same Service port is annotated with multiple Services
on the destination endpoint. This is limited to two Services. If there are more
than that, the endpoint will be annotated with a special MANY_SERVICES
marker.
Pod annotations on internet traffic
Traffic between a Pod and the internet doesn't receive Pod annotations by default. VPC Flow Logs can't add Pod annotations because, for packets to the internet, the masquerade agent translates the Pod IP address to the node IP address before VPC Flow Logs sees the packet.
Because of the masquerade, Pod annotations are only visible if the destinations
are within either the default non-masquerade
destinations
or in a custom nonMasqueradeCIDRs
list.
If you include internet destinations in a custom nonMasqueradeCIDRs
list, you
need to provide a way for the internal Pod IP addresses to be translated before
they are delivered to the internet. For both private and non-private clusters,
you can use Cloud NAT. See GKE
interaction for more details.
Log filtering
When you enable VPC Flow Logs, you can set a filter based on both base and metadata fields that only preserves logs that match the filter. All other logs are discarded before being written to Logging, which saves you money and reduces the time needed to find the information you are looking for.
You can filter on any subset of fields listed in Record format, except for the following fields:
rtt_msec
bytes_sent
packets_sent
start_time
end_time
VPC Flow Logs filtering uses CEL, an embedded expression language for attribute-based logic expressions. Filter expressions for VPC Flow Logs have a limit of 2,048 characters. For more information, see Supported CEL logic operators.
For more information about CEL, see the CEL introduction and the language definition. The generation filter feature supports a limited subset of CEL syntax.
For information about creating a subnet that uses log filtering, see the gcloud CLI or API instructions for Enabling VPC Flow Logs when you create a subnet.
For information about configuring log filtering, see the gcloud CLI or API instructions for Updating VPC Flow Logs parameters.
Example 1: Limit logs collection to a specific VM named my-vm
. In this case,
only logs where the src_instance
field as reported by the source of the
traffic is my-vm
or the dst_instance
field as reported by the destination
of the traffic is my-vm
are recorded.
gcloud compute networks subnets update my-subnet \ --logging-filter-expr="(src_instance.vm_name == 'my-vm' && reporter=='SRC') || (dest_instance.vm_name == 'my-vm' && reporter=='DEST')"
Example 2: Limit logs collection to packets whose source IP addresses are in the
10.0.0.0/8
subnet.
gcloud compute networks subnets update my-subnet \ --logging-filter-expr="inIpRange(connection.src_ip, '10.0.0.0/8')"
Example 3: Limit logs collection to traffic that is external to a VPC.
gcloud compute networks subnets update my-subnet \ --logging-filter-expr '!(has(src_vpc.vpc_name) && has(dest_vpc.vpc_name))'
Supported CEL logic operators
Expression | Supported types | Description |
---|---|---|
true, false | Boolean | Boolean constants |
x == y x != y |
Boolean, Int, String | Comparison operators Example: connection.protocol == 6 |
x && y x || y |
Boolean | Boolean logic operators Example: connection.protocol == 6 && src_instance.vm_name == "vm_1" |
!x | Boolean | Negation |
1, 2.0, 0, ... | Int | Constant numeric literals |
x + y | String | String concatenation |
"foo", 'foo', ... | String | Constant string literal |
x.lower() | String | Returns the lowercase value of the string |
x.upper() | String | Returns the uppercase value of the string |
x.contains(y) | String | Returns true if the string contains the specified substring |
x.startsWith(y) | String | Returns true if the string begins with the specified substring |
x.endsWith(y) | String | Returns true if the string ends with the specified substring |
inIpRange(X, Y) | String | Returns true if X is an IP and Y is an IP range that contains X Example: inIpRange("1.2.3.1", "1.2.3.0/24") |
x.containsFieldValue(y) |
x: list y: map(string, string) |
Returns true if the list contains an object with fields that match the specified key-value pairs Example: dest_gke_details.service.containsFieldValue({'service_name': 'service1', 'service_namespace': 'namespace1'}) |
has(x) | String | Returns true if the field is present. |