Using VPC Flow Logs

VPC Flow Logs records network flows sent from or received by VM instances. These logs can be used for network monitoring, forensics, real-time security analysis, and expense optimization.

You can view flow logs in Stackdriver Logging, and you can export logs to any destination that Stackdriver Logging export supports (Cloud Pub/Sub, BigQuery, etc.).

Flow logs are aggregated by connection, at 5-second intervals, from Compute Engine VMs and exported in real time. By subscribing to Cloud Pub/Sub, you can analyze flow logs using real-time streaming APIs.

Key Properties

  • You can enable or disable VPC Flow Logs per VPC network subnet.
  • Traffic coverage: you can monitor network traffic to and from your VMs, including:

    • Network flows between VMs in the same VPC
    • Network flows between VMs in a VPC network and hosts in your on-premises network that are connected via VPN or Cloud Interconnect
    • Network flows between VMs and end locations on the Internet
    • Network flows between VMs and Google services in production
  • Protocols: you can monitor network flows for TCP and UDP.

  • Each flow record includes:

    • Flow 5-tuple: source and destination IP addresses, source and destination ports, and the IANA protocol number
    • Timestamps in RFC3339 format of the first and last observed packet during the aggregated time interval (5 seconds)
    • Metrics

      • The number of packets for the interval
      • The number of bytes for the interval (a.k.a. throughput)
      • RTT for TCP flows
    • GCP annotations

      • VPC network and subnet name for source/destination IPs if the endpoint is within the VPC
      • Region and zone for source/destination IPs if the endpoint is within the VPC
      • VM instance name for source/destination IPs if the endpoint is within the VPC
    • Geo annotations for endpoints not in the VPC: continent, country, region and city for source/destination IPs

  • Filters: you can use filters to select which flow logs are excluded from Stackdriver Logging and which are exported to external APIs.

  • For a given flow, if both VMs are in the same VPC, flow logs are reported from both the source and destination of the flow. Reporting from both source and destination enables you to troubleshoot connectivity issues.
  • VPC Flow Logs is natively built into the networking stack of the VPC network infrastructure. There is no extra delay and no performance penalty in routing the logged IP packets to their destination.

Use cases

Network monitoring

VPC Flow Logs provides you with real-time visibility into network throughput and performance. You can:

  • Monitor the VPC network
  • Perform network diagnosis
  • Filter the flow logs by VMs and by applications to understand traffic changes
  • Understand traffic growth for capacity forecasting

Understand network usage, and optimize network traffic expenses

You can analyze network usage with VPC Flow Logs. You can analyze the network flows for:

  • Traffic between regions and zones
  • Traffic to specific countries on the Internet
  • Top talkers

Based on the analysis, you can optimize network traffic expenses.

Network forensics

You can utilize VPC Flow Logs for network forensics. For example, if an incident occurs:

  • You can find out which IPs talked with whom and when
  • You can investigate any compromised IPs by analyzing all the incoming and outgoing network flows

Real-time security analysis

You can leverage the real-time streaming APIs (through Cloud Pub/Sub), and integrate with SIEM (Security Information and Event Management) systems. This can provide real-time monitoring, correlation of events, analysis, and security alerts.

Logs collection

Flow logs are collected for each VM connection every 5 seconds. This data is then annotated and sent to you with the data and format described here.

Record format

Field Field Format Default Field type: Base or Optional metadata
connection IpConnection
5-Tuple describing this connection.
Base
start_time string
Timestamp (RFC3339 date string format) of the first observed packet during the aggregated time interval
unknown Base
end_time string
Timestamp (RFC3339 date string format) of the last observed packet during the aggregated time interval
unknown Base
bytes_sent int64
Amount of bytes sent from the source to the destination
0 Base
packets_sent int64
Number of packets sent from the source to the destination
0 Base
rtt_msec int64
Latency as measured (for TCP flows only) during the time interval. This is the time elapsed between sending a SEQ and receiving a corresponding ACK and it contains the network RTT as well as the application related delay.
Base
reporter string
The side which reported the flow. Can be either ‘SRC' or ‘DEST'.
unknown Base
src_instance InstanceDetails
If the source of the connection was a VM located on the Google VPC, this field will be populated with VM instance details. Note that in a Shared VPC configuration, project_id will correspond to that of the service project.
Metadata
dest_instance InstanceDetails
If the destination of the connection was a VM located on the Google VPC, this field will be populated with VM instance details. Note that in a Shared VPC configuration, project_id will correspond to that of the service project.
Metadata
src_vpc VpcDetails
If the source of the connection was a VM located on the Google VPC, this field will be populated with VPC network details. Note that in a Shared VPC configuration, project_id will correspond to that of the host project.
Metadata
dest_vpc VpcDetails
If the destination of the connection was a VM located on the Google VPC, this field will be populated with VPC network details. Note that in a Shared VPC configuration, project_id will correspond to that of the host project.
Metadata
src_location GeographicDetails
If the source of the connection was external to the Google VPC, this field will be populated with available location metadata.
Metadata
dest_location GeographicDetails
If the destination of the connection was external to the Google VPC, this field will be populated with available location metadata.
Metadata

IpConnection field format

Field Type Description Default Value
src_ip string Source IP address unknown
src_port int32 Source port 0
dest_ip string Destination IP address unknown
dest_port int32 Destination port 0
protocol int32 The IANA protocol number 255

InstanceDetails field format

Field Type Description
project_id string ID of the project containing the VM
vm_name string Instance name of the VM
region string Region of the VM
zone string Zone of the VM

VpcDetails field format

Field Type Description
project_id string ID of the project containing the VPC
vpc_name string VPC on which the VM is operating
subnetwork_name string Subnetwork on which the VM is operating

GeographicDetails field format

Field Type Description
continent string Continent for external endpoints
country string Country for external endpoints
region string Region for external endpoints
city string City for external endpoints

Traffic pattern examples

This section demonstrates how VPC Flow Logs works for the following use cases:

VM-to-VM flows in the same VPC

VM flows within a VPC (click to enlarge)
VM flows within a VPC (click to enlarge)

For VM-to-VM flows in the same VPC, flow logs are reported from both source and destination by default. For example, VMs 10.50.0.2 and 10.10.0.1 are connected in a VPC. The traffic (5342 bytes) sent from 10.50.0.2 to 10.10.0.1 is reported from both SRC (10.50.0.2) and DEST (10.10.0.1).

reporter connection.src_ip connection.dst_ip sent_bytes VPC annotations
SRC 10.10.0.1 10.50.0.2 1224 src_instance.*
dest_instance.*
src_vpc.*
dest_vpc.*
SRC 10.50.0.2 10.10.0.1 5342 src_instance.*
dest_instance.*
src_vpc.*
dest_vpc.*
DEST 10.50.0.2 10.10.0.1 5342 src_instance.*
dest_instance.*
src_vpc.*
dest_vpc.*
DEST 10.10.0.1 10.50.0.2 1224 src_instance.*
dest_instance.*
src_vpc.*
dest_vpc.*

VM-to-external flows

VM-to-external flows (click to enlarge)
VM-to-external flows (click to enlarge)

For VM-to-external flows, flow logs are reported from the VMs:

  • From the source VM for egress flows
  • From the destination VM for ingress flows

This applies to:

  • Traffic between a VPC network and an on-premises network through VPN or Cloud Interconnect
  • Traffic between VMs and locations on the Internet

For example, VM 10.10.0.1 and external endpoint 10.30.0.2 are connected through a VPN gateway. The traffic (1224 bytes) sent from 10.10.0.1 to 10.30.0.2 is reported from SRC (VM 10.10.0.1); the traffic (5342 bytes) sent from 10.30.0.2 to 10.10.0.1 is reported from DEST (VM 10.10.0.1).

reporter connection.src_ip connection.dst_ip sent_bytes VPC annotations
SRC 10.10.0.1 10.30.0.2 1224 src_instance.*
src_vpc.*
dest_location.*
DEST 10.30.0.2 10.10.0.1 5342 dest_instance.*
dest_vpc.*
src_location.*

VM-to-VM flows for Shared VPC

Shared VPC flows (click to enlarge)
Shared VPC flows (click to enlarge)

For VM-to-VM flows for Shared VPC, you could enable VPC Flow Logs in the host project. For example, subnet 10.10.0.0/20 belongs to a Shared VPC Network defined in a host project. You will be able to see flow logs from VMs belonging to this subnet. This includes the flows from and to the VMs in the service projects (project Webserver, project Recommendation, and project Analytics).

  • src_vpc.project_id and dest_vpc.project_id are for the host project
  • src_instance.project_id and dest_instance.project_id are for the service projects
connection
.src_ip
src_instance
.project_id
src_vpc
.project_id
connection
.dst_ip
dest_instance
.project_id
dest_vpc
.project_id
10.10.0.1 ‘Project Webserver' ‘host_project' 10.20.0.1 ‘Project Recommendation' ‘host_project'

Service projects do not own the Shared VPC network and do not have access to the flow logs of the Shared VPC network.

VM-to-VM flows for VPC Peering

VPC Peering flows (click to enlarge)
VPC Peering flows (click to enlarge)

VM-to-VM flows for peered VPCs are reported in the same way as for external endpoints. Annotations for VMs in the other (peered) VPC are not supported.

For example, VM 10.10.0.1 (in project Analytics-prod) and VM 10.50.0.2 (in project Webserver-test) are connected through VPC Peering. When VPC Flow Logs is enabled in project Analytics-prod, the traffic (1224 bytes) sent from 10.10.0.1 to 10.50.0.2 is reported from SRC (VM 10.10.0.1); the traffic (5342 bytes) sent from 10.50.0.2 to 10.10.0.1 is reported from DEST (VM 10.10.0.1).

reporter connection.src_ip connection.dst_ip sent_bytes VPC annotations
SRC 10.10.0.1 10.50.0.2 1224 src_instance.*
src_vpc.*
DEST 10.50.0.2 10.10.0.1 5342 dest_instance.*
dest_vpc.*

VM-to-VM flows for Internal Load Balancing

Internal Load Balancing flows (click to enlarge)
Internal Load Balancing flows (click to enlarge)

VM-to-VM flows sent through an internal load balancer are reported from both source and destination by default. For example, the internal load balancer IP 10.240.0.200 is serving client instances 192.168.1.1 and 192.168.1.5.

The traffic (1224 bytes) sent from 192.168.1.1 to the internal load balancer IP 10.240.0.200 is reported from both SRC (192.168.1.1) and DEST (10.240.0.200). When reporting from DEST (internal load balancer IP 10.240.0.200), VPC annotations will be available for both SRC and DEST. When reporting from SRC (client instance IP 192.168.1.1), VPC annotations are available only for SRC.

reporter connection.src_ip connection.dst_ip sent_bytes VPC annotations
SRC 192.168.1.1 10.240.0.200 1224 src_instance.*
src_vpc.*
SRC 10.240.0.200 192.168.1.1 5342 src_instance.*
src_vpc.*
dest_instance.*
dest_vpc.*
DEST 10.240.0.200 192.168.1.1 5342 dest_instance.*
dest_vpc.*
DEST 192.168.1.1 10.240.0.200 1224 src_instance.*
src_vpc.*
dest_instance.*
dest_vpc.*

Enabling VPC flow logging

When you enable VPC Flow Logs, you enable for all VMs in a subnet.

Enabling VPC flow logging when you create a subnet

Console

  1. Go to the VPC networks page in the Google Cloud Platform Console.
    Go to the VPC networks page
  2. Click the network where you want to add a subnet.
  3. Click Add subnet.
  4. Under Flow logs, select On.
  5. Populate other fields as appropriate.
  6. Click Add.

gcloud

gcloud beta compute networks subnets create [NAME] \
    --enable-flow-logs
    [other flags as needed]

Enabling VPC flow logging for a subnet

Console

  1. Go to the VPC networks page in the Google Cloud Platform Console.
    Go to the VPC networks page
  2. Click the subnet you want to update.
  3. Click Edit.
  4. Under Flow logs, select On.
  5. Click Save.

gcloud

gcloud beta compute networks subnets update [NAME] \
    --enable-flow-logs

Disabling VPC flow logging for a subnet

Console

  1. Go to the VPC networks page in the Google Cloud Platform Console.
    Go to the VPC networks page
  2. Click the subnet you want to update.
  3. Click Edit.
  4. Under Flow logs, select Off.
  5. Click Save.

gcloud

gcloud beta compute networks subnets update [NAME] \
    --no-enable-flow-logs

Accessing logs via Stackdriver Logging

Configuring IAM

Follow the access control guide for Stackdriver Logging.

View logs through the logs viewer page.

You need your project's project ID for these commands.

Accessing all flow logs

  1. Go to the Logs page in the Google Cloud Platform Console.
    Go to the Logs page
  2. Select Subnetwork in the first pull-down menu.
  3. Select vpc_flows in the second pull-down menu.

Alternatively, go to the Logs page and paste the following into the Filter by label or text search field.

resource.type="gce_subnetwork"
logName="projects/{#project_id}/logs/compute.googleapis.com%2Fvpc_flows"

Accessing logs for specific subnets

  1. Go to the Logs page in the Google Cloud Platform Console.
    Go to the Logs page
  2. Select Subnetwork > [SUBNET_NAME] in the first pull-down menu.
  3. Select vpc_flows in the second pull-down menu.

Alternatively, go to the Logs page and paste the following into the Filter by label or text search field.

resource.type="gce_subnetwork"
logName="projects/{#project_id}/logs/compute.googleapis.com%2Fvpc_flows"
resource.labels.subnetwork_name="{#subnetwork_name}"

Accessing logs for specific VMs

  1. Go to the Logs page in the Google Cloud Platform Console.
    Go to the Logs page
  2. Paste the following into the Filter by label or text search field.
    resource.type="gce_subnetwork"
    logName="projects/{#project_id}/logs/compute.googleapis.com%2Fvpc_flows"
    jsonPayload.src_instance.vm_name="{#vm_name}"
    

Accessing logs for traffic to a specific prefix

  1. Go to the Logs page in the Google Cloud Platform Console.
    Go to the Logs page
  2. Paste the following into the Filter by label or text search field.
    resource.type="gce_subnetwork"
    logName="projects/{#project_id}/logs/compute.googleapis.com%2Fvpc_flows"
    ip_in_net(jsonPayload.connection.dest_ip, {#subnet})
    

Accessing logs for specific ports and protocols

For an individual port

  1. Go to the Logs page in the Google Cloud Platform Console.
    Go to the Logs page
  2. Paste the following into the Filter by label or text search field.
    resource.type="gce_subnetwork"
    logName="projects/{#project_id}/logs/compute.googleapis.com%2Fvpc_flows"
    jsonPayload.connection.src_port={#port}
    jsonPayload.connection.protocol={#protocol}
    

For more than one port

  1. Go to the Logs page in the Google Cloud Platform Console.
    Go to the Logs page
  2. Paste the following into the Filter by label or text search field.
    resource.type="gce_subnetwork"
    logName="projects/{#project_id}/logs/compute.googleapis.com%2Fvpc_flows"
    jsonPayload.connection.src_port=({#port1} OR {#port2})
    jsonPayload.connection.protocol={#protocol}
    

Exporting logs to BigQuery, Cloud Pub/Sub, and custom targets

You can export flow logs from Stackdriver Logging to a destination of your choice as described in the Stackdriver Logging documentation. Refer to the previous section for example filters.

Troubleshooting

No ‘vpc_flows’ appear in Stackdriver Logging under the ’gce_subnetwork’ resource

  • VPC flows are only supported for VPC network. If you have a legacy network, you will not see any logs.
  • In Shared VPC networks, logs only appear in the host project, not the service projects. Make sure you look for the logs in the host project.
  • Stackdriver Logging exclusion filters block specified logs. Make sure there are no exclusion rules that discard VPC Flow Logs.
    1. Go to Resource usage.
    2. Click the Exclusions tab.
    3. Make sure there are no exclusion rules that might discard VPC Flow Logs.

No RTT or byte values on some of the logs

  • RTT measurements may be missing if not enough packets were sampled to capture RTT. This is more likely to happen for low volume connections.
  • No RTT values are available for UDP flows.
  • Some packets are sent with no payload. If header-only packets were sampled, the bytes value will be 0.

Some flows are missing

  • Only UDP and TCP protocols are supported. There will be no logs for other protocols.
  • There is some sampling in the packet process. Some packets in very low volume flows may be missed.

Pricing

Standard pricing for Stackdriver Logging, BigQuery, or Cloud Pub/Sub apply. VPC flow logs generation will be charged starting in GA as described in Virtual Private Cloud pricing.

FAQ

  • Do logs cover both allowed and denied traffic based on the firewall rules?

    • Logs cover all traffic seen by the VM. If traffic leaving a VM was blocked by an egress rule for that VM, the traffic is still logged by the VM. If incoming traffic is blocked by an ingress rule, that traffic is not seen by the VM and will not be logged for that VM.
  • Does VPC Flow Logs work with instances with multiple interfaces?

  • Does VPC Flow Logs work with legacy networks?

What's next

Send feedback about...