Analyze your traffic flows

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

Using Flow Analyzer, you can perform the following tasks:

• Build and run a simple query on VPC Flow Logs
• Build a SQL filter (using a WHERE statement) for the query on VPC Flow Logs
• Organize the results using selected fields and sort the query results using the total traffic and aggregate packets
• View the traffic at chosen time intervals
• View the top five highest traffic flows over time in a graphical format, when compared with the rest of the traffic
• View the resources with highest traffic aggregated over the selected duration in a tabular format
• View the details of the traffic between a specific source and destination pair from the query results
• Drill down the query results using the remaining fields available in VPC Flow Logs

Before you begin

1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

2. Enable VPC Flow Logs when you create a subnet or enable VPC Flow Logs for an existing subnet.

Required roles and permissions

Because Flow Analyzer reads data on behalf of the user, ensure that you have sufficient permissions to read the bucket containing the logs. The bucket must also be upgraded to use Log Analytics.

• To let a user read logs in the buckets, use the Logs Explorer page. Use the Log Analytics page to grant one of the following roles:

  • For access to the _Default view on the _Default bucket, grant the Logs Viewer role (roles/logging.viewer).
  • For access to all logs in the _Default log bucket, including data access logs, grant the Private Logs Viewer role (roles/logging.privateLogViewer).

  For more information, see Logging roles.

• To let a user read logs that are stored in a user-defined bucket, grant the Logs View Accessor role (roles/logging.viewAccessor). You can restrict authorization to a specific log view. For more information, see Control access to a log view.

• Alternatively, create a custom role that grants the following permissions:

  • logging.buckets.get
  • logging.buckets.list
  • logging.logEntries.list
  • logging.logs.list
  • resourcemanager.projects.get

Build and run the query

To build and run the query, do the following:

Console

1. In the Google Cloud console, go to the Flow Analyzer page.

   Go to Flow Analyzer

2. Select a log bucket. If you plan to use the _Default log bucket, you can skip this step.

3. In the Traffic aggregation menu, select one of the following options:

   • Source - Destination: aggregate the traffic from the source to the destination.
   • Client - Server: aggregate the traffic in both directions by considering the resources with lower port numbers and service definitions or having GKE service properties as servers.

   For more information, see Traffic aggregation.

4. To set the time range of your query, use the time-range selector. The default time range is one hour, but you can select from the preset time options. You can specify a custom start and end time, or choose the time range around a specific time.

5. Click Re-run selected period in the chart.

6. In the source and destination or client and server fields, select a field from the list of fields.

7. Add one or more filter expressions to filter flows that match the selected key-value pairs using the key as the selected field.

   If you select more filters for the same field, an OR operator is used. If you select filters for different fields, an AND operator is used. For example, if you select two IP address values: 1.2.3.4 and 10.20.10.30 and two Country values: US and France, the following filter logic is applied to the query:

   (IP=1.2.3.4 OR IP=10.20.10.30) AND (Country=US OR Country=France)

8. Organize flow by using the fields. Select a field to organize the flow details.

9. Click Run new query.

   The Highest data flows chart and the All data flows table are updated.

10. Use the display options to customize the query results. For more information about the various display options available, see Display options. To select custom options, see Customize display options.

11. When you have finished making changes to the display options, click OK.

12. Click Re-run to run the query again with the selected display options.

Build and run a SQL query

To build and run a query in Flow Analyzer using SQL filter options, do the following:

1. In the Google Cloud console, go to the Flow Analyzer page.

   Go to Flow Analyzer

2. Select a log bucket. If you plan to use the _Default log bucket, you can skip this step.

3. To set the time range of your query, use the time-range selector or select Re-run selected period.

4. In the Traffic aggregation menu, select one of the following options:

   • Source - Destination: aggregate the traffic from the source to the destination.

   • Client - Server: aggregate the traffic in both directions by considering the resources with lower port numbers and service definitions as servers.

     For more information, see Traffic aggregation.

5. Click SQL Filters.

6. Enter the SQL filter query using BigQuery SQL syntax.

7. To view filter expression syntax and examples, click Filter expression syntax and examples.

8. Organize flow by using the fields. Select a field to organize the flow details.

9. Click Run new query.

   The Highest data flows chart and the All data flows table are updated.

10. Use the display options to customize the query results. For more information about the various display options available, see Display options. To select custom options, see Customize display options.

11. When you have finished making changes to the display options, click OK.

12. To run the query again with the selected display options, click Re-run.

Customize display options

To view specific details of the traffic flows, you can customize the display options. For more information about the various display options available, see Display options.

Console

1. Build the query.
   1. Select a log bucket. If you plan to use the _Default log bucket, you can skip this step.
   2. To set the time range of your query, use the time-range selector or select Re-run selected period.
   3. Select the filters.
   4. Select the fields to organize the results.
2. Run the query.
3. Select the metric type: Bytes sent or Packets sent.

4. Select a metric aggregation option.

   If you select Bytes sent as the metric, choose one of the following options:

   1. Total traffic: the total traffic for the chosen time period. Enabled by default.
   2. Average traffic rate: the average traffic rate for the chosen time period. Calculated only for the alignment periods during which the traffic was observed.
   3. Median traffic rate: the median traffic rate for the chosen time period. Calculated only for the alignment periods during which the traffic was observed.
   4. P95 traffic rate: the 95th percentile traffic rate for the chosen time period. Calculated only for the alignment periods during which the traffic was observed.
   5. Maximum traffic rate: the maximum traffic rate for the chosen time period.

   If you select Packets sent as the metric, choose one of the following options:

   1. Aggregate packets: the aggregate number of packets for the chosen time period. Enabled by default.
   2. Average packets rate: the average packet rate for the chosen time period. Calculated only for the alignment periods during which the traffic was observed.
   3. Median packets rate: the median packet rate for the chosen time period. Calculated only for the alignment periods during which the traffic was observed.
   4. P95 packets rate: the 95th percentile packet rate for the chosen time period. Calculated only for the alignment periods during which the traffic was observed.
   5. Maximum packets rate: the maximum packet rate for the chosen time period.

   For more information about the various metric aggregation options, see Metric aggregations.

5. Select the Alignment period. For more information about the alignment period, see Alignment period.

6. Choose a sampling point.

   • Source endpoint: the number of bytes sent or packets sent as reported at the source endpoint of a flow.
   • Destination endpoint: the number of bytes sent or packets sent as reported at the destination endpoint of a flow.
   • Sum of source and destination endpoint: the sum of bytes sent or packets sent as reported by both endpoints of a flow.
   • Average of source and destination endpoint: an average of bytes sent or packets sent as reported by both endpoints of a flow if both source and destination details are available in VPC Flow Logs.

   For more information, see Sampling point.

View flow details

To view flow details for a selected flow in the data flows table, do the following:

Console

1. Build the query.
   1. Select a log bucket. If you plan to use the _Default log bucket, you can skip this step.
   2. To set the time range of your query, use the time-range selector or select Re-run selected period.
   3. Select the filters.
   4. Select the fields to organize the results.
2. Run the query.

3. In the All data flows table, click Show details for any flow.

   The Flow details page that appears shows all the resources that match the selected filters and the traffic of these resources.

Drill down traffic flows

You can further refine the traffic of the selected resources. Using Flow Analyzer, you can drill down into the query results by using the remaining fields available in VPC Flow Logs. For more information, see Breakdown or drill down flow details.

To drill down traffic flows using more fields, do the following:

Console

1. Build the query.
   1. Select a log bucket. If you plan to use the _Default log bucket, you can skip this step.
   2. To set the time range of your query, use the time-range selector or select Re-run selected period.
   3. Select the filters.
   4. Select the fields to organize the results.
2. Run the query.

3. In the All data flows table, click Show details for any flow.

   The Flow details page that appears shows all the resources matching the selected filters and the traffic of these resources.

4. In the Drill down by list, select a field to do a drill down.

5. To compare with past traffic, click the Compare to past toggle. This feature lets you view six lines: three solid lines for the three top traffic flows from the drill down and three dashed lines in corresponding colors representing the past traffic.

What's next

• Metrics and alignment period
• Run Connectivity Tests from Flow Analyzer
• Monitor your traffic flows
• Troubleshoot data issues in Flow Analyzer