Use Firewall Insights

This page describes how to use Firewall Insights, which provides access to insights, recommendations, and metrics about your firewall rules. You can view this information on the Firewall Insights page in the Google Cloud console, as well as in several other places in the console. You can also get data by using the Recommender API or the Google Cloud CLI.

For an overview of the available insights, see the Firewall Insights overview.

For a list of firewall usage metrics, see Firewall Insights metrics.

Before you begin

Before you can use Firewall Insights, you must select a project, and then complete the required setup tasks. The prerequisites for some features differ from those of others. For details, see the following table.

Task	All metrics	Shadowed rule insights	Overly permissive rule insights	Deny rules with hits
Enable the Firewall Insights API	✔	✔	✔	✔
Enable Firewall Rules Logging	✔		✔	✔
Enable the Recommender API		✔	✔
Enable this type of insight		✔	✔
Configure an observation period			✔	✔

The following sections describe each of these tasks.

Select a project

Before you complete any prerequisites or take other actions with Firewall Insights, we recommend that you create or select a Google Cloud project. Use the following steps:

1. In the Google Cloud console, go to the Project selector page.

   Go to Project selector

2. Select or create a Google Cloud project.

3. Make sure that billing is enabled for your Cloud project.

Enable the Firewall Insights API

Before you can perform any tasks using Firewall Insights, you must enable the Firewall Insights API. Use the following steps. Alternatively, you can use the console API Library, as described in Enabling APIs.

Console

1. In the Google Cloud console, go to the Firewall Insights page.

   Go to Firewall Insights

2. Under Firewall Insights API, click Enable.

gcloud

Run the following command:

gcloud services enable firewallinsights.googleapis.com

Enable Firewall Rules Logging

If you want to view any of the following, you must enable Firewall Rules Logging:

• Metrics about firewall rules
• Insights about overly permissive rules or deny rules; these insights are known collectively as log-based insights

Firewall Insights produces metrics and log-based insights only for those rules that have logging enabled. For more information, see the Firewall Rules Logging overview.

Enable the Recommender API

You must enable the Recommender API if you want to do one of the following:

• Use shadowed rule insights
• Use overly permissive rule insights

• Retrieve any data by making API calls or by using the Google Cloud CLI

Console

1. In the console, go to the Enable access to API page.

   Enable access to API

2. Make sure that the correct project is selected, and then click Next.

3. Click Enable.

gcloud

Run the following command:

gcloud services enable recommender.googleapis.com

Enable shadowed rule or overly permissive rule insights

Firewall Insights does not generate shadowed rule or overly permissive rule insights unless you actively enable these features on the Firewall Insights page.

After you enable either feature, you might have to wait up to 48 hours to see the generated insights.

When you create or update a firewall rule, you might have to wait up to ten days to see machine learning predictions for overly permissive rule insights. In the meantime, you can view insights based on data gathered from Firewall Rules Logging.

Note: If you enable these features and decide to disable them later, consider downloading the insights that Firewall Insights has generated so far. If you disable either feature, Firewall Insights does not save the insights that were previously generated for your project, even if you enable the feature again later.

Console

1. In the Google Cloud console, go to the Firewall Insights page.

   Go to Firewall Insights

2. Click Configuration.

3. Click Enablement.

4. As appropriate, move the slider to Enabled or Disabled for one or both of the following:

   • Shadowed rule insights

   • Overly permissive rule insights

API

You can use the Recommender API to enable or disable shadowed rule insights and/or overly permissive rule insights. You can also use the API to set the observation period for overly permissive rule insights and retrieve configuration details.

Enable insights and set the observation period

You can enable shadowed rule insights and/or overly permissive rule insights by using the updateConfig method.

To use the updateConfig method, you must set values for all of its parameters. This means that, at the same time you enable or disable insights, you must also configure the observation period for overly permissive insights.

To make this type of update, use the following request.

  PATCH https://recommender.googleapis.com/v1beta1/projects/PROJECT_ID/locations/global/insightTypes/google.compute.firewall.Insight/config
  {
    "name": "projects/PROJECT_ID/locations/global/insightTypes/google.compute.firewall.Insight/config",
      "insightTypeGenerationConfig": {
        "params": {
          "observation_period": "OBSERVATION_PERIOD_OVERLY_PERMISSIVE",
          "enable_shadowed_rule_insights": ENABLEMENT_SHADOWED,
          "enable_overly_permissive_rule_insights": ENABLEMENT_OVERLY_PERMISSIVE
         }
       },
    "etag": "\"ETAG\"",
  }

Replace the following values:

• PROJECT_ID: the ID of your project
• OBSERVATION_PERIOD_OVERLY_PERMISSIVE: the time, in seconds, of the observation period for overly permissive rules insights
• ENABLEMENT_SHADOWED: a boolean value that represents whether shadowed rule insights are enabled
• ENABLEMENT_OVERLY_PERMISSIVE: a boolean value that represents whether overly permissive rule insights are enabled
• ETAG: the IAM policy etag value; to retrieve the etag value, use the getConfig method, as described in the following section

Example

  PATCH https://recommender.googleapis.com/v1beta1/projects/PROJECT_ID/locations/global/insightTypes/google.compute.firewall.Insight/config
  {
    "name": "projects/PROJECT_ID/locations/global/insightTypes/google.compute.firewall.Insight/config",
      "insightTypeGenerationConfig": {
        "params": {
          "observation_period": "604800s",
          "enable_shadowed_rule_insights": true,
          "enable_overly_permissive_rule_insights": true
         }
       },
    "etag": "\"ETAG\"",
  }

Retrieve configuration details

To retrieve details about how Firewall Insights is configured, use the getConfig method, as shown in the following example.

  GET https://recommender.googleapis.com/v1beta1/projects/PROJECT_ID/locations/global/insightTypes/google.compute.firewall.Insight/config

Configure observation periods

For some insights, you can configure an observation period—the period of time that the insight covers. You can do this for overly permissive rule and deny rule insights.

For example, suppose you set the observation period for deny rule insights to two months. In this case, when you review the list of deny rules with hits, Firewall Insights shows you only those rules that had hits at some point during the past two months. Suppose you later change the observation period to one month. In this case, a different number of rules might be identified, because Firewall Insights would be analyzing a smaller period of time.

When reviewing insights and configuring observation periods, be aware of the following:

• When you configure the observation period for deny rules with hits, Firewall Insights updates the insight results immediately.

• When you update the observation period for overly permissive rule insights, Firewall Insights can take up to 48 hours to update existing results. In the interim, the observation period for existing results matches the previously configured observation period.

• For overly permissive insights, if no firewall rules were identified by the insight, Firewall Insights does not display the observation period that was used.

• Shadowed rule insights do not have an observation period because they do not evaluate historical data. Shadowed rule analysis evaluates your existing firewall rule configuration every 24 hours.

• Traffic log data from the last 24 hours may not be included when generating insights.

Console

To configure an observation period:

1. In the Google Cloud console, go to the Firewall Insights page.

   Go to Firewall Insights

2. Click Configuration.

3. Click Observation period.

4. As appropriate, set the Observation period drop-down list to the appropriate time for each of the following:

   • Overly permissive rule insights

   • Deny rule insights

API

To set the observation period for deny rules with hits, you must use the console. However, you can use the Recommender API to set the observation period for overly permissive rule insights. You can also use the API to enable insights and to retrieve configuration details.

Set the observation period and enable insights

You can set the observation period for overly permissive rules insights by using the updateConfig method.

To use the updateConfig method, you must set values for all of its parameters. This means you must also specify whether shadowed rule insights and overly permissive rules insights are enabled or disabled.

To make this type of update, use the following request.

  PATCH https://recommender.googleapis.com/v1beta1/projects/PROJECT_ID/locations/global/insightTypes/google.compute.firewall.Insight/config
  {
    "name": "projects/PROJECT_ID/locations/global/insightTypes/google.compute.firewall.Insight/config",
      "insightTypeGenerationConfig": {
        "params": {
          "observation_period": "OBSERVATION_PERIOD_OVERLY_PERMISSIVE",
          "enable_shadowed_rule_insights": ENABLEMENT_SHADOWED,
          "enable_overly_permissive_rule_insights": ENABLEMENT_OVERLY_PERMISSIVE
         }
       },
    "etag": "\"ETAG\"",
  }

Replace the following values:

• PROJECT_ID: the ID of your project
• OBSERVATION_PERIOD_OVERLY_PERMISSIVE: the time, in seconds, of the observation period for overly permissive rules insights
• ENABLEMENT_SHADOWED: a boolean value that represents whether shadowed rule insights are enabled
• ENABLEMENT_OVERLY_PERMISSIVE: a boolean value that represents whether overly permissive rule insights are enabled
• ETAG: the IAM policy etag value; to retrieve the etag value, use the getConfig method, as described in the following section

Example

  PATCH https://recommender.googleapis.com/v1beta1/projects/PROJECT_ID/locations/global/insightTypes/google.compute.firewall.Insight/config
  {
    "name": "projects/PROJECT_ID/locations/global/insightTypes/google.compute.firewall.Insight/config",
      "insightTypeGenerationConfig": {
        "params": {
          "observation_period": "604800s",
          "enable_shadowed_rule_insights": true,
          "enable_overly_permissive_rule_insights": true
         }
       },
    "etag": "\"ETAG\"",
  }

Retrieve configuration details

To retrieve details about how Firewall Insights is configured, use the getConfig method, as shown in the following example.

  GET https://recommender.googleapis.com/v1beta1/projects/PROJECT_ID/locations/global/insightTypes/google.compute.firewall.Insight/config

Schedule a custom refresh cycle

Set up a refresh cycle to generate shadowed rule insights for your project.

You can schedule the refresh cycle to begin on a specified date, and customize the cycle frequency. The default cycle frequency is one day (24 hours).

Console

Configure a custom refresh cycle for insights:

1. In the Google Cloud console, go to the Firewall Insights page.

   Go to Firewall Insights

2. Click Configuration.

3. Click Enablement.

4. To enable shadowed rule insights, click the toggle.

5. In the Start on field, enter a date from which the custom refresh cycle starts.

6. In the Repeat every field, select the frequency for the refresh cycle starting from the cycle start date:

   • day: every 24 hours
   • week: every week on the days you select
   • month: every month
   • quarter: every quarter

   The new insight generation schedule takes effect 24 hours after saving changes to the schedule.

Use the Firewall Insights landing page

The Firewall Insights landing page in the console provides cards for all insights, including the following:

• Shadowed firewall rules
• Overly permissive rules, including each of the following:
  • Allow rules with no hits
  • Allow rules with unused attributes
  • Allow rules with overly permissive IP address or port ranges
• Deny rules with no hits

Each card includes a list of all the rules in your project that meet the insight criteria. If you want to limit the results to one VPC network only, use the filter bar at the top of the page to select a network.

The following sections describe how to view each insight.

View shadowed firewall rules

To learn about this insight, see Shadowed rules in the Firewall Insights overview.

Console

1. In the console, go to the Firewall Insights page.

   Go to Firewall Insights

2. On the card named Shadowed rules, click View full list. In response, the console displays the Shadowed rules page, which lists all the rules that were identified as shadowed. The Insight column for each rule provides a summary of why the rule was identified as a shadowed rule.

3. To view more details about the shadowed rule and the rules that shadow it, click the insight.

View allow rules with no hits

To learn about this insight, see Allow rules with no hits in the Firewall Insights overview.

Console

1. In the console, go to the Firewall Insights page.

   Go to Firewall Insights

2. On the card named Allow rules with no hit, click View full list. In response, the console displays the Allow rules with no hits page, which lists all the rules that had no hits during the observation period. The Insight column for each rule shows if the firewall rule had no hits during the observation period. The Future hit prediction column shows a prediction of future usage based on firewall rules in the same organization.

3. For any rule in the list, do any of the following as appropriate:

   • To view the Firewall rules details page for the rule, click the name of the rule.
   • To view logging for the rule, click the View audit logs link.
   • To see details about the prediction, click the prediction link. In response, the Insight details panel is displayed. This panel describes the rule's main attributes. It also describes the attributes of other similar rules in the project, along with their hit counts.

View allow rules with unused attributes

To learn about this insight, see Allow rules with unused attributes in the Firewall Insights overview.

Console

1. In the console, go to the Firewall Insights page.

   Go to Firewall Insights

2. On the card named Allow rules with unused attributes, click View full list. In response, the console displays the Allow rules with unused attributes page. This page lists all rules identified that had unused attributes during the observation period. The Insight column for each rule shows the number of attributes that were unused during the observation period.

3. For any rule in the list, do any of the following as appropriate:

   • To view the Firewall rules details page for the rule, click the name of the rule.
   • To view logging for the rule, click the View audit logs link.
   • To see details about the prediction, click the prediction link. In response, the Insight details panel is displayed. This panel describes the rule's main attributes. It also describes other rules in the project that have similar attributes.

View allow rules with overly permissive IP address or port ranges

To learn about this insight, see Allow rules with overly permissive IP address or port ranges in the Firewall Insights overview.

Be aware that your project might have firewall rules that allow access from certain IP address blocks for load balancer health checks or for other Google Cloud functionality. These IP addresses might not be hit, but they should not be removed from your firewall rules. For more information about these ranges, see the Compute Engine documentation.

Console

1. In the console, go to the Firewall Insights page.

   Go to Firewall Insights

2. On the card named Allow rules with overly permissive IP address or port ranges, click View full list. In response, the console displays a list of all the rules that had overly permissive ranges during the observation period.

3. For any rule in the list, do any of the following as appropriate:

   • To view the Firewall rules details page for any rule, click the name of the rule.
   • To view logging for the rule, click the View audit logs link.
   • To see suggestions about how to narrow the range, click the link in the Insight column. In response, the Insight details panel is displayed. This panel describes the rule's main attributes. It suggests more narrowly defined IP address or port ranges that you could use.

View deny rules with hits

To learn about this insight, see Deny rules with hits in the Firewall Insights overview.

Console

1. In the console, go to the Firewall Insights page.

   Go to Firewall Insights

2. On the card named Deny rules with hits, click View full list. In response, the console displays the Deny rules with hits page, which lists all the deny rules that had hits during the observation period.

3. To review the packets dropped by a firewall, click Hit count.

Mark an insight as dismissed

If any insight isn't meaningful, or if you want to hide it for any other reason, you can dismiss it. After you dismiss an insight, the console no longer displays the insight to you or other users unless you restore it.

Console

1. In the console, go to the Firewall Insights page.

   Go to Firewall Insights

2. Find the appropriate card and click View full list.

3. Select the rules that you want to dismiss, and then click Dismiss.

Restore a dismissed insight

If you dismissed an insight that you later think is relevant, you or another user can restore it and make it visible in the console.

Console

1. In the console, go to the Firewall Insights page.

   Go to Firewall Insights

2. Click Dismiss History. In response, the console displays the Dismissed insights page.

3. Select the insights that you want to restore, and then click Restore.

Export insights

If needed, you can export insights in CSV or JSON format.

You might want to export insights for any of the following reasons:

• You need to import the data into another system.
• You want to access the data while you are offline.

• You intend to disable Firewall Insights, but you want to retain access to previously generated insights.

Console

1. In the console, go to the Firewall Insights page.

   Go to Firewall Insights

2. Click Save as.

3. Follow the prompts to choose a format for your insights and download them.

Use the Recommendation Hub

The Recommendation Hub is a feature of the Recommender product that provides usage recommendations for Google Cloud products and services.

The console for the Recommendation Hub shows the following firewall rule insights:

• Shadowed firewall rules
• Overly permissive rules, including each of the following:
  • Allow rules with no hits
  • Allow rules with unused attributes
  • Allow rules with overly permissive IP address or port ranges

The Recommendation Hub shows these insights along with recommendations for other products, such as Identity and Access Management (IAM) and VM Rightsizing.

For general information about Recommendation Hub, see Getting started with Recommendation Hub.

Note: Within Recommendation Hub, the names of the cards differ from those used on the Firewall Insights page. For details, review the following sections.

View shadowed firewall rules

To learn about this insight, see Shadowed rules in the Firewall Insights overview.

Console

1. In the console, go to the Recommendation Hub.

   Go to the Recommendation Hub

2. On the card named Simplify firewall configuration, click View all. The Recommendation Hub displays a page that lists shadowed rules. The Insight column for each rule provides a summary of why the rule was identified as a shadowed rule.

3. To view more details about the shadowed rule and the rules that shadow it, click the insight.

View overly permissive rule insights

To learn about overly permissive rule insights, see the following sections in the Firewall Insights overview:

• Allow rules with no hits
• Allow rules with unused attributes

• Allow rules with overly permissive IP address or port ranges

Console

1. In the console, go to the Recommendation Hub.

   Go to the Recommendation Hub

2. On the card named Review overly permissive firewall rules, click View all.

3. Use the tabs at the top of the page to toggle between the insight types.

   You can click the insight for any rule listed to understand why the rule was included in the list. The insight detail shows more information about the rule's hit count and attributes.

Mark an insight as dismissed

If any insight isn't meaningful, or if you want to hide it for any other reason, you can dismiss it. After you dismiss an insight, the console no longer displays the insight to you or other users unless you restore it.

Console

1. In the console, go to the Recommendation Hub.

   Go to the Recommendation Hub

2. Find the appropriate card and click View all.

3. If you want to dismiss an insight about an overly permissive rule, use the tabs at the top of the page to navigate to the appropriate insight category.

4. Select the insights that you want to dismiss, and then click Dismiss.

5. In the confirmation dialog, click Yes, dismss.

Restore a dismissed insight

If you have dismissed an insight that you later think is relevant, you or a user can restore it and make it visible in the console by following these steps.

Console

1. In the console, go to the Recommendation Hub.

   Go to the Recommendation Hub

2. Find the appropriate card and click View all.

3. If you want to dismiss an insight about an overly permissive rule, use the tabs at the top of the page to navigate to the appropriate insight category.

4. At the top of the page, click History.

5. Click the Dismissed tab, which shows dismissed recommendations and insights for the project.

6. Select the insights that you want to restore.

7. Click Restore.

8. In the confirmation dialog, click Restore.

Use the Firewall details page

For more information about this page, see Listing firewall rules for a VPC network.

List insights for a project

Console

1. In the console, go to the Firewall page.

   Go to Firewall

2. For each firewall rule, view the name of available insights in the Insights column.

3. You can click the name of an insight to view its detail. The following sections describe how to view and interpret the detail for each type of insight.

View allow rules with no hit in the last 24 months

Console

1. In the console, go to the Firewall page.

   Go to Firewall

2. In the Last hit column, review the last time that a given firewall rule was used in the last 24 months.

View the usage history chart for a rule

Console

1. In the console, go to the Firewall page.

   Go to Firewall

2. Click a firewall rule name.

3. In the Hit count monitoring section of the page, view the resulting chart that shows the firewall hit count for a given time period. You can select tabs for different time periods above the chart.

View deny rules with hits for an observation period

Console

1. In the console, go to the Firewall page.

   Go to Firewall

2. In the Hit count column, view the number of unique connections used for a given firewall rule in the last 24 months (default).

Use the VM network interface details page

View firewall usage on the Network interface details page for a VM.

For more information about this page, see Listing firewall rules for a network interface of a VM instance.

View rules with hits in the last 24 months

Console

1. In the console, go to the Compute Engine VM instances page.

   Go to Compute Engine VM instances

2. Choose a VM, and on the far right of the page, click its more actions menu .

3. On the menu, select View network details.

4. In the Firewall and routes details section, click the Firewall Rules tab.

5. In the Hit count column, view the hit counts for allow and deny traffic in the last 24 months for all firewall rules associated with a specific network interface.

Work with insights using gcloud commands or the API

Firewall Insights uses Recommender commands. Recommender is a Google Cloud service that provides usage recommendations for Google Cloud products and services.

List insights

gcloud

To list insights for a project, run the following command:

gcloud beta recommender insights list \
    --project=PROJECT_ID \
    --location=global \
    --insight-type=google.compute.firewall.Insight \
    --filter=EXPRESSION \
    --limit=LIMIT \
    --page-size=PAGE_SIZE \
    --sort-by=SORT_BY \
    --format=json

Replace PROJECT_ID with the project ID that you want to list insights for.

The location flag always uses the location named global. The insight-type flag always uses the insight type named google.compute.firewall.Insight. Unless you format the output in JSON, the command output is tabular.

The following fields are optional:

• EXPRESSION: apply this Boolean filter to each resource that you want to list

  If the expression evaluates as True, that item is listed. For more details and examples of filter expressions, run $ gcloud topic filters or see the gcloud topic filters documentation.

• LIMIT: the maximum number of resources to list; the default number of resources listed is unlimited

• PAGE_SIZE: the maximum number of resources to list per page

  The default page size is determined by the service; otherwise, there is no paging. Paging might be applied before or after FILTER and LIMIT.

• SORT_BY: a list of comma-separated field key names to sort by for a resource

  The default order is ascending. To specify a descending order, prefix a field with ~ (a tilde).

API

To get all of the insights for a Google Cloud project, make a GET request to the projects.locations.insightTypes.insights method:

GET https://recommender.googleapis.com/v1/{parent=projects/*/locations/global/insightTypes/*}/insights

The following example shows a sample response for this command:

insights {
"name": "projects/{project_number}/locations/global/insightTypes/google.compute.firewall.Insight/insights/{insight-id}"
"description": "Firewall projects/{project_id}/global/firewalls/{shadowed_firewall_name} is shadowed by   projects/{project_id}/global/firewalls/{shadowing_firewall_name}."
"content": {
  "shadowingFirewalls": [
    "//compute.googleapis.com/projects/{project_id}/global/firewalls/shadowing_firewall_name1}"
  ]
},
"lastRefreshTime": "2020-04-01T19:16:43Z",
"observationPeriod": "0s",
"stateInfo" {
 "state": "ACTIVE"
}
"category": "SECURITY"
"targetResources":[
 "//compute.googleapis.com/projects/{project_id}/global/firewalls/{shadowed_firewall_name}"
 ],
"insightSubtype": "SHADOWED_RULE"
}

Describe insights

gcloud

To list details for an insight about a particular firewall rule, run the following command:

gcloud beta recommender insights describe INSIGHT_ID \
    --project=PROJECT_NAME \
    --location=global \
    --insight-type=google.compute.firewall.Insight

Replace the following with values for your network:

• INSIGHT_ID: the ID of the insight to describe
• PROJECT_NAME: the name of the project that you want to list insights for

The location flag always uses the location named global. The insight-type flag always uses the insight type named google.compute.firewall.Insight.

API

To get details for an insight, make a GET request to the projects.locations.insightTypes.insights method:

GET
https://recommender.googleapis.com/v1/{name=projects/*/locations/global/insightTypes/*/insights/*}
{
 "name": projects/PROJECT_ID/locations/LOCATION/insightTypes/INSIGHT_TYPE_ID/insights/INSIGHT_ID,
{

Replace the following with values for your network:

• PROJECT_ID: the project ID
• LOCATION: always use the location named global
• INSIGHT_TYPE_ID: always use a value of google.compute.firewall.Insight
• INSIGHT_ID: the insight ID for the insight

What's next

• To review your VPC firewall runtime usage, clean up and optimize your firewall rule configurations, and tighten up security boundaries, see Common use cases.