Using logging and monitoring

Logging

Cloud NAT logging allows you to log NAT connections and errors. When Cloud NAT logging is enabled, one log entry can be generated for each of the following scenarios:

  • When a network connection using NAT is created.
  • When a packet is dropped because no port was available for NAT.

You can choose to log both kinds of events, or only one or the other.

Created logs are sent to Cloud Logging.

Specifications

The following specifications apply to Cloud NAT logging:

  • Cloud NAT logging handles TCP and UDP traffic only.

  • Cloud NAT logging only logs dropped packets if they are egress (outbound) TCP and UDP packets. It does not log dropped incoming packets. For example, if an inbound response to an outbound request is dropped for any reason, no error is logged.

Each VM instance can only generate a certain number of log entries per unit time, proportional to its number of vCPUs. The VM can generate 50-100 log entries per second per vCPU.

  • This rate threshold affects the number of events that can be logged. Even if some events are filtered out, their occurrence counts toward the number of possible log entries. Limiting logs to only errors or only network address translation connections does not necessarily increase the number of viewed log entries. For example, if you choose to log only successful connections, periods of excessive failed connection attempts and NAT errors can still restrict the number of successful connection log entries.

  • Cloud NAT logging does not log every single packet. Even if the VM's rate threshold has not been reached, some conditions can cause events to be omitted from the log. You should rely on the presence of entries in Cloud NAT logging to make informed decisions, but you should not assume that the absence of entries means that an event didn't happen.

Configuring logging

Use the following steps to configure Cloud NAT logging.

Enabling logging

If logging is enabled, all collected logs are sent to Cloud Logging by default. You can filter these so that only certain logs are sent.

You can also specify these values when you create or edit a NAT gateway. The following directions show how to enable logging for an existing NAT gateway.

Console

  1. In the Google Cloud Console, go to the Cloud NAT page.

    Go to the Cloud NAT page

  2. Click your NAT gateway.

  3. Click Edit.

  4. Click Logging, minimum ports, timeout to open that section.

  5. Under Stackdriver logging, select one of the following:

    • No logging: disables logging
    • Translation and errors: sends all logs to Logging
    • Translation only: sends a log only when a connection is created; does not log dropped packets
    • Errors only: sends a log when a packet is dropped because no port was available; does not log new connections

  6. Click Save.

gcloud

The following commands enable logging for an existing NAT gateway.

In each command, replace the following:

  • NAT_GATEWAY: the name of the NAT gateway
  • ROUTER_NAME: the name of the Cloud Router that hosts the NAT gateway
  • REGION: the region of the Cloud Router

To log network address translation events and errors:

gcloud compute routers nats update NAT_GATEWAY \
    --router=ROUTER_NAME \
    --region=REGION \
    --enable-logging

To log only network address translation events:

gcloud compute routers nats update NAT_GATEWAY \
    --router=ROUTER_NAME \
    --region=REGION \
    --enable-logging \
    --log-filter=TRANSLATIONS_ONLY

To log only errors:

gcloud compute routers nats update NAT_GATEWAY \
    --router=ROUTER_NAME \
    --region=REGION \
    --enable-logging \
    --log-filter=ERRORS_ONLY

Clearing log filters

If you have a filter set, you can clear it. Clearing a log filter means that both network address translation events and errors are logged, provided that logging is enabled.

Console

  1. In the Cloud Console, go to the Cloud NAT page.

    Go to the Cloud NAT page

  2. Click your NAT gateway.

  3. Click Edit.

  4. Click Logging, minimum ports, timeout to open that section.

  5. Under Stackdriver logging, select Translation and errors.

  6. Click Save.

gcloud 

gcloud compute routers nats update NAT_GATEWAY \
    --router=ROUTER_NAME \
    --region=REGION \
    --log-filter=ALL

Replace the following:

  • NAT_GATEWAY: the name of the NAT gateway
  • ROUTER_NAME: the name of the Cloud Router that hosts the NAT gateway
  • REGION: the region of the Cloud Router
  • --log-filter=ALL: sets the log filter to accept all logs

Disabling logging

To disable logging, do the following:

Console

  1. In the Cloud Console, go to the Cloud NAT page.

    Go to the Cloud NAT page

  2. Click your NAT gateway.

  3. Click Edit.

  4. Click Logging, minimum ports, timeout to open that section.

  5. Under Stackdriver logging, select No logging.

  6. Click Save.

gcloud 

gcloud compute routers nats update NAT_GATEWAY \
    --router=ROUTER_NAME \
    --region=REGION \
    --no-enable-logging

Replace the following:

  • NAT_GATEWAY: the name of the NAT gateway
  • ROUTER_NAME: the name of the Cloud Router that hosts the NAT gateway
  • REGION: the region of the Cloud Router

Determining logging status

To determine the status for logging, do the following:

Console

  1. In the Google Cloud Console, go to the Cloud NAT page.

    Go to the Cloud NAT page

  2. Click your NAT gateway.

  3. In the Logging, minimum ports, timeout section, inspect the selections under Stackdriver logging.

gcloud 

gcloud compute routers nats describe NAT_GATEWAY \
    --router=ROUTER_NAME \
    --region=REGION

Replace the following:

  • NAT_GATEWAY: the name of the NAT gateway
  • ROUTER_NAME: the name of the Cloud Router that hosts the NAT gateway
  • REGION: the region of the Cloud Router

Viewing logs

To view NAT logs, do the following:

Console

  1. In the Google Cloud Console, go to the Logs Viewer page.

    Go to the Logs Viewer page

    • To see all NAT logs, in the first pull-down menu, select Cloud NAT Gateway.
    • To see logs for only one region, in the first pull-down menu, select Cloud NAT Gateway, and then slide the cursor right to select a region.
    • To see logs for only one gateway, in the first pull-down menu, select Cloud NAT Gateway, and then slide the cursor right to select a region. Slide the cursor right again to select a single gateway.

Alternatively, go to the Logs Viewer page and paste the following into the Filter by label or text search field:

resource.type="nat_gateway"
logName="projects/{#project_id}/logs/compute.googleapis.com%2Fnat_flows"

gcloud 

gcloud logging read 'resource.type=nat_gateway' \
    --limit=10 \
    --format=json

Where:

  • resource.type=nat_gateway: limits the output to your NAT gateways
  • --limit=10: limits the output to 10 entries; you can input a different value to see more or fewer entries, or omit it entirely to see a continuous scroll of logs
  • --format=json: displays the output in JSON format

For more options, see Reading log entries.

You can configure the export of logs-based metrics for resource logs.

What is logged

Cloud NAT log entries contain information useful for monitoring and debugging your NAT traffic. Log entries contain the following types of information:

  • General information shown in most Google Cloud logs, such as severity, project ID, project number, and timestamp.
  • Specific information related to Cloud NAT. Some log fields contain entries that are themselves multiple fields. These entries and field descriptions are shown in the following tables.

Log fields

Field Value Meaning
connection object(NatIpConnection) 7-tuple describing the source VM IP address and port, NAT source IP address and port, destination IP address and port, and IP address protocol of this connection.
allocation_status enum Indicates whether this connection was successfully allocated or dropped. One of OK or DROPPED.
gateway_identifiers object(NatGateway) The NAT gateway configuration that the connection used.
endpoint object(InstanceDetails) VM instance details. In a Shared VPC configuration, project_id corresponds to the service project.
vpc object(VpcDetails) VPC network details. In a Shared VPC configuration, project_id corresponds to that of the host project.
destination object(DestinationDetails) Details of the destination of the connection.

NatIpConnection field format

Field Type Description
src_ip string Source IP address
src_port int32 Source port
nat_ip string NAT IP address
nat_port int32 NAT assigned port
dest_ip string Destination IP address
dest_port int32 Destination port
protocol int32 IANA protocol number

NatGateway field format

Field Type Description
gateway_name string Name of the NAT gateway
router_name string Cloud Router associated with the NAT gateway
region string Region of the Cloud Router

InstanceDetails field format

Field Type Description
project_id string ID of the project containing the VM
vm_name string Instance name of the VM
region string Region of the Cloud Router
zone string Zone of the VM

VpcDetails field format

Field Type Description
project_id string ID of the project containing the network
vpc_name string Network on which the VM is operating
subnetwork_name string Subnet on which the VM is operating

DestinationDetails field format

Field Type Description
geo_location object(GeographicDetails) If the destination of the connection was external to Google Cloud, this field is populated with available location metadata.
instance object(InstanceDetails) If the destination of the connection is an instance within the same project as the source, this field is populated with VM instance details.

GeographicDetails field format

Field Type Description
continent string Continent for external endpoints
country string Country for external endpoints
region string Region for external endpoints
city string City for external endpoints
asn string The autonomous system number (ASN) of the external network to which this endpoint belongs.

Examples

Example 1: NAT-ed TCP connection record from a VM instance in a Shared VPC network going to an external server in France.

{
insertId: "1the8juf6vab1t"
jsonPayload: {
       connection: {
            Src_ip: "10.0.0.1"
            Src_port: 45047
            Nat_ip: "203.0.113.17"
            Nat_port: 34889
            dest_ip : "198.51.100.142"
            Dest_port: 80
            Protocol: "tcp"
       }
       allocation_status: "OK"
       Gateway_identifiers: {
             Gateway_name: "my-nat-1"
             router_name: "my-router-1"
             Region: "europe-west1"
       }
       Endpoint: {
             Project_id: "service-project-1"
             Vm_name: "vm-1"
             Region: "europe-west1"
             Zone: "europe-west1-b"
       }
       Vpc: {
             Project_id: "host-project"
             Vpc_name: "network-1"
             Subnetwork_name: "subnetwork-1"
       }
       Destination: {
             Geo_location: {
                   Continent: "Europe"
                   Country: "France"
                   Region: "Nouvelle-Aquitaine"
                   City: "Bordeaux"
             }
       }
}
logName: "projects/host-project/logs/compute.googleapis.com%2Fnat_flows"
receiveTimestamp: "2018-06-28T10:46:08.123456789Z"
resource: {
      labels: {
            region: "europe-west1-d"
            project_id: "host-project"
            router_id: "987654321123456"
            gateway_name: "my-nat-1"
      }
      type: "nat_gateway"
}
labels: {
      nat.googleapis.com/instance_name: "vm-1"
      nat.googleapis.com/instance_zone: "europe-west1-b"
      nat.googleapis.com/nat_ip: "203.0.113.17"
      nat.googleapis.com/network_name: "network-1"
      nat.googleapis.com/router_name: "my-router-1"
      nat.googleapis.com/subnetwork_name: "subnetwork-1"
}
timestamp: "2018-06-28T10:46:00.602240572Z"
}

Example 2: Record for a packet dropped because there were no available ports. Sending VM was trying to reach the external IP address of another VM in the same project.

{
insertId: "1the8juf6vab1l"
jsonPayload: {
       connection: {
            Src_ip: "10.0.128.1"
            Src_port: 45047
            dest_ip : "192.0.2.87"
            Dest_port: 80
            Protocol: "tcp"
       }
       allocation_status: "DROPPED"
       Gateway_identifiers: {
             Gateway_name: "my-nat-2"
             Cloud_router: "my-router-1"
             Region: "europe-west1"
       }
       Endpoint: {
             Project_id: "service-project-1"
             Vm_name: "vm-1"
             Region: "europe-west1"
             Zone: "europe-west1-b"
       }
       Vpc: {
             Project_id: "host-project"
             Vpc_name: "network-1"
             Subnetwork_name: "subnetwork-1"
       }
       Destination: {
             Instance: {
                   Project_id: "service-project-1"
                   Vm_name: "vm-2"
                   Region: "asia-east1"
                   Zone: "asia-east1-b"
             }
       }
}
logName: "projects/host-project/logs/compute.googleapis.com%2Fnat_flows"
receiveTimestamp: "2018-06-28T10:46:09.123456789Z"
resource: {
       labels: {
              region: "europe-west1-d"
              project_id: "host-project"
              router_id: "987654321123456"
              gateway_name: "my-nat-2"
       }
       type: "nat_gateway"
}
timestamp: "2018-06-28T10:46:01.602240572Z"
}

Logging pricing

See Logging pricing.

Monitoring

Cloud NAT exposes key metrics to Cloud Monitoring that give you insights into your fleet's usage of NAT gateways.

Metrics are sent automatically to Cloud Monitoring. There, you can create custom dashboards, set up alerts, and query the metrics.

Following are the required Identity and Access Management (IAM) roles:

  • For Shared VPC users with VMs and NAT gateways defined in different projects, access to the VM level metrics requires the roles/monitoring.viewer IAM role for the project of each VM.

  • For the NAT gateway resource, access to the gateway metrics requires the roles/monitoring.viewer IAM role for the project that contains the gateway.

Viewing predefined dashboards

Cloud NAT provides a set of predefined dashboards that display activity across your gateway:

  • Open connections
  • Egress data processed
  • Ingress data processed
  • Port utilization
  • NAT allocation errors

To view predefined dashboards from the details page of a particular gateway, see the following steps:

Console

  1. In the Cloud Console, go to the Cloud NAT page.

    Go to the Cloud NAT page

  2. Click an existing gateway.

  3. Click the Monitoring tab.

  4. Scroll down to see all the dashboards. You can change the view from 1 hour to 30 days by using the control at the top of the page. Hovering over a point on the graph gives you details for that specific time.

Defining custom dashboards

To create custom Cloud Monitoring dashboards over Cloud NAT metrics, see the following steps:

Console

  1. In the Cloud Console, go to Monitoring.

    Go to Monitoring

  2. Click Dashboards, and then click Create dashboard.

  3. Create a name for your dashboard, and then click Confirm.

  4. Click Add chart.

  5. Give the chart a title.

  6. Specify the resource type and metric:

    • For Cloud NAT metrics for a VM, search by using the compute.googleapis.com/nat/ prefix.
    • For metrics for a NAT gateway, use the router.googleapis.com/nat/ prefix.

    For lists of metrics and filtering dimensions, see Using VM instance metrics and Using NAT gateway metrics.

  7. Specify any filtering dimensions.

  8. Click Save.

Defining alerting policies

Console

You can create alerting policies to monitor the values of metrics and to notify you when those metrics violate a condition.

To create an alerting policy that monitors one or more Cloud NAT gateway or Compute Engine VM instance resources, follow these steps:

  1. In the Google Cloud Console, go to the Monitoring page.

    Go to Monitoring

    If you have never used Cloud Monitoring, then on your first access of Monitoring in the Google Cloud Console, a Workspace is automatically created and your project is associated with that Workspace. Otherwise, if your project isn't associated with a Workspace, then a dialog appears and you can either create a Workspace or add your project to an existing Workspace. We recommend that you create a Workspace. After you make your selection, click Add.

  2. In the Monitoring navigation pane, select Alerting, and then select Create policy.
  3. Click Add condition:
    1. The settings in the Target pane specify the resource and metric to be monitored. In the Find resource type and metric field, select the resource Cloud NAT gateway or Compute Engine VM instance. Next, select a metric from the metrics list.
    2. The settings in the Configuration pane of the alerting policy determine when the alert is triggered. Most fields in this pane are populated with default values. For more information about the fields in the pane, see Configuration in the Alerting policies documentation.
    3. Click Add.
  4. To advance to the notifications section, click Next.
  5. Optional: To add notifications to your alerting policy, click Notification channels. In the dialog, select one or more notification channels from the menu, and then click OK.

    If a notification channel that you want to add isn't listed, then click Manage notification channels. You are taken to the Notification channels page in a new browser tab. From this page, you can update the configured notification channels. After you have completed your updates, return to the original tab, click Refresh, and then select the notification channels to add to the alerting policy.

  6. To advance to the documentation section, click Next.
  7. Click Name and enter a name for the alerting policy.
  8. Optional: Click Documentation, and then add any information that you want included in a notification message.
  9. Click Save.
For more information, see Alerting policies.

Metric reporting frequency and retention

Metrics for Cloud NAT are batched and exported to Monitoring every one (1) minute. Monitoring data is retained for six (6) weeks. The dashboard provides data analysis in default intervals of 1H (one hour), 6H (six hours), 1D (one day), 1W (one week), and 6W (six weeks). You can manually request analysis in any interval from 6W to 1 minute.

Using VM instance metrics

The following metric type names must be prefixed with compute.googleapis.com/nat/. That prefix has been omitted from the entries in the table.

metric.type Type Description
sent_bytes_count COUNTER Count of bytes sent (VM -> internet) over the NAT gateway.
sent_packets_count COUNTER Count of packets sent (VM -> internet) over the NAT gateway.
received_bytes_count COUNTER Count of bytes received (internet -> VM) over the NAT gateway.
received_packets_count COUNTER Count of packets received (internet -> VM) over the NAT gateway.
new_connections_count COUNTER Count of new connections created over the NAT gateway.
open_connections GAUGE Number of connections currently open on the NAT gateway.
closed_connections_count COUNTER Count of connections closed over the NAT gateway.
dropped_sent_packets_count COUNTER Count of sent packets dropped by the NAT gateway (because of port exhaustion or because of port conflict due to endpoint independent mapping).
dropped_received_packets_count COUNTER Count of received packets dropped by the NAT gateway because a connection was not found.
port_usage GAUGE Maximum number of connections from a VM to a single internet endpoint (IP:port).
allocated_ports GAUGE Number of ports allocated to a VM by the NAT gateway.

VM instance filtering dimensions

label_key Type Description
project_id STRING The project ID of the VM instance.
instance_id STRING The ID of the VM instance.
zone STRING The zone of the VM instance.
nat_project_number STRING The project number to which the NAT gateway belongs.
router_id STRING The Cloud Router ID to which the NAT gateway belongs.
nat_gateway_name STRING The name of the NAT gateway.
nat_ip STRING The NAT IP address allocated to the NAT gateway. Valid for metrics allocated_ports.
ip_protocol STRING The protocol of the connection. Can be TCP, UDP or ICMP. Valid for all metrics except allocated_ports.
reason STRING

The reason for the packet drop. Possible values are:

  • OUT_OF_RESOURCES, if Cloud NAT runs out of NAT IP addresses or ports.
  • ENDPOINT_INDEPENDENT_CONFLICT, when an endpoint independent conflict arises.

Valid for metrics dropped_sent_packets_count.

Using NAT gateway metrics

The following metric type names must be prefixed with router.googleapis.com/nat/. That prefix has been omitted from the entries in the table.

metric.type Type Description
nat_allocation_failed BOOL Indicates `true` if there is a failure in allocating NAT IP addresses to any VM in the NAT gateway, else `false`. A common reason is having insufficient NAT IP addresses in the gateway.
sent_bytes_count COUNTER Count of bytes sent (VMs -> internet) over the NAT gateway.
sent_packets_count COUNTER Count of packets sent (VMs -> internet) over the NAT gateway.
received_bytes_count COUNTER Count of bytes received (internet -> VMs) over the NAT gateway.
received_packets_count COUNTER Count of packets received (internet -> VMs) over the NAT gateway.
new_connections_count COUNTER Count of new connections created over the NAT gateway.
open_connections GAUGE Number of connections currently open on the NAT gateway.
closed_connections_count COUNTER Count of connections closed over the NAT gateway.
dropped_sent_packets_count COUNTER Count of sent packets dropped by the NAT gateway (because of port exhaustion, or port conflict due to endpoint independent mapping).
dropped_received_packets_count COUNTER Count of received packets dropped by the NAT gateway because a connection was not found.
port_usage GAUGE The port usage for a single VM is the maximum number of connections from the VM to a single internet endpoint (IP:port). This field shows the highest port usage among all the VMs for a NAT gateway.
allocated_ports GAUGE Number of ports allocated to all VMs by the NAT gateway.

Gateway filtering dimensions

label_key Type Description
project_id STRING The project ID of the gateway.
region STRING The region of the gateway.
router_id STRING The Cloud Router ID to which the NAT gateway belongs.
gateway_name STRING The name of the NAT gateway.
ip_protocol STRING The protocol of the connection. Can be TCP, UDP or ICMP. Valid for all metrics except allocated_ports.
nat_ip STRING The NAT IP address allocated to the NAT gateway. Valid for metrics allocated_ports.
reason STRING

The reason for the packet drop. Possible values are:

  • OUT_OF_RESOURCES, if Cloud NAT runs out of NAT IP addresses or ports.
  • ENDPOINT_INDEPENDENT_CONFLICT, when an endpoint independent conflict arises.

Valid for metrics dropped_sent_packets_count.

Using the API

You can construct arbitrary queries over the Cloud NAT metrics by using the Monitoring API's v3 projects.timeSeries.list request.

API example

Getting allocated ports of a VM, with 1m aggregates, timeSeries.list parameters:

  • name: projects/PROJECT_ID
  • Filter: resource.type = "gce_instance" AND metric.name="compute.googleapis.com/nat/allocated_ports" AND metric.labels.nat_gateway_name="nat-1-237227-1569344091-5"
  • interval.start_time: 2019-09-24T16:58:53Z
  • interval.end_time: 2019-09-24T16:58:53Z
  • aggregation.alignmentPeriod: 60s
  • aggregation.crossSeriesReducer: REDUCE_SUM
  • aggregation.perSeriesAligner: ALIGN_SUM
"timeSeries": [
    {
      "resource": {
        "labels": {
          "instance_id": "1406292833167995938",
          "project_id": "PROJECT_ID",
          "zone": "us-west1-c"
        },
        "type": "gce_instance"
      },
      "metric": {
        "labels": {
          "nat_ip": "70.32.157.11",
          "nat_gateway_name": "nat-1-237227-1569344091-5",
          "nat_project_number": "PROJECT_NUMBER",
          "router_id": "1380055590921303155"
        },
        "type": "compute.googleapis.com/nat/allocated_ports"
      },
      "metricKind": "GAUGE",
      "points": [
        {
          "interval": {
            "endTime": "2019-09-24T16:58:53.699Z",
            "startTime": "2019-09-24T16:58:53.699Z"
          },
          "value": {
            "int64Value": "64"
          }
        }
      ],
      "valueType": "INT64"
    },

Monitoring pricing

There is no cost for Cloud NAT monitoring.

What's next