Using logging and monitoring

Logging

Cloud NAT logging allows you to log NAT connections and errors. When Cloud NAT logging is enabled, one log entry can be generated for each of the following scenarios:

  • when a network connection using NAT is created
  • when a packet is dropped because no port was available for NAT

You can opt to log both kinds of events, or just one or the other.

Created logs are sent to Cloud Logging.

Specifications

The following specifications apply to Cloud NAT logging:

  • Cloud NAT logging handles TCP and UDP traffic only.

  • Cloud NAT logging only logs dropped packets if they are egress (outbound) TCP and UDP packets. It does not log dropped incoming packets. For example, if an inbound response to an outbound request is dropped for any reason, no error is logged.

Each VM instance can only generate a certain number of log entries per unit time, proportional to its number of vCPUs. The VM can generate 50-100 log entries per second per vCPU.

  • This rate threshold affects the number of events that can be logged. Even if some events are filtered out, their occurrence counts toward the number of possible log entries. Limiting logs to just errors or just NAT translation connections does not necessarily increase the number of viewed log entries. For example, if you choose to log only successful connections, periods of excessive failed connection attempts and NAT errors can still restrict the number of successful connection log entries.

  • Cloud NAT logging does not log every single packet. Even if the VM's rate threshold has not been reached, there are conditions that can cause events to be omitted from the log. You should rely on the presence of entries in Cloud NAT logging to make informed decisions, but you should not assume that the absence of entries means that an event didn't happen.

Configuring logging

Enabling logging

If logging is enabled, all collected logs are sent to Logging by default. You can filter these so that only certain logs are sent.

You can also specify these values when you create a NAT gateway or by editing one after it has been created. The following directions show how to enable logging for an existing NAT gateway.

Console

  1. Go to the Cloud NAT page in the Google Cloud Console.
    Go to the Cloud NAT page
  2. Click on your NAT gateway.
  3. Click Edit.
  4. Click Logging, minimum ports, timeout to open that section.
  5. Under Stackdriver logging, select one of the following:
    • No logging — disables logging
    • Translation and errors — sends all logs to Logging
    • Translation only — sends a log only when a connection is created. Does not log dropped packets.
    • Errors only — sends a log when a packet is dropped because no port was available. Does not log new connections.
  6. Click Save.

gcloud

The following commands enable logging for an existing NAT gateway. In each command, replace [NAT_GATEWAY] with the name of the NAT gateway, [ROUTER_NAME] with the name of the Cloud Router on which it is hosted, and [REGION] with the region of the Cloud Router.

To log NAT translation events and errors:

gcloud compute routers nats update [NAT_GATEWAY] \
    --router=[ROUTER_NAME] \
    --region=[REGION] \
    --enable-logging

To just log NAT translation events:

gcloud compute routers nats update [NAT_GATEWAY] \
    --router=[ROUTER_NAME] \
    --region=[REGION] \
    --enable-logging \
    --log-filter=TRANSLATIONS_ONLY

To just log errors:

gcloud compute routers nats update [NAT_GATEWAY] \
    --router=[ROUTER_NAME] \
    --region=[REGION] \
    --enable-logging \
    --log-filter=ERRORS_ONLY

Clearing log filters

If you have a filter set, you can clear it. Clearing a log filter means that both NAT translation events and errors will be logged, provided that logging is enabled.

Console

  1. Go to the Cloud NAT page in the Google Cloud Console.
    Go to the Cloud NAT page
  2. Click on your NAT gateway.
  3. Click Edit.
  4. Click Logging, minimum ports, timeout to open that section.
  5. Under Stackdriver logging, select the following:
    • Translation and errors — sends all logs to Google Cloud's operations suite
  6. Click Save.

gcloud

gcloud compute routers nats update [NAT_GATEWAY] \
    --router=[ROUTER_NAME] \
    --region=[REGION] \
    --log-filter=ALL

where

  • [NAT_GATEWAY] is the name of the NAT gateway.
  • [ROUTER_NAME] is the name of the Cloud Router that hosts the NAT gateway.
  • [REGION] is the Cloud Router's region.
  • --log-filter=ALL sets the log filter to accept all logs.

Disabling logging

Console

  1. Go to the Cloud NAT page in the Google Cloud Console.
    Go to the Cloud NAT page
  2. Click on your NAT gateway.
  3. Click Edit.
  4. Click Logging, minimum ports, timeout to open that section.
  5. Under Stackdriver logging, select the following:
    • No logging — disables logging
  6. Click Save.

gcloud

gcloud compute routers nats update [NAT_GATEWAY] \
    --router=[ROUTER_NAME] \
    --region=[REGION] \
    --no-enable-logging

where

  • [NAT_GATEWAY] is the name of the NAT gateway.
  • [ROUTER_NAME] is the name of the Cloud Router that hosts the NAT gateway.
  • [REGION] is the Cloud Router's region.

Determining logging status

To determine the status for logging, do the following:

Console

  1. Go to the Cloud NAT page in the Google Cloud Console.
    Go to the Cloud NAT page
  2. Click on your NAT gateway.
  3. In the Logging, minimum ports, timeout section, inspect Stackdriver logging.

gcloud

gcloud compute routers nats describe [NAT_GATEWAY] \
    --router=[ROUTER_NAME] \
    --region=[REGION]

where

  • [NAT_GATEWAY] is the name of the NAT gateway.
  • [ROUTER_NAME] is the name of the Cloud Router that hosts the NAT gateway.
  • [REGION] is the Cloud Router's region.

Viewing logs

Console

To view logs, go to the Logs Viewer.

  • To see all NAT logs, in the first pull-down menu select Cloud NAT Gateway.
  • To see logs for just one region, in the first pull-down menu select Cloud NAT Gateway, then slide the cursor right to select a region.
  • To see logs for just one gateway, in the first pull-down menu select Cloud NAT Gateway, then slide the cursor right to select a region, then slide the cursor right again to select a single gateway.

Alternatively, go to the Logs page and paste the following into the Filter by label or text search field.

resource.type="nat_gateway"
logName="projects/{#project_id}/logs/compute.googleapis.com%2Fnat_flows"

gcloud

  • To see the most recent NAT logs, run the following command:

    gcloud logging read 'resource.type=nat_gateway' \
        --limit=10 \
        --format=json
    

    where

    • resource.type=nat_gateway limits the output to your NAT gateways.
    • --limit=10 limits the output to 10 entries. You can put a different value to see more or fewer entries, or omit it entirely to see a continuous scroll of logs.
    • --format=json displays the output in JSON format.

For more options, see Reading log entries

You can configure export of logs based metrics for resource logs.

What is logged

Cloud NAT log entries contain information useful for monitoring and debugging your NAT traffic. Log entries contain the following types of information:

  • General information shown in most Google Cloud logs, such as severity, project ID, project number, timestamp, and so on.
  • Specific information related to Cloud NAT. Some log fields contain entries that are themselves multiple fields. These entries and field descriptions are shown in tables below.

Log fields

Field Value Meaning
connection object(NatIpConnection) 7-Tuple describing the source VM IP and port, source NAT IP and port, destination IP and port, and IP protocol of this connection.
allocation_status enum Indicate whether this connection was successfully allocated or dropped. One of OK or DROPPED.
gateway_identifiers object(NatGateway) The NAT gateway configuration that the connection used.
endpoint object(InstanceDetails) VM instance details. Note that in a Shared VPC configuration, project_id corresponds to the service project.
vpc object(VpcDetails) VPC network details. Note that in a Shared VPC configuration, project_id corresponds to that of the host project.
destination object(DestinationDetails) Details of the destination of the connection.

NatIpConnection field format

Field Type Description
src_ip string Source IP address
src_port int32 Source port
nat_ip string NAT IP address
nat_port int32 NAT assigned port
dest_ip string Destination IP address
dest_port int32 Destination port
protocol int32 IANA protocol number

NatGateway field format

Field Type Description
gateway_name string Name of the NAT gateway
router_name string Cloud Router associated with the NAT gateway
region string Region of the Cloud Router

InstanceDetails field format

Field Type Description
project_id string ID of the project containing the VM
vm_name string Instance name of the VM
region string Region of the Cloud Router
zone string Zone of the VM

VpcDetails field format

Field Type Description
project_id string ID of the project containing the network
vpc_name string Network on which the VM is operating
subnetwork_name string Subnet on which the VM is operating

DestinationDetails field format

Field Type Description
geo_location object(GeographicDetails) If the destination of the connection was external to GCP, this field is populated with available location metadata.
instance object(InstanceDetails) If the destination of the connection is an instance within the same project as the source, this field is populated with VM instance details.

GeographicDetails field format

Field Type Description
continent string Continent for external endpoints
country string Country for external endpoints
region string Region for external endpoints
city string City for external endpoints
asn string The autonomous system number (ASN) of the external network to which this endpoint belongs.

Examples

Example 1: NAT-ed TCP connection record from a VM instance in a Shared VPC network going to an external server in France.

{
insertId: "1the8juf6vab1t"
jsonPayload: {
       connection: {
            Src_ip: "10.0.0.1"
            Src_port: 45047
            Nat_ip: "203.0.113.17"
            Nat_port: 34889
            dest_ip : "198.51.100.142"
            Dest_port: 80
            Protocol: "tcp"
       }
       allocation_status: "OK"
       Gateway_identifiers: {
             Gateway_name: "my-nat-1"
             router_name: "my-router-1"
             Region: "europe-west1"
       }
       Endpoint: {
             Project_id: "service-project-1"
             Vm_name: "vm-1"
             Region: "europe-west1"
             Zone: "europe-west1-b"
       }
       Vpc: {
             Project_id: "host-project"
             Vpc_name: "network-1"
             Subnetwork_name: "subnetwork-1"
       }
       Destination: {
             Geo_location: {
                   Continent: "Europe"
                   Country: "France"
                   Region: "Nouvelle-Aquitaine"
                   City: "Bordeaux"
             }
       }
}
logName: "projects/host-project/logs/compute.googleapis.com%2Fnat_flows"
receiveTimestamp: "2018-06-28T10:46:08.123456789Z"
resource: {
      labels: {
            region: "europe-west1-d"
            project_id: "host-project"
            router_id: "987654321123456"
            gateway_name: "my-nat-1"
      }
      type: "nat_gateway"
}
labels: {
      nat.googleapis.com/instance_name: "vm-1"
      nat.googleapis.com/instance_zone: "europe-west1-b"
      nat.googleapis.com/nat_ip: "203.0.113.17"
      nat.googleapis.com/network_name: "network-1"
      nat.googleapis.com/router_name: "my-router-1"
      nat.googleapis.com/subnetwork_name: "subnetwork-1"
}
timestamp: "2018-06-28T10:46:00.602240572Z"
}

Example 2: Record for a packet dropped because there were no available ports. Sending VM was trying to reach the external IP address of another VM in the same project.

{
insertId: "1the8juf6vab1l"
jsonPayload: {
       connection: {
            Src_ip: "10.0.128.1"
            Src_port: 45047
            dest_ip : "192.0.2.87"
            Dest_port: 80
            Protocol: "tcp"
       }
       allocation_status: "DROPPED"
       Gateway_identifiers: {
             Gateway_name: "my-nat-2"
             Cloud_router: "my-router-1"
             Region: "europe-west1"
       }
       Endpoint: {
             Project_id: "service-project-1"
             Vm_name: "vm-1"
             Region: "europe-west1"
             Zone: "europe-west1-b"
       }
       Vpc: {
             Project_id: "host-project"
             Vpc_name: "network-1"
             Subnetwork_name: "subnetwork-1"
       }
       Destination: {
             Instance: {
                   Project_id: "service-project-1"
                   Vm_name: "vm-2"
                   Region: "asia-east1"
                   Zone: "asia-east1-b"
             }
       }
}
logName: "projects/host-project/logs/compute.googleapis.com%2Fnat_flows"
receiveTimestamp: "2018-06-28T10:46:09.123456789Z"
resource: {
       labels: {
              region: "europe-west1-d"
              project_id: "host-project"
              router_id: "987654321123456"
              gateway_name: "my-nat-2"
       }
       type: "nat_gateway"
}
timestamp: "2018-06-28T10:46:01.602240572Z"
}

Logging pricing

See Logging pricing.

Monitoring

Cloud NAT exposes key metrics to Cloud Monitoring that give you insights into your fleet's usage of NAT gateways.

Metrics are sent automatically to Cloud Monitoring. There, you can create custom dashboards, set up alerts, and query the metrics.

IAM

For Shared VPC users with VMs and NAT gateways defined in different projects, access to the VM level metrics requires the roles/monitoring.viewer IAM role for the project of each VM. For the NAT gateway resource, access to the gateway metrics requires the roles/monitoring.viewer IAM role for the project containing the gateway.

Viewing predefined dashboards

Cloud NAT provides a set of predefined dashboards that display activity across your gateway:

  • Open connections
  • Egress data processed
  • Ingress data processed
  • Port utilization
  • NAT allocation errors

You can view predefined dashboards from the details page of a particular gateway:

  1. Go to the Cloud NAT page in the Google Cloud Console.
    Go to the Cloud NAT page
  2. Click an existing gateway.
  3. Click the Monitoring tab.

Scroll down to see all the dashboards. You can change the view from 1 hour to 30 days by using the control at the top of the page. Hovering over a point on graph gives you details for that specific time.

Defining custom dashboards

You can create custom Cloud Monitoring dashboards over Cloud NAT metrics:

  1. Go to Monitoring in the Google Cloud Console.
    Go to Monitoring
  2. Select Dashboards > Create Dashboard.
  3. Click on Add Chart.
  4. Give the chart a title.
  5. Specify the resource type and metric. You can find Cloud NAT metrics for a VM by searching on the compute.googleapis.com/nat/ prefix. For metrics for a NAT gateway, use the prefix router.googleapis.com/nat/.
    See VM instance metrics and NAT Gateway metrics below for lists of metrics and filtering dimensions.
  6. Specify any filtering dimensions.
  7. Click Save.

Defining alerting policies

You can create alerting policies to monitor the values of metrics and to notify you when those metrics violate a condition. The general steps for creating an alerting policy that monitors the Cloud NAT gateway or GCE VM Instance resource(s) are listed below:

  1. In the Google Cloud Console, go to Monitoring or use the following button:
    Go to Monitoring
  2. In the Monitoring navigation pane, select Alerting and then select Create Policy.
  3. Enter a name for the alerting policy.
  4. Click Add Condition:
    1. The settings in the Target pane specify the resource and metric to be monitored. Click the text box to enable a menu and then select the resource Cloud NAT gateway or GCE VM Instance. Next, select a metric from the metrics list.
    2. The settings in the Configuration pane of the alerting policy determine when the alert is triggered. Most fields in this pane are populated with default values. For more information on the fields in the pane, see Configuration in the alerting policy documentation.
    3. Click Add.
  5. (Optional) Click Add Notification Channel and enter your notification channel information.
  6. (Optional) Click Documentation and add any information that you want included in a notification message.
  7. Click Save.
For more information, see Alerting policies.

Metric reporting frequency and retention

Metrics for Cloud NAT are batched and exported to Monitoring every one (1) minute. Monitoring data is retained for six (6) weeks. The dashboard provides data analysis in default intervals of 1H (one hour), 6H (six hours), 1D (one day), 1W (one week), and 6W (six weeks). You can manually request analysis in any interval from 6W to 1 minute.

VM instance metrics

The following metric type names must be prefixed with compute.googleapis.com/nat/. That prefix has been omitted from the entries in the table.

metric.type Type Description
sent_bytes_count COUNTER Count of bytes sent (VM -> Internet) over the NAT gateway.
sent_packets_count COUNTER Count of packets sent (VM -> Internet) over the NAT gateway.
received_bytes_count COUNTER Count of bytes received (Internet -> VM) via the NAT gateway.
received_packets_count COUNTER Count of packets received (Internet -> VM) via the NAT gateway.
new_connections_count COUNTER Count of new connections created over the NAT gateway.
open_connections GAUGE Number of connections currently open on the NAT gateway.
closed_connections_count COUNTER Count of connections closed over the NAT gateway.
dropped_sent_packets_count COUNTER Count of sent packets dropped by the NAT gateway (because of port exhaustion, or port conflict due to endpoint independent mapping).
dropped_received_packets_count COUNTER Count of received packets dropped by the NAT gateway because a connection was not found.
port_usage GAUGE Maximum number of connections from a VM to a single internet endpoint (IP:port).
allocated_ports GAUGE Number of ports allocated to a VM by the NAT gateway.

VM instance filtering dimensions

label_key Type Description
project_id STRING The Project ID of the VM instance.
instance_id STRING The ID of the VM instance.
zone STRING The zone of the VM instance.
nat_project_number STRING The project number to which the NAT gateway belongs.
router_id STRING The Cloud Router ID to which the NAT gateway belongs.
nat_gateway_name STRING The name of the NAT gateway.
nat_ip STRING The NAT IP allocated to the NAT gateway. Valid for metrics allocated_ports.
ip_protocol STRING The protocol of the connection. Can be TCP, UDP or ICMP. Valid for all metrics except allocated_ports.
reason STRING The reason for the packet drop. Possible values are: Valid for metrics dropped_sent_packets_count.

NAT gateway metrics

The following metric type names must be prefixed with router.googleapis.com/nat/. That prefix has been omitted from the entries in the table.

metric_type Type Description
nat_allocation_failed BOOL Indicates true if there is a failure in allocating NAT IPs to any VM in the NAT gateway, else false. A common reason is having insufficient NAT IPs in the gateway.
sent_bytes_count COUNTER Count of bytes sent (VMs -> Internet) over the NAT gateway.
sent_packets_count COUNTER Count of packets sent (VMs -> Internet) over the NAT gateway.
received_bytes_count COUNTER Count of bytes received (Internet -> VMs) via the NAT gateway.
received_packets_count COUNTER Count of packets received (Internet -> VMs) via the NAT gateway.
new_connections_count COUNTER Count of new connections created over the NAT gateway.
open_connections GAUGE Number of connections currently open on the NAT gateway.
closed_connections_count COUNTER Count of connections closed over the NAT gateway.
dropped_sent_packets_count COUNTER Count of sent packets dropped by the NAT gateway (because of port exhaustion, or port conflict due to endpoint independent mapping).
dropped_received_packets_count COUNTER Count of received packets dropped by the NAT gateway because a connection was not found.
port_usage GAUGE The port usage for a single VM is the maximum number of connections from the VM to a single internet endpoint (IP:port). This field shows the highest port usage among all the VMs for a NAT gteway.
allocated_ports GAUGE Number of ports allocated to all VMs by the NAT gateway.

Gateway filtering dimensions

label_key Type Description
project_id STRING The Project ID of the gateway.
region STRING The region of the gateway.
router_id STRING The Cloud Router ID to which the NAT gateway belongs.
gateway_name STRING The name of the NAT gateway.
ip_protocol STRING The protocol of the connection. Can be TCP, UDP or ICMP. Valid for all metrics except allocated_ports.
nat_ip STRING The NAT IP allocated to the NAT gateway. Valid for metrics allocated_ports.
reason STRING The reason for the packet drop. Possible values are: Valid for metrics dropped_sent_packets_count.

Using the API

You can construct arbitrary queries over the Cloud NAT metrics using the Monitoring API's v3 projects.timeSeries.list request.

API example

Getting allocated ports of a VM, with 1m aggregates, timeSeries.list parameters:

  • name: projects/project_id
  • Filter: resource.type = "gce_instance" AND metric.name="compute.googleapis.com/nat/allocated_ports" AND metric.labels.nat_gateway_name="nat-1-237227-1569344091-5"
  • interval.start_time: 2019-09-24T16:58:53Z
  • interval.end_time: 2019-09-24T16:58:53Z
  • aggregation.alignmentPeriod: 60s
  • aggregation.crossSeriesReducer: REDUCE_SUM
  • aggregation.perSeriesAligner: ALIGN_SUM
"timeSeries": [
    {
      "resource": {
        "labels": {
          "instance_id": "1406292833167995938",
          "project_id": "project_id",
          "zone": "us-west1-c"
        },
        "type": "gce_instance"
      },
      "metric": {
        "labels": {
          "nat_ip": "70.32.157.11",
          "nat_gateway_name": "nat-1-237227-1569344091-5",
          "nat_project_number": "project_number",
          "router_id": "1380055590921303155"
        },
        "type": "compute.googleapis.com/nat/allocated_ports"
      },
      "metricKind": "GAUGE",
      "points": [
        {
          "interval": {
            "endTime": "2019-09-24T16:58:53.699Z",
            "startTime": "2019-09-24T16:58:53.699Z"
          },
          "value": {
            "int64Value": "64"
          }
        }
      ],
      "valueType": "INT64"
    },

Monitoring pricing

There is no cost for Cloud NAT monitoring.

What's next