Monitor Private Service Connect connections

Private Service Connect exposes key metrics to Cloud Monitoring that give you insights into your Private Service Connect connections.

Metrics are sent automatically to Monitoring. There, you can create custom dashboards, set up alerts, and query the metrics.

Monitor published services

You can monitor published services by using predefined dashboards or Google Cloud metrics.

View dashboards for published services

Private Service Connect provides a set of predefined dashboards that display the following metrics for a published service:

  • Connected forwarding rules
  • NAT IP addresses in use
  • Open connections
  • New connections
  • Closed connections
  • Network traffic
  • Network packets
  • Dropped sent packets
  • Dropped received packets

To view predefined dashboards from the details page of a particular Private Service Connect published service, follow these steps:

Console

  1. In the Google Cloud console, go to the Private Service Connect page.

    Go to Private Service Connect

  2. Click the Published services tab.

  3. Click an existing service.

  4. Click the Monitoring tab.

    You can change the view of the charts by using the control at the top of the page. Hovering over a point on the graph gives you details for that specific time.

Metrics for published services

The "metric type" strings in this table must be prefixed with compute.googleapis.com/. That prefix has been omitted from the entries in the table.

For a full list of Google Cloud metrics, see Google Cloud metrics.

Metric type Launch stage
Display name
Kind, Type, Unit
Monitored resources		 Description
Labels
private_service_connect/producer/closed_connections_count BETA
Closed connections count
DELTA, INT64, {connection}
gce_service_attachment 		Count of connections closed over a PSC Attachment ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.
psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/connected_consumer_forwarding_rules GA
Connected consumer forwarding rules
GAUGE, INT64, 1
gce_service_attachment 		Number of Consumer Forwarding Rules connected to a PSC Attachment ID. Sampled every 60 seconds. After sampling, data is not visible for up to 165 seconds.
private_service_connect/producer/dropped_received_packets_count BETA
Received packets dropped count
DELTA, INT64, {packet}
gce_service_attachment 		Count of received packets dropped by a PSC Attachment ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.
psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/dropped_sent_packets_count BETA
Sent packets dropped count
DELTA, INT64, {packet}
gce_service_attachment 		Count of sent packets dropped by a PSC Attachment ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.
psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/new_connections_count BETA
New connections count
DELTA, INT64, {connection}
gce_service_attachment 		Count of new connections created over a PSC Attachment ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.
psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/open_connections BETA
Open connections
GAUGE, INT64, {connection}
gce_service_attachment 		Number of connections currently open on a PSC Attachment ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.
psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/received_bytes_count BETA
Received bytes count
DELTA, INT64, By
gce_service_attachment 		Count of bytes received (PSC -> Service) over a PSC Attachment ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.
psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/received_packets_count BETA
Received packets count
DELTA, INT64, {packet}
gce_service_attachment 		Count of packets received (PSC -> Service) over a PSC Attachment ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.
psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/sent_bytes_count BETA
Sent bytes count
DELTA, INT64, By
gce_service_attachment 		Count of bytes sent (Service -> PSC) over a PSC Attachment ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.
psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/sent_packets_count BETA
Sent packets count
DELTA, INT64, {packet}
gce_service_attachment 		Count of packets sent (Service -> PSC) over a PSC Attachment ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.
psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/used_nat_ip_addresses GA
Used nat ip addresses
GAUGE, INT64, 1
gce_service_attachment 		IP usage of the monitored service attachment. Sampled every 60 seconds. After sampling, data is not visible for up to 165 seconds.

Monitor endpoints and backends

You can monitor endpoints that connect to published services by using predefined dashboards or Google Cloud metrics. You can monitor backends that connect to published services by using Google Cloud metrics.

View dashboards for endpoints

Private Service Connect provides a set of predefined dashboards that display the following metrics for endpoints that connect to published services:

  • Open connections
  • New connections
  • Closed connections
  • Network traffic
  • Network packets
  • Dropped sent packets
  • Dropped received packets

To view predefined dashboards from the details page of a particular Private Service Connect endpoint, follow these steps:

Console

  1. In the Google Cloud console, go to the Private Service Connect page.

    Go to Private Service Connect

  2. Click the Connected endpoints tab.

  3. Click an endpoint that connects to a published service.

  4. Click the Monitoring tab.

    You can change the view of the charts by using the control at the top of the page. Hovering over a point on the graph gives you details for that specific time.

Metrics for endpoints and backends

Both Private Service Connect endpoints and backends are monitored as Private Service Connect Endpoint resources.

The metrics in this table are not generated for endpoints or backends that connect to Google APIs.

The "metric type" strings in this table must be prefixed with compute.googleapis.com/. That prefix has been omitted from the entries in the table.

For a full list of Google Cloud metrics, see Google Cloud metrics.

Metric type Launch stage
Display name
Kind, Type, Unit
Monitored resources		 Description
Labels
private_service_connect/consumer/closed_connections_count BETA
Closed connections count
DELTA, INT64, {connection}
compute.googleapis.com/PrivateServiceConnectEndpoint 		Count of TCP/UDP connections closed over a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.
private_service_connect/consumer/dropped_received_packets_count BETA
Received packets dropped count
DELTA, INT64, {packet}
compute.googleapis.com/PrivateServiceConnectEndpoint 		Count of received packets dropped by a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.
private_service_connect/consumer/dropped_sent_packets_count BETA
Sent packets dropped count
DELTA, INT64, {packet}
compute.googleapis.com/PrivateServiceConnectEndpoint 		Count of sent packets dropped by a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.
private_service_connect/consumer/new_connections_count BETA
New connections count
DELTA, INT64, {connection}
compute.googleapis.com/PrivateServiceConnectEndpoint 		Count of new TCP/UDP connections created over a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.
private_service_connect/consumer/open_connections BETA
Open connections
GAUGE, INT64, {connection}
compute.googleapis.com/PrivateServiceConnectEndpoint 		Number of TCP/UDP connections currently open on a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.
private_service_connect/consumer/received_bytes_count BETA
Received bytes count
DELTA, INT64, By
compute.googleapis.com/PrivateServiceConnectEndpoint 		Count of bytes received (PSC -> Clients) over a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.
private_service_connect/consumer/received_packets_count BETA
Received packets count
DELTA, INT64, {packet}
compute.googleapis.com/PrivateServiceConnectEndpoint 		Count of packets received (PSC -> Clients) over a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.
private_service_connect/consumer/sent_bytes_count BETA
Sent bytes count
DELTA, INT64, By
compute.googleapis.com/PrivateServiceConnectEndpoint 		Count of bytes sent (Clients -> PSC) over a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.
private_service_connect/consumer/sent_packets_count BETA
Sent packets count
DELTA, INT64, {packet}
compute.googleapis.com/PrivateServiceConnectEndpoint 		Count of packets sent (Clients -> PSC) over a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 315 seconds.
ip_protocol: The protocol of the connection. Can be TCP or UDP.

Define alerting policies

To create a metrics-based alerting policy, follow these steps. Use a resource type of Service Attachment for metrics about published services. Use a resource type of Private Service Connect Endpoint for metrics about endpoints or backends.

Console

You can create alerting policies to monitor the values of metrics and to notify you when those metrics violate a condition.

  1. In the Google Cloud console, go to the  Alerting page:

    Go to Alerting

    If you use the search bar to find this page, then select the result whose subheading is Monitoring.

  2. If you haven't created your notification channels and if you want to be notified, then click Edit Notification Channels and add your notification channels. Return to the Alerting page after you add your channels.
  3. From the Alerting page, select Create policy.
  4. To select the metric, expand the Select a metric menu and then do the following:
    1. To limit the menu to relevant entries, enter the resource type into the filter bar. If there are no results after you filter the menu, then disable the Show only active resources & metrics toggle.
    2. For the Resource type, select the resource type.
    3. For the Metric category, select Private_service_connect.
    4. For the Metric, select the metric to use for this policy.
    5. Select Apply.
  5. Click Next.
  6. The settings in the Configure alert trigger page determine when the alert is triggered. Select a condition type and, if necessary, specify a threshold. For more information, see Create metric-threshold alerting policies.
  7. Click Next.
  8. Optional: To add notifications to your alerting policy, click Notification channels. In the dialog, select one or more notification channels from the menu, and then click OK.
  9. Optional: Update the Incident autoclose duration. This field determines when Monitoring closes incidents in the absence of metric data.
  10. Optional: Click Documentation, and then add any information that you want included in a notification message.
  11. Click Alert name and enter a name for the alerting policy.
  12. Click Create Policy.
For more information, see Alerting policies.

View logs

You can view logs for Private Service Connect endpoints and published services by using Cloud Logging. Cloud Logging is a fully managed service that lets you store, search, analyze, monitor, and alert on logging data and events.

  • Audit logs let you monitor Private Service Connect activity. Admin Activity audit logs are always written.
  • VPC Flow Logs let you monitor Private Service Connect traffic. You must enable VPC Flow Logs for each subnet that you want to monitor.

You can use these logs to correlate events between the service consumer and service producer. For example, if the connection status of a consumer forwarding rule changes unexpectedly, you can request that the service producer verify their logs for any service attachment deletion or update events.

Console

  1. In the Google Cloud console, go to the Logs Explorer page.

    Go to Logs Explorer

  2. If you don't see the query editor field in the Query pane, click the Show query toggle.

  3. In the query editor field, enter a query. For example, to view an endpoint's connection status change, enter the following query, replacing CONSUMER_PROJECT_ID with the consumer project ID:

    resource.type="gce_forwarding_rule"
log_name="projects/CONSUMER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event"
protoPayload.methodName="LogPscConnectionStatusUpdate"

    For more examples of queries that you can run to view common logging events, see Common logging events for endpoints.

  4. Click Run query.

For more information about querying your audit logs, see Viewing audit logs.

Common logging events for published services

The following table lists common logging events for Private Service Connect published services.

Event description Logging advanced filter
Service attachment deletion 
resource.type="audited_resource"
log_name="projects/PRODUCER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity"
resource.labels.method="compute.serviceAttachments.delete"
Service attachment enabling connection reconciliation 
resource.type="audited_resource"
log_name="projects/PRODUCER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity"
resource.labels.method="compute.serviceAttachments.patch"
protoPayload.request.reconcileConnections="true"
Service attachment rejecting a consumer project URI 
resource.type="audited_resource"
log_name="projects/PRODUCER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity"
protoPayload.request.consumerRejectLists="CONSUMER_PROJECT_ID"
VPC Flow Logs for traffic from a Private Service Connect subnet to any backend VM instance (including GKE nodes) 
resource.type="gce_subnetwork"
logName="projects/PRODUCER_PROJECT_ID/logs/compute.googleapis.com%2Fvpc_flows"
json_payload.connection.src_ip=~"PSC_SUBNET_REGEX.*"
jsonPayload.dest_instance.vm_name=~"VM_INSTANCE_PREFIX.*"

Replace the following:

  • PRODUCER_PROJECT_ID: the project ID of the service producer.
  • CONSUMER_PROJECT_ID: the project ID of the service consumer.
  • PSC_SUBNET_REGEX: a regular expression that matches a pattern in the Private Service Connect subnet. For example, replace PSC_SUBNET_REGEX with 172\.16\.[0-1] if the Private Service Connect subnet is 172.16.0.0/23.
  • VM_INSTANCE_PREFIX: the prefix of the backend VM instances.

Common logging events for endpoints

The following table lists common logging events for Private Service Connect endpoints.

Event description Logging advanced filter
Private Service Connect endpoint creation 
resource.type="gce_forwarding_rule"
log_name="projects/CONSUMER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity"
protoPayload.methodName="v1.compute.forwardingRules.insert"
"compute.forwardingRules.pscCreate"
Private Service Connect endpoint creation failure 
resource.type="gce_forwarding_rule"
log_name="projects/CONSUMER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity"
protoPayload.methodName="v1.compute.forwardingRules.insert"
"compute.forwardingRules.pscCreate"
severity>=ERROR
Private Service Connect endpoint connection status change 
resource.type="gce_forwarding_rule"
log_name="projects/CONSUMER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event"
protoPayload.methodName="LogPscConnectionStatusUpdate"
Rejected Private Service Connect endpoint connection 
resource.type="gce_forwarding_rule"
log_name="projects/CONSUMER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event"
protoPayload.methodName="LogPscConnectionStatusUpdate"
protoPayload.metadata.pscConnectionStatus="REJECTED"
Quota PSC_INTERNAL_LB_FORWARDING_RULES exceeded 
resource.type="gce_forwarding_rule"
log_name="projects/CONSUMER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity"
protoPayload.methodName="v1.compute.forwardingRules.insert"
"QUOTA_EXCEEDED"
severity=ERROR
VPC Flow Logs for traffic from a VM instance to a Private Service Connect endpoint 
resource.type="gce_subnetwork"
logName="projects/CONSUMER_PROJECT_ID/logs/compute.googleapis.com%2Fvpc_flows"
jsonPayload.connection.dest_ip="PSC_ENDPOINT_IP_ADDRESS"
jsonPayload.src_instance.vm_name="VM_INSTANCE_NAME"

Replace the following:

  • CONSUMER_PROJECT_ID: the project ID of the service consumer.
  • PSC_ENDPOINT_IP_ADDRESS: the IP address of the Private Service Connect endpoint.
  • VM_INSTANCE_NAME: the name of a source VM instance.

Known issues

Metrics not generated for cross-region global access

There is a known issue where the Private Service Connect metrics on this page might not be generated for consumers or producers in the following scenario:

  • A consumer resource exists in one region.
  • The consumer resource accesses a Private Service Connect endpoint in a different region by using global access.
  • The endpoint's region doesn't contain any consumer virtual machine (VM) instances.

Workaround

To generate metrics in this scenario, create a VM instance in the same region as the endpoint that is being accessed.

What's next