Diagnose an access denial event using the VPC Service Controls violation analyzer

Preview — VPC Service Controls violation analyzer

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This page describes how to use the VPC Service Controls violation analyzer to understand and diagnose access denials from service perimeters in your organization.

VPC Service Controls generates a troubleshooting token when VPC Service Controls denies an access request. The violation analyzer lets you diagnose the access denial using this troubleshooting token. You can find this troubleshooting token in Cloud Audit Logs. VPC Service Controls logs all the information about an access denial event in Cloud Audit Logs, including the troubleshooting token.

You can also use the violation analyzer to diagnose access denials from the dry run configuration of a service perimeter.

For information about diagnosing access denials using the VPC Service Controls troubleshooter, see Diagnose issues by using the VPC Service Controls troubleshooter.

Before you begin

• Enable the Policy Troubleshooter API.

  Enable the API

• To understand the device policies in an access level and retrieve the device context details, make sure that you have the required permissions in Google Workspace to view the device details. Without these permissions, if you troubleshoot a denial event involving an access level with device attribute based conditions, the troubleshooting result can differ.

  To get these permissions, make sure that you have any one of the following Google Workspace roles:

  • Super Admin, Services Admin, or Mobile Admin role.

  • A custom administrator role that contains the Manage Devices and Settings privilege. You can find this privilege under Services > Mobile Device Management.

  For more information about assigning roles, see Assign specific admin roles.

  You can use the violation analyzer without these permissions in Google Workspace. However, the troubleshooting result might differ as specified earlier.

Required roles

To get the permissions that you need to use the violation analyzer, ask your administrator to grant you the following IAM roles:

• To diagnose an access denial event using the violation analyzer: Access Context Manager Reader (roles/accesscontextmanager.policyReader) on your organization-level access policy
• To fetch the troubleshooting token from Cloud Audit Logs: Logs Viewer (roles/logging.viewer) on the projects that have VPC Service Controls audit logs

For more information about granting roles, see Manage access to projects, folders, and organizations.

These predefined roles contain the permissions required to use the violation analyzer. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to use the violation analyzer:

• To diagnose an access denial event using the violation analyzer:
  • accesscontextmanager.accessLevels.list on your organization-level access policy
  • accesscontextmanager.policies.get on your organization-level access policy
  • accesscontextmanager.servicePerimeters.list on your organization-level access policy

You might also be able to get these permissions with custom roles or other predefined roles.

Troubleshoot an access denial event

When VPC Service Controls denies an access request, VPC Service Controls generates a unique ID and logs an encrypted troubleshooting token in Cloud Audit Logs. When you use the Google Cloud CLI, VPC Service Controls returns an access denial event's unique ID in the error. After the access denial event, you can search and find the troubleshooting token in Cloud Audit Logs using the unique ID.

You need the troubleshooting token to diagnose an access denial event using the violation analyzer.

Note: If you set up the VPC Service Controls violation dashboard in your organization, the violation dashboard displays access denials by your service perimeters, including the troubleshooting tokens. To diagnose an access denial event, click the troubleshooting token of the access denial event listed in the Violations table of the violation dashboard. VPC Service Controls opens the violation analyzer and displays the troubleshooting result of the access denial.

For more information, see Set up and view the violation dashboard (Preview).

Get the troubleshooting token

1. In the Google Cloud console, go to the Logs Explorer page.

   Go to Logs Explorer

2. On the Logs Explorer page, select the project scope that the access denial event belongs to.

3. Use the log filters to find the log entry of the access denial event.

   You can also use the unique ID to view the log entry. To search and display audit logs using the unique ID, enter the following query into the query-editor field:

   log_id("cloudaudit.googleapis.com/policy")
   severity=ERROR
   resource.type="audited_resource"
   protoPayload.metadata."@type"="type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   protoPayload.metadata.vpcServiceControlsUniqueId="UNIQUE_ID"
   

   Replace UNIQUE_ID with the unique ID of the access denial event.

4. Click Run query. This query displays the VPC Service Controls audit logs for the access denial event.

5. To expand the audit logs, in the Query results pane, click the expander arrow.

6. Expand the protoPayload > metadata section in the log entry.

7. Copy the vpcServiceControlsTroubleshootToken value from the log entry.

Troubleshoot using the token

1. In the Google Cloud console, go to the VPC Service Controls page.

   Go to VPC Service Controls

   If you are prompted, select your organization. You can access the VPC Service Controls page only at the organization level.

2. On the VPC Service Controls page, click Troubleshoot.

3. On the Troubleshoot violation page, in the Troubleshooting token (or unique ID) field, enter the troubleshooting token of the access denial event.

   Note: If you want to use the troubleshooter to diagnose the access denial event, enter the unique ID of the event in the Troubleshooting token (or unique ID) field.

4. Click Continue.

The violation analyzer evaluates the audit logs of the access denial event and displays the troubleshooting result.

Understand the troubleshooting result

Before you read the troubleshooting result of an access denial event, make sure that you refer to the following considerations.

Sensitive information redaction

To protect sensitive data, the violation analyzer redacts the following information in the troubleshooting result:

• IP address: When an access request originates from a Google Cloud service inside an internal production network, the violation analyzer redacts the IP address of the access request as private.

• Network information: The violation analyzer redacts the network information of the access request as redacted_network, except in the following scenarios:

  • When you are from the same organization as the network.

  • When you have the necessary permission to view the network information.

• Principal: The violation analyzer redacts the email address of a principal with ... (for example, cl...o@gm...m), except in the following scenarios:

  • When you are from the same organization as the access-denied principal.

  • When the access-denied principal is a service agent or service account.

  Some Google Cloud services don't collect identity information. For example, the legacy App Engine API doesn't collect the caller identities. When the violation analyzer observes that the principal information is missing in the logs, the troubleshooting result displays the principal as no information available.

Evaluation status

The violation analyzer evaluates an access denial event against all the perimeter components and assigns an evaluation status for each component.

The violation analyzer might display the following evaluation statuses in the troubleshooting result:

Status	Description
Granted	This status indicates that the perimeter component allows the evaluated access request.
Denied	This status indicates that the perimeter component denies the evaluated access request.
Not applicable	This status indicates that the perimeter component doesn't restrict the resource or service from the evaluated access request or doesn't enforce the VPC accessible services feature.

View the troubleshooting result

The troubleshooting result page provides a detailed assessment of an access denial event. This result presents the assessment of the event at the specific point in time when you requested the violation analyzer to diagnose the event. The troubleshooting result page categorizes the assessment information under different sections.

The troubleshooting result of an access denial event can have the following sections:

• Violation details

• Violation evaluation

• Restricted resources

• Restricted services

• Ingress

• Egress

• VPC accessible services

To view the assessment of a specific perimeter component, select the perimeter component from the list or click the expander arrow next to the perimeter component. For example, to view the troubleshooting assessment for an egress rule, select the egress rule or click the expander arrow next to the egress rule.

Note: When you click an access level in the troubleshooting result, the violation analyzer opens a pane that displays the assessment of the access denial event against the access level.

Violation details

The Violation details section lists the following information about the access denial event:

• The time of the access denial event.

• The identity of the principal that requested access.

• The service for which the principal requested access.

• The service method for which the principal requested access.

• The IP address of the principal that requested access. This IP address is the same as the caller_ip value of the access denial event's log entry in Cloud Audit Logs. For more information, see IP address of the caller in audit logs.

• The troubleshooting token of the access denial event.

• The details of the involved device and region. To view this information, click View additional details.

Violation evaluation

The Violation evaluation section shows the overall assessment of the access denial event. The assessment includes both the enforced and dry run mode troubleshooting results of the perimeter.

The troubleshooting results for an access denial event can vary over time if there are changes in service perimeters or access policies after VPC Service Controls logs the access denial event. This behavior is due to the fact that the violation analyzer fetches the latest information from the relevant service perimeters and access policies for assessment.

Outcome

The Outcome section shows the assessment of the access denial event against all the perimeters involved. The value can be Granted, Denied, or Not applicable.

Protected resources accessed

The Protected resources accessed section lists the perimeters with the corresponding evaluation status against the access denial event. In this section, you can view the following information:

• A list of all resources involved in this access denial event:

  • The Resources accessed column displays all involved resources protected by the perimeter.

  • When you don't have sufficient permissions to view the restricted resources, the Protected resources accessed section doesn't list the perimeter name and the Resources accessed column displays the involved project with a warning icon.

  • The Other resources accessed section lists all the other involved resources, grouped under one of the following states:

    State	Description
    Unrestricted	This state indicates that the resource is not protected by any service perimeter.
    Info denied	This state indicates that you don't have sufficient permissions to view the service perimeters protecting the resource.
    Error	This state indicates that an internal error has occurred while trying to view the service perimeters protecting the resource.

• When you select a perimeter from the list, you can view the troubleshooting result for the access denial event against the selected perimeter.

• You can view the troubleshooting results for different enforcement modes of the perimeter as well. By default, the troubleshooting result page displays the Enforced mode troubleshooting result. If you want to view the dry run mode troubleshooting result, click Dry run. For more information about the perimeter enforcement modes, see Service perimeter details and configuration.

  Since the enforced mode and dry run mode configurations of a perimeter can be different, the violation analyzer can generate different troubleshooting results for the enforced mode and dry run mode configurations.

Restricted resources

By default, the Restricted resources section displays only the resources involved in this violation and protected by the selected perimeter. To view the other resources that are protected by the selected perimeter, click See other restricted resources.

Restricted services

By default, the Restricted services section displays only the services involved in this violation and protected by the selected perimeter. To view the other services that are protected by the selected perimeter, click See other restricted services.

Ingress

The Ingress section shows the assessment of the access denial event against all the ingress rules and access levels involved. For each access request, the violation analyzer evaluates the service agents or networks and the corresponding target resources against the ingress rules and access levels.

The violation analyzer groups and displays the ingress rule assessment information based on the ingress rules and access levels. You can click each rule or access level in this section and the violation analyzer opens an expandable section that displays the target resource names assessed against the selected ingress rule or access level.

Egress

The Egress section shows the assessment of the access denial event against all the egress rules involved. The violation analyzer evaluates the source and target resource pairs of the access request against the egress rules.

The violation analyzer groups and displays the egress rule assessment information based on the egress rules. You can click each rule in this section and the violation analyzer opens an expandable section that displays the detailed assessment of the resources against the selected egress rule.

VPC accessible services

The VPC accessible services section shows the status of the services that are accessible from network endpoints inside the perimeter. These statuses correspond to the time when the access denial event occurred. If the evaluation status for a service is Denied, you can't access the service from network endpoints inside the perimeter.

For more information, see VPC accessible services.

Compare the enforced and dry run mode results

You can compare the troubleshooting results of an access denial event between the enforced and dry run modes of the selected perimeter. To compare the troubleshooting results, click Compare to dry run on the enforced mode troubleshooting result page of a perimeter.

If the dry run mode inherits the configuration from the enforced mode of the perimeter, the dry run mode also inherits the enforced mode troubleshooting result.

Note: The enforced mode and dry run mode configurations of a perimeter might not restrict the same resources and services.

Limitations

• You must use the violation analyzer only at the organization scope, and the violation analyzer is not accessible at the project scope.

• The violation analyzer fetches the latest information from the relevant service perimeters and access policies for assessment. So the troubleshooting results for an access denial event can vary over time if there are changes in service perimeters or access policies after VPC Service Controls logs the access denial event.

  • Also, if you diagnose an access denial event multiple times, the troubleshooting results might vary for each diagnosis if the access policy has changed.

• The troubleshooting result of an access denial event might differ in the following scenarios:

  • When you have defined an access level within a scoped policy and used the access level in your service perimeter.

  • When you have defined VPC network and internal IP address based conditions in an access level and used the access level in your service perimeter.

  • When you have defined device attribute based conditions in an access level and used the access level in your service perimeter, but you don't have the permissions required to view the device details.

  • When an ingress or egress rule of the service perimeter uses identity groups or third-party identities.

  • When an egress rule of the service perimeter uses the sources attribute.

  • When the access request doesn't contain any resources.

  • When you have configured a credential strength policy in the access level.

  • When an ingress or egress rule of the service perimeter uses a service permission as an API operation condition.

What's next

• Diagnose issues by using the VPC Service Controls troubleshooter
• For information about the access denial reasons, see Debugging requests blocked by VPC Service Controls.