Troubleshooting

This page lists various issues that might be encountered when attempting to configure VPC Service Controls.

Find VPC Service Controls errors

This section details two methods for finding VPC errors in your audit logs: using an error's unique ID, or using metadata to identify errors.

Using the error's unique ID

An error generated by VPC Service Controls includes a unique ID that is used to identify relevant audit logs.

This procedure uses terms from the Logging documentation. For more information, refer to Basic log filters.

To obtain information about an error using the unique ID:

In the Cloud Console, go to the Stackdriver Logging page for the project inside the service perimeter that triggered the error.

Go to the Stackdriver Logging page
In the search-filter box, enter the error's unique ID.

Filter logs using metadata

To find errors related to VPC Service Controls, use Cloud Logging.

Console

This procedure uses terms from the Logging documentation. For more information, refer to Basic log filters.

To obtain the last 24 hours of VPC Service Controls errors in Logging:

In the Google Cloud Console, go to the Stackdriver Logging page for the project inside the service perimeter.

Go to the Stackdriver Logging page

In the search-filter box, enter the following:

protoPayload.metadata.@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"

In the resource basic selector menu, select Audited Resource.
In the time-range selector drop-down menu, select Last 24 hours.
Optionally, if you want to find the VPC Service Controls errors that have occurred during a different period of time, use the time-range selector drop-down menu to change the range.

gcloud

To obtain the last 24 hours of VPC Service Controls errors, use the following command:

gcloud logging read 'protoPayload.metadata.@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"'

By default, the read command is limited to the last 24 hours. To obtain VPC Service Controls logs for a different period of time, use one of the following commands.

To limit the logs relative to the current date:

gcloud logging read \
  'protoPayload.metadata.@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"' \
  --freshness=DURATION

Where:

DURATION is a formatted period of time. For more information on formatting, refer to the relative duration/time formats for gcloud.

For example, to obtain all VPC Service Controls errors that have occurred over the last week:

gcloud logging read \
  'protoPayload.metadata.@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"' \
  --freshness=7d

To limit the logs to a specific time frame:

gcloud logging read \
  'protoPayload.metadata.@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata" AND
  timestamp>="DATETIME" AND
  timestamp<="DATETIME"'

Where:

DATETIME is a formatted date/time string. For more information on formatting, refer to the absolute date/time formats for gcloud.

For example, to obtain all VPC Service Controls errors that have occurred between March 22, 2019 and March 26, 2019:

gcloud logging read \
  'protoPayload.metadata.@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata" AND
  timestamp>="2019-03-22T23:59:59Z" AND
  timestamp<="2019-03-26T00:00:00Z"'

Unsupported Services

For more information on products and services that are supported by VPC Service Controls, refer to the Supported products page.

Warning: While it may be possible to enable unsupported services to access the data of supported products and services, we recommend that you do not. Unexpected issues might occur when attempting to access a supported service using an unsupported service, especially within the same project.

Unsupported services may not function at all when enabled in a project protected by VPC Service Controls, especially when low-level storage services like Cloud Storage or Pub/Sub are restricted. We recommend deploying unsupported services in projects outside perimeters. To allow these services to access data in resources within a perimeter, create an access level that includes the service account for that service and apply it to perimeters as needed.

Attempting to restrict an unsupported service using the gcloud command-line tool or the Access Context Manager API will result in an error.

Cross-project access to data of supported services will be blocked by VPC Service Controls. Additionally, the restricted VIP can be used to block the ability of workloads to call unsupported services.

Shared VPC

When using Shared VPC, a service perimeter that includes projects that belong to a Shared VPC network must also include the project that hosts the network. When projects that belong to a Shared VPC network are not in the same perimeter as the host project, services may not work as expected or be blocked entirely.

Ensure that the Shared VPC network host is in the same service perimeter as projects connected to the network.

Requests between perimeters

Normally, access levels are used to allow requests from outside a service perimeter for protected resources inside a perimeter.

However, requests from a project in a perimeter for a protected resource in another perimeter will be denied, even if an access level would normally allow the request.

For example, assume Project A in Perimeter 1 requests a resource from Project B. The resource in Project B is protected by Perimeter 2. Because Project A is in a perimeter, even if an access level for Perimeter 2 would normally permit the request for the protected resource, the request is denied.

There are two ways to facilitate requests between perimeters:

Use perimeter bridges. Bridges allow two or more projects in different perimeters to make requests to any services in those projects. These requests are permitted even if the services are protected by the perimeters.
Ensure that both the requesting service and the target resource are unprotected by the perimeters. In this scenario, the operation will succeed because no services are protected.

Determine if an error is due to VPC Service Controls

Because VPC Service Controls modifies certain low-level properties of Google Cloud, it can have cascading effects across services that can be challenging to debug if you don't know what to look for.

To identify if the error is related to VPC Service Controls, check whether you have enabled VPC Service Controls and applied it to the projects and services you are attempting to use. This can be done via the gcloud command-line tool or Cloud Console.

If you are using a service (potentially indirectly, via another service) that is marked as a restricted service by VPC Service Controls in a project that is inside a service perimeter, then VPC Service Controls could be at fault.

For information about known problematic cases, refer to Known service limitations.

Usually, services will propagate error messages from their dependencies. If you encounter one of the following errors, it is indicative of a problem with VPC Service Controls.

Cloud Storage: 403: Request violates VPC Service Controls.
BigQuery: 403: VPC Service Controls: Request is prohibited by organization's policy.
Other services: 403: Request is prohibited by organization's policy.

Debugging a request blocked for unexpected reasons by VPC Service Controls

The VPC Service Controls audit log is the primary tool for debugging why a request has been blocked by VPC Service Controls.

In most cases when access has been blocked unexpectedly, consult the audit logs in the project that was the source of the request. These logs contain significant data about the requested resources and the reason why the request was denied.

For more information on viewing logs, read Viewing logs.

The following table lists the violationReason values that you may encounter when using VPC Service Controls.

violationReason Explanation

RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER The resources listed under resourceNames in the audit log record are not in the same service perimeter.

NETWORK_NOT_IN_SAME_SERVICE_PERIMETER The resources corresponding to callerNetwork and resourceNames in the audit log record are not in the same service perimeter.

violationReason	Explanation
`RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER`	The resources listed under `resourceNames` in the audit log record are not in the same service perimeter.
`NETWORK_NOT_IN_SAME_SERVICE_PERIMETER`	The resources corresponding to `callerNetwork` and `resourceNames` in the audit log record are not in the same service perimeter.
`NO_MATCHING_ACCESS_LEVEL`	The IP address corresponding to `callerIp` in the audit log record does not match any IP ranges in access levels assigned to the service perimeter. Note: If there is no caller IP address or the IP address appears to be private, this can indicate that the request was made by a Google Cloud service that is not yet integrated with VPC Service Controls and is failing as expected.

NO_MATCHING_ACCESS_LEVEL

The IP address corresponding to callerIp in the audit log record does not match any IP ranges in access levels assigned to the service perimeter.

Example Scenarios

The following examples cover scenarios that you may encounter while using VPC Service Controls.

Cloud Storage access from on-premises
BigQuery access from VM outside of project
Cross-project BigQuery query
Cloud Storage file move from VM inside perimeter
Cloud Storage file move from VM outside perimeter
BigQuery dataset copy from VM inside perimeter
Dataproc job reading from project
Unsupported service with restricted VIP
Log export to project outside perimeter
BigQuery extract to Cloud Storage

Cloud Storage access from on-premises

In this example, a request from an employee workstation (identified by callerIp) to a Cloud Storage bucket in project corp-storage is blocked by VPC Service Controls.

The request generates the following audit log record:

{
 insertId:  "222lvajc6f7"
 logName:  "projects/corp-storage/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
   principalEmail:  "someone@google.com"
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "projects/_"
   ]
   violationReason:  "NO_MATCHING_ACCESS_LEVEL"
  }
  methodName:  "google.storage.NoBillingOk"
  requestMetadata: {
   callerIp:  "b1d5:d26d:5b17:43fe:d358:586b:db59:9617"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/690885588241"
  serviceName:  "storage.googleapis.com"
  status: {
   code:  7
   details: [
    0: {
     @type:  "type.googleapis.com/google.rpc.PreconditionFailure"
     violations: [
      0: {
       type:  "VPC_SERVICE_CONTROLS"
      }
     ]
    }
   ]
   message:  "Request is prohibited by organization's policy"
  }
 }
 receiveTimestamp:  "2018-11-27T21:40:43.823209571Z"
 resource: {
  labels: {
   method:  "google.storage.NoBillingOk"
   project_id:  "corp-storage"
   service:  "storage.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-11-27T21:40:42.973784140Z"
}

The corp-storage project is included in a service perimeter. The employee workstation is not a part of any networks within that perimeter. Because the employee workstation exists outside the perimeter, the request is blocked.

BigQuery access from VM outside of project

In this example, a VM that belongs to project 458854174376 (data-collector) attempts to run a BigQuery query against a dataset in project 798816221974 (corp-resources-protected) and it is denied.

The VM uses the following query:

bq --project=corp-resources-protected query 'select count(*) from babynames.yob2000'

The query returns the following:

BigQuery error in query operation: VPC Service Controls: Request is
prohibited by organization's policy. Operation ID:
33643962-6a0f-4091-9283-bcdf7e9271f0

The following audit log record is generated:

{
 insertId:  "1ei551d2pdq"
 logName:  "projects/corp-resources-protected/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
   principalEmail:  "714877721106-compute@example.gserviceaccount.com"
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "projects/1004338142803"
   ]
   violationReason:  "NETWORK_NOT_IN_SAME_SERVICE_PERIMETER"
  }
  methodName:  "bigquery.googleapis.com/bigquery.jobs.create"
  requestMetadata: {
   callerIp:  "10.105.0.2"
   callerNetwork:  "//compute.googleapis.com/projects/ameet-dataflow/global/networks/__unknown__"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/1004338142803"
  serviceName:  "bigquery.googleapis.com"
  status: {
   code:  7
   details: [
    0: {
     @type:  "type.googleapis.com/google.rpc.PreconditionFailure"
     violations: [
      0: {
       type:  "VPC_SERVICE_CONTROLS"
      }
     ]
    }
   ]
   message:  "Request is prohibited by organization's policy"
  }
 }
 receiveTimestamp:  "2018-11-28T23:06:13.579882505Z"
 resource: {
  labels: {
   method:  "bigquery.googleapis.com/bigquery.jobs.create"
   project_id:  "corp-resources-protected"
   service:  "bigquery.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-11-28T23:06:12.799656975Z"
}

In this example, the violationReason is NETWORK_NOT_IN_SAME_SERVICE_PERIMETER. callerNetwork is included in addition to callerIp. The IP address is private and the network is provided to disambiguate it. The relevant resources at issue here are listed in two places: VpcServiceControlAuditMetadata.resourceNames and requestMetadata.callerNetwork (the project that owns the network).

The problem is that the corp-resources-protected project is inside a service perimeter, while data-collector, the project that includes the network that the VM belongs to, is not. In this case, access is denied as expected.

Cross-project BigQuery query

In this example, a VM that belongs to the perimeter-network project attempts to query the BigQuery instances of two different projects: corp-resources-protected, which is in the same service perimeter as perimeter-network, and corp-resources-public, which is not.

The VM uses the following command:

bq query --use_legacy_sql=false \
  'select count(priv.name),count(pub.name) from \
  `corp-resources-protected.babynames.yob2000` as priv, \
  `corp-resources-public.babynames.yob2000` as pub'

The query returns the following:

BigQuery error in query operation: Error processing job
'example:bqjob_r211e6f6eec928ffb_000001675c996aa8_1': VPC Service Controls:
Request is prohibited by organization's policy. Operation ID:
dc4fc177-4850-4fc5-b2e7-8c33f302149a

The following audit log record is generated:

{
 insertId:  "17kg4exd24ag"
 logName:  "projects/perimeter-network/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "projects/117961063178"
    1:  "projects/690885588241"
   ]
   violationReason:  "RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"
  }
  methodName:  "bigquery.googleapis.com/bigquery.tables.getData"
  requestMetadata: {
   callerIp:  "130.211.225.66"
   callerNetwork:  "//compute.googleapis.com/projects/perimeter-network/global/networks/__unknown__"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/927005422713"
  serviceName:  "bigquery.googleapis.com"
  status: {
   code:  7
   details: [
    0: {
     @type:  "type.googleapis.com/google.rpc.PreconditionFailure"
     violations: [
      0: {
       type:  "VPC_SERVICE_CONTROLS"
      }
     ]
    }
   ]
   message:  "Request is prohibited by organization's policy"
  }
 }
 receiveTimestamp:  "2018-11-28T20:48:51.384237810Z"
 resource: {
  labels: {
   method:  "bigquery.googleapis.com/bigquery.tables.getData"
   project_id:  "perimeter-network"
   service:  "bigquery.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-11-28T20:48:50.561884949Z"
}

Looking at callerNetwork and VpcServiceControlAuditMetadata.resourceNames, we can see three projects: perimeter-network, 117961063178 (corp-resources-public), and 690885588241 (corp-resources-protected). Recall that corp-resources-public is not in the same service perimeter as perimeter-network and corp-resources-protected.

The violationReason, RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER indicates that some resource in the request is outside of a perimeter that applies to the request. In this case, that resource is corp-resources-public.

Cloud Storage file move from VM inside perimeter

In this example, a VM in project perimeter-network uses a command to move a file from one Cloud Storage bucket, located in project corp-resources-protected, to another bucket, located in project corp-resources-public.

The VM uses the following command:

gsutil mv gs://corp-resources-private-1/yob2000.txt gs://corp-resources-public-1/babynames/

The command returns the following:

Copying gs://corp-resources-private-1/yob2000.txt [Content-Type=text/plain]...
AccessDeniedException: 403 Request violates VPC Service Controls.

The following audit log record is generated:

{
 insertId:  "1xxnssmd2hqo"
 logName:  "projects/perimeter-network/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
   principalEmail:  "storage-accessing@example.iam.gserviceaccount.com"
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "projects/_/buckets/corp-resources-public-1"
   ]
   violationReason:  "NETWORK_NOT_IN_SAME_SERVICE_PERIMETER"
  }
  methodName:  "google.storage.BillingRequiredRead"
  requestMetadata: {
   callerIp:  "130.211.225.66"
   callerNetwork:  "//compute.googleapis.com/projects/perimeter-network/global/networks/__unknown__"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/927005422713"
  serviceName:  "storage.googleapis.com"
  status: {…}
 }
 receiveTimestamp:  "2018-11-28T00:45:31.531623485Z"
 resource: {
  labels: {
   method:  "google.storage.BillingRequiredRead"
   project_id:  "perimeter-network"
   service:  "storage.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-11-28T00:45:31.351140381Z"
}

In this case, the log is less clear because the method listed is BillingRequiredRead and the action taken is move. This is a limitation of VPC Service Controls's present audit log functionality.

While the reason is less clear, this audit log record indicates that some resource in the request is outside of a perimeter that applies to the request. In this case, that resource is corp-resources-public.

Cloud Storage file move from VM outside perimeter

In this example, a VM in project public-network uses a command to move a file from one Cloud Storage bucket, located in project corp-resources-protected, to another bucket, located in project corp-resources-public.

corp-resources-protected is protected by a service perimeter. public-network and corp-resources-public exist outside the perimeter.

The VM uses the following command:

gsutil mv gs://corp-resources-private-1/yob2000.txt gs://corp-resources-public-1/babynames/

The command returns the following:

Copying gs://corp-resources-private-1/yob2000.txt [Content-Type=text/plain]...
AccessDeniedException: 403 Request violates VPC Service Controls.

The following audit log record is generated:

{
 insertId:  "10moqhsch9v"
 logName:  "projects/corp-resources-private/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
   principalEmail:  "user@example.biz"
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "projects/_/buckets/corp-resources-private-1/objects/yob2000.txt"
    1:  "projects/_/buckets/corp-resources-public-1/objects/out.txt"
   ]
   violationReason:  "RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"
  }
  methodName:  "google.storage.Write"
  requestMetadata: {
   callerIp:  "2620:15c:2c4:203:63d6:5eb8:418d:c034"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/1004338142803"
  serviceName:  "storage.googleapis.com"
  status: {
   code:  7
   details: [
    0: {
     @type:  "type.googleapis.com/google.rpc.PreconditionFailure"
     violations: [
      0: {
       type:  "VPC_SERVICE_CONTROLS"
      }
     ]
    }
   ]
   message:  "Request is prohibited by organization's policy"
  }
 }
 receiveTimestamp:  "2018-11-30T16:34:46.948010626Z"
 resource: {
  labels: {
   method:  "google.storage.Write"
   project_id:  "corp-resources-private"
   service:  "storage.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-11-30T16:34:46.898098978Z"
}

In this example, the audit log indicates that one cannot copy data across the boundary of a service perimeter (both resources are in the audit log record). Recall that the request originates from outside the perimeter (the VM in public-network), and that one of the buckets exists outside the perimeter (corp-resources-public-1).

From outside the perimeter, one is able to write to bucket corp-resources-public-1, so the check that failed in the previous example passes. However, the subsequent check to actually copy the data fails.

This example demonstrates how sometimes a single user operation results in multiple internal operations that must pass VPC Service Controls enforcement.

BigQuery dataset copy from VM inside perimeter

In this example, a VM in project 927005422713 (perimeter-network) tries to copy a BigQuery dataset from project corp-resources-private to corp-resources-public (117961063178). perimeter-network and corp-resources-private share a perimeter, while corp-resources-public exists outside the perimeter.

The VM uses the following command:

bq cp corp-resources-private:babynames.yob2000 \
  corp-resources-public:babynames.yob2000

The command returns the following:

BigQuery error in cp operation: VPC Service Controls: Request is prohibited by
organization's policy. Operation ID: c00dbc44-460f-4bd0-9d09-cda98ac800f9

The following audit log record is generated:

{
 insertId:  "146o5fd2hbp"
 logName:  "projects/perimeter-network/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "projects/117961063178"
   ]
   violationReason:  "RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"
  }
  methodName:  "bigquery.googleapis.com/bigquery.tables.get"
  requestMetadata: {
   callerIp:  "131.201.221.16"
   callerNetwork:  "//compute.googleapis.com/projects/perimeter-network/global/networks/__unknown__"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/927005422713"
  serviceName:  "bigquery.googleapis.com"
  status: {
   code:  7
   details: [
    0: {
     @type:  "type.googleapis.com/google.rpc.PreconditionFailure"
     violations: [
      0: {
       type:  "VPC_SERVICE_CONTROLS"
      }
     ]
    }
   ]
   message:  "Request is prohibited by organization's policy"
  }
 }
 receiveTimestamp:  "2018-11-28T00:27:05.688803777Z"
 resource: {
  labels: {
   method:  "bigquery.googleapis.com/bigquery.tables.get"
   project_id:  "perimeter-network"
   service:  "bigquery.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-11-28T00:27:05.378584819Z"
}

In this example, there is no single underlying API action that shows all of the resources at play in this request due to limitations of the logging mechanism and the distributed architecture of BigQuery.

The audit log record indicates that the operation failed because in order to copy the data, BigQuery must access the target project (corp-resources-public) using the network in project perimeter-network (the source of the request). Recall that corp-resources-public is outside the perimeter that protects perimeter-network. The request is denied as an attempt to exfiltrate data to corp-resources-public.

This example illustrates that one conceptual operation, such as copying data, can trigger multiple attempts to access data from different storage systems, such as Cloud Storage, BigQuery, and Cloud Bigtable. Based on how the operation is executed, the audit log record that is generated may look different than the original user command, especially when multiple checks within a given service are made and potentially fail.

Dataproc job reading from project

This example shows how to debug indirect VPC Service Controls errors that occur when using data processing services like Dataproc.

In this example, a Dataproc cluster is running in a project protected by VPC Service Controls. Hello-world.py is a pyspark job that attempts to access data from Cloud Storage bucket inside the perimeter and then write it to another bucket that exists outside the perimeter, an operation that should be blocked by VPC Service Controls.

The following command is used to execute Hello-world.py:

gcloud dataproc jobs submit pyspark hello-world.py --cluster test-cluster-new2

The command returns the following:

Job [50f16ca8-5102-442b-a545-eed5e4f5f5da] submitted.
Waiting for job output...
18/11/29 00:31:34 INFO org.spark_project.jetty.util.log: Logging initialized @2552ms
18/11/29 00:31:34 INFO org.spark_project.jetty.server.Server: jetty-9.3.z-SNAPSHOT
18/11/29 00:31:34 INFO org.spark_project.jetty.server.Server: Started @2640ms
18/11/29 00:31:34 INFO org.spark_project.jetty.server.AbstractConnector: Started ServerConnector@1f1c18ec{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
18/11/29 00:31:34 INFO com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase: GHFS version: 1.6.4-hadoop2
18/11/29 00:31:35 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at test-cluster-new2-m/10.246.0.3:8032
18/11/29 00:31:37 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1522454176466_0005
Traceback (most recent call last):
  File "/tmp/50f16ca8-5102-442b-a545-eed5e4f5f5da/hello-world.py", line 8, in <module>
    lear.saveAsTextFile("gs://corp-resources-public-1/out.txt")
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 1553, in saveAsTextFile
  File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o49.saveAsTextFile.
: java.io.IOException: Error accessing: bucket: corp-resources-public-1, object: out.txt
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.wrapException(GoogleCloudStorageImpl.java:1767)
$sp(PairRDDFunctions.scala:961)

 (truncated)

Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Request violates VPC Service Controls.",
    "reason" : "vpcServiceControls"
  } ],
  "message" : "Request violates VPC Service Controls."
}
    at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)

 (truncated)

18/11/29 00:31:43 INFO org.spark_project.jetty.server.AbstractConnector: Stopped Spark@1f1c18ec{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
ERROR: (gcloud.dataproc.jobs.submit.pyspark) Job [50f16ca8-5102-442b-a545-eed5e4f5f5da] entered state [ERROR] while waiting for [DONE].

Note the IO exception that occurs when saveAsTextFile is called. Cloud Storage returns a 403 error with the message Request violates VPC Service Controls. The error indicates that the Cloud Storage audit log operation should be reviewed.

In the audit logs for the perimeter-network project, where the command was executed, there is an audit log record for the saveAsTextFile operation:

{
 insertId:  "qdj1o9d1run"
 logName:  "projects/corp-resources-private/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
   principalEmail:  "1004338142803-compute@example.gserviceaccount.com"
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "projects/_/buckets/corp-resources-public-1/objects/out.txt"
   ]
   violationReason:  "RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"
  }
  methodName:  "google.storage.BillingRequiredRead"
  requestMetadata: {
   callerIp:  "10.246.0.3"
   callerNetwork:  "//compute.googleapis.com/projects/corp-resources-private/global/networks/__unknown__"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/1004338142803"
  serviceName:  "storage.googleapis.com"
  status: {
   code:  7
   details: [
    0: {
     @type:  "type.googleapis.com/google.rpc.PreconditionFailure"
     violations: [
      0: {
       type:  "VPC_SERVICE_CONTROLS"
      }
     ]
    }
   ]
   message:  "Request is prohibited by organization's policy"
  }
 }
 receiveTimestamp:  "2018-11-29T00:31:43.666227930Z"
 resource: {
  labels: {
   method:  "google.storage.BillingRequiredRead"
   project_id:  "corp-resources-private"
   service:  "storage.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-11-29T00:31:43.608250320Z"
}

Due to audit log limitations, the methodName for Cloud Storage is listed as Read even though it is actually a write operation. The audit log record indicates that the operation failed because a network in project corp-resources-private was attempting to access the data (writing, in this case) of a resource in bucket corp-resources-public-1. Due to the limitations of the Cloud Storage audit log, it's not clear what project bucket corp-resources-public-1 belongs to.

To identify the project that contains corp-resources-public-1, the following command is used:

gsutil --debug ls -L -b gs://corp-resources-public-1 2>&1 | grep projectNumber

The command returns the following:

projectNumber: u'117961063178'

117961063178 is project corp-resources-public, which is outside perimeter. Thus, the failure is expected.

Unsupported service with restricted VIP

Attempting to access an API that is not supported by the VPC Service Controls restricted VIP will result in a 404 error. For example, because Cloud DNS is not currently supported by VPC Service Controls, the Cloud DNS API is not available when using the restricted VIP.

For example, suppose the following command is used:

gcloud dns managed-zones list

The command returns the following:

ERROR: (gcloud.dns.managed-zones.list) Project [corp-resources-private] not found: <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){ #logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){ #logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That's an error.</ins>
  <p>The requested URL <code>/dns/v1/projects/corp-resources-private/managedZones</code> was not found on this server.  <ins>That's all we know.</ins>

This type of error is expected for services that are not supported by VPC Service Controls and not available on the restricted VIP. If this error is occurs for a service that is supported by VPC Service Controls, then you should check the known service limitations for that service to see if it is a known limitation. Otherwise, the issue should be reported.

Log export to project outside perimeter

In this example, a log export is blocked by VPC Service Controls. The export destination, project corp-resources-public, is outside of the VPC Service Controls perimeter, while the sink is created on project perimeter-network, which is inside the perimeter.

For example, suppose the following command is used:

gcloud logging sinks describe example-sink

The command returns the following:

destination: bigquery.googleapis.com/projects/corp-resources-public/datasets/logs
filter: |-
  resource.type="audited_resource"
  resource.labels.service="bigquery.googleapis.com"
name: example-sink
outputVersionFormat: V2
writerIdentity: serviceAccount:p927005422713-439672@gcp-sa-logging.iam.gserviceaccount.com

The following audit log record is generated:

{
 insertId:  "e5i2i8cbqw"
 logName:  "projects/perimeter-network/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
   principalEmail:  "p927005422713-439672@gcp-sa-logging.iam.gserviceaccount.com"
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "corp-resources-public"
   ]
   violationReason:  "RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"
  }
  methodName:  "google.cloud.bigquery.v2.TableDataService.InsertAll"
  requestMetadata: {
   callerIp:  "2002:a49:8c51::"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/927005422713"
  serviceName:  "bigquery.googleapis.com"
  status: {
   code:  7
   details: [
    0: {
     @type:  "type.googleapis.com/google.rpc.PreconditionFailure"
     violations: [
      0: {
       type:  "VPC_SERVICE_CONTROLS"
      }
     ]
    }
   ]
   message:  "Request is prohibited by organization's policy"
  }
 }
 receiveTimestamp:  "2018-11-29T17:32:19.287138882Z"
 resource: {
  labels: {
   method:  "google.cloud.bigquery.v2.TableDataService.InsertAll"
   project_id:  "perimeter-network"
   service:  "bigquery.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-11-29T17:32:19.054662413Z"
}

The audit log record is generated for BigQuery, not for Logging. This is because BigQuery is the sink service that Logging is attempting to write to.

The export fails because corp-resources-public exists outside the perimeter that protects perimeter-network.

This example shows that for cases where one Google Cloud service calls another using an internal-to-GCP managed service account such as p927005422713-439672@gcp-sa-logging.iam.gserviceaccount.com, the "network project" (in this case, perimeter-network) of the request is derived from that identity. The same identity represents the log export resource itself.

This pattern is common in Google Cloud and applies to numerous cases of service-to-service interaction.

BigQuery extract to Cloud Storage

This example describes how to debug failed BigQuery extractions to Cloud Storage.

In this example, corp-resources-private and perimeter-network are projects protected by a service perimeter. corp-resources-public is a project that exists outside the perimeter.

Suppose the following command was used:

bq extract babynames.yob2000

The command returns the following:

gs://corp-resources-public-1/export.txt
Waiting on bqjob_r47ee34109d02b41_000001676b27157c_1 ... (1s) Current status: DONE
BigQuery error in extract operation: Error processing job 'corp-resources-private:bqjob_r47ee34109d02b41_000001676b27157c_1': Access
Denied: BigQuery BigQuery: Permission denied while writing data.

In this case, the error does not specifically implicate VPC Service Controls (a similar error would be shown if there was an Identity and Access Management failure).

The following audit log record is generated:

{
 insertId:  "4gbh6pe8jld7"
 logName:  "projects/corp-resources-private/logs/cloudaudit.googleapis.com%2Fdata_access"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
   principalEmail:  "storage-accessing@example.iam.gserviceaccount.com"
  }
  methodName:  "jobservice.jobcompleted"
  requestMetadata: {
   callerIp:  "10.5.0.4"
   callerNetwork:  "//compute.googleapis.com/projects/perimeter-network/global/networks/__unknown__"
   callerSuppliedUserAgent:  "google-api-python-client/1.6.5 (gzip),gzip(gfe)"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/corp-resources-private/jobs/bqjob_r47ee34109d02b41_000001676b27157c_1"
  serviceData: {
   @type:  "type.googleapis.com/google.cloud.bigquery.logging.v1.AuditData"
   jobCompletedEvent: {
    eventName:  "extract_job_completed"
    job: {
     jobConfiguration: {
      extract: {
       destinationUris: [
        0:  "gs://corp-resources-public-1/export.txt"
       ]
       sourceTable: {
        datasetId:  "babynames"
        projectId:  "corp-resources-private"
        tableId:  "yob2000"
       }
      }
     }
     jobName: {
      jobId:  "bqjob_r47ee34109d02b41_000001676b27157c_1"
      location:  "US"
      projectId:  "corp-resources-private"
     }
     jobStatistics: {
      createTime:  "2018-12-01T19:03:03.908Z"
      endTime:  "2018-12-01T19:03:05.494Z"
      startTime:  "2018-12-01T19:03:04.013Z"
     }
     jobStatus: {
      additionalErrors: [
       0: {
        code:  7
        message:  "Access Denied: BigQuery BigQuery: Permission denied while writing data."
       }
      ]
      error: {
       code:  7
       message:  "Access Denied: BigQuery BigQuery: Permission denied while writing data."
      }
      state:  "DONE"
     }
    }
   }
  }
  serviceName:  "bigquery.googleapis.com"
  status: {
   code:  7
   message:  "Access Denied: BigQuery BigQuery: Permission denied while writing data."
  }
 }
 receiveTimestamp:  "2018-12-01T19:03:05.532169998Z"
 resource: {
  labels: {
   project_id:  "corp-resources-private"
  }
  type:  "bigquery_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-12-01T19:03:05.503Z"
}

In this audit log record, storage-accessing@example.iam.gserviceaccount.com is identified as the identity attempting to run the operation. In this example, assume that storage-accessing@example.iam.gserviceaccount.com has the required IAM permissions to execute the command.

As IAM permissions are not the issue, the next step is to check for VPC Service Controls failures.

The audit log record for the destination service (Cloud Storage) contains detailed reasons for the failure:

{
 insertId:  "1bq397kcfj1"
 logName:  "projects/corp-resources-private/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
   principalEmail:  "storage-accessing@example.iam.gserviceaccount.com"
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "projects/1004338142803"
    1:  "projects/_/buckets/corp-resources-public-1"
   ]
   violationReason:  "RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"
  }
  methodName:  "google.storage.BillingRequiredRead"
  requestMetadata: {
   callerIp:  "10.5.0.4"
   callerNetwork:  "//compute.googleapis.com/projects/perimeter-network/global/networks/__unknown__"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/1004338142803"
  serviceName:  "storage.googleapis.com"
  status: {
   code:  7
   details: [
    0: {
     @type:  "type.googleapis.com/google.rpc.PreconditionFailure"
     violations: [
      0: {
       type:  "VPC_SERVICE_CONTROLS"
      }
     ]
    }
   ]
   message:  "Request is prohibited by organization's policy"
  }
 }
 receiveTimestamp:  "2018-12-01T19:03:05.617451586Z"
 resource: {
  labels: {
   method:  "google.storage.BillingRequiredRead"
   project_id:  "corp-resources-private"
   service:  "storage.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-12-01T19:03:05.420005215Z"
}

From this log, it is clear that the two projects 1004338142803 (corp-resources-private-1) and corp-resources-public are both being used to complete the command. Because those projects do not share a perimeter, the extract job fails.

This example illustrates that in complex multi-service operations, the audit logs for both the source and destination services can contain useful debugging data.