Troubleshooting

This page lists various issues that might be encountered when attempting to configure VPC Service Controls.

Find VPC Service Controls errors

To find errors related to VPC Service Controls, use Stackdriver Logging.

Console

This procedure uses terms from the Logging documentation. For more information, refer to Basic log filters.

To obtain the last 24 hours of VPC Service Controls errors in Logging:

  1. In the Google Cloud Platform Console, go to the Stackdriver Logging page for the project inside the VPC Service Perimeter.

    Go to the Stackdriver Logging page

  2. In the search-filter box, enter the following:

    protoPayload.metadata.@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
    
  3. In the resource basic selector menu, select Audited Resource.

  4. In the time-range selector drop-down menu, select Last 24 hours.

  5. Optionally, if you want to find the VPC Service Controls errors that have occurred during a different period of time, use the time-range selector drop-down menu to change the range.

gcloud

To obtain the last 24 hours of VPC Service Controls errors, use the following command:

gcloud logging read 'protoPayload.metadata.@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"'

By default, the read command is limited to the last 24 hours. To obtain VPC Service Controls logs for a different period of time, use one of the following commands.

To limit the logs relative to the current date:

gcloud logging read \
  'protoPayload.metadata.@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"' \
  --freshness=DURATION

Where:

For example, to obtain all VPC Service Controls errors that have occurred over the last week:

gcloud logging read \
  'protoPayload.metadata.@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"' \
  --freshness=7d

To limit the logs to a specific time frame:

gcloud logging read \
  'protoPayload.metadata.@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata" AND
  timestamp>="DATETIME" AND
  timestamp<="DATETIME"'

Where:

  • DATETIME is a formatted date/time string. For more information on formatting, refer to the absolute date/time formats for gcloud.

For example, to obtain all VPC Service Controls errors that have occurred between March 22, 2019 and March 26, 2019:

gcloud logging read \
  'protoPayload.metadata.@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata" AND
  timestamp>="2019-03-22T23:59:59Z" AND
  timestamp<="2019-03-26T00:00:00Z"'

Unsupported Services

For more information on products and services that are supported by VPC Service Controls, refer to the Supported products page.

Attempting to restrict an unsupported service using the gcloud command-line tool or the Access Context Manager API will result in an error.

Cross-project access to data of supported services will be blocked by VPC Service Controls. Additionally, the restricted VIP can be used to block the ability of workloads to call unsupported services.

Determine if an error is due to VPC Service Controls

Because VPC Service Controls modifies certain low-level properties of Google Cloud Platform (GCP), it can have cascading effects across services that can be challenging to debug if you don't know what to look for.

To identify if the error is related to VPC Service Controls, check whether you have enabled VPC Service Controls and applied it to the projects and services you are attempting to use. This can be done via the gcloud command-line tool or GCP Console.

If you are using a service (potentially indirectly, via another service) that is marked as a restricted service by VPC Service Controls in a project that is inside a service perimeter, then VPC Service Controls could be at fault.

For information about known problematic cases, refer to Known service limitations.

Usually, services will propagate error messages from their dependencies. If you encounter one of the following errors, it is indicative of a problem with VPC Service Controls.

  • Cloud Storage: 403: Request violates VPC Service Controls.

  • BigQuery: 403: VPC Service Controls: Request is prohibited by organization's policy.

  • Other services: 403: Request is prohibited by organization's policy.

Debugging a request blocked for unexpected reasons by VPC Service Controls

The VPC Service Controls audit log is the primary tool for debugging why a request has been blocked by VPC Service Controls.

In most cases when access has been blocked unexpectedly, consult the audit logs in the project that was the source of the request. These logs contain significant data about the requested resources and the reason why the request was denied.

For more information on viewing logs, read Viewing logs.

The following table lists the violationReason values that you may encounter when using VPC Service Controls.

violationReason Explanation
RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER The resources listed under resourceNames in the audit log record are not in the same service perimeter.
NETWORK_NOT_IN_SAME_SERVICE_PERIMETER The resources corresponding to callerNetwork and resourceNames in the audit log record are not in the same service perimeter.
NO_MATCHING_ACCESS_LEVEL

The IP address corresponding to callerIp in the audit log record does not match any IP ranges in access levels assigned to the service perimeter.

Example Scenarios

The following examples cover scenarios that you may encounter while using VPC Service Controls.

Cloud Storage access from on-premises

In this example, a request from an employee workstation (identified by callerIp) to a Cloud Storage bucket in project corp-storage is blocked by VPC Service Controls.

The request generates the following audit log record:

{
 insertId:  "222lvajc6f7"
 logName:  "projects/corp-storage/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
   principalEmail:  "someone@google.com"
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "projects/_"
   ]
   violationReason:  "NO_MATCHING_ACCESS_LEVEL"
  }
  methodName:  "google.storage.NoBillingOk"
  requestMetadata: {
   callerIp:  "b1d5:d26d:5b17:43fe:d358:586b:db59:9617"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/690885588241"
  serviceName:  "storage.googleapis.com"
  status: {
   code:  7
   details: [
    0: {
     @type:  "type.googleapis.com/google.rpc.PreconditionFailure"
     violations: [
      0: {
       type:  "VPC_SERVICE_CONTROLS"
      }
     ]
    }
   ]
   message:  "Request is prohibited by organization's policy"
  }
 }
 receiveTimestamp:  "2018-11-27T21:40:43.823209571Z"
 resource: {
  labels: {
   method:  "google.storage.NoBillingOk"
   project_id:  "corp-storage"
   service:  "storage.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-11-27T21:40:42.973784140Z"
}

The corp-storage project is included in a service perimeter. The employee workstation is not a part of any networks within that perimeter. Because the employee workstation exists outside the perimeter, the request is blocked.

BigQuery access from VM outside of project

In this example, a VM that belongs to project 458854174376 (data-collector) attempts to run a BigQuery query against a dataset in project 798816221974 (corp-resources-protected) and it is denied.

The VM uses the following query:

bq --project=corp-resources-protected query 'select count(*) from babynames.yob2000'

The query returns the following:

BigQuery error in query operation: VPC Service Controls: Request is
prohibited by organization's policy. Operation ID:
33643962-6a0f-4091-9283-bcdf7e9271f0

The following audit log record is generated:

{
 insertId:  "1ei551d2pdq"
 logName:  "projects/corp-resources-protected/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
   principalEmail:  "714877721106-compute@example.gserviceaccount.com"
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "projects/1004338142803"
   ]
   violationReason:  "NETWORK_NOT_IN_SAME_SERVICE_PERIMETER"
  }
  methodName:  "bigquery.googleapis.com/bigquery.jobs.create"
  requestMetadata: {
   callerIp:  "10.105.0.2"
   callerNetwork:  "//compute.googleapis.com/projects/ameet-dataflow/global/networks/__unknown__"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/1004338142803"
  serviceName:  "bigquery.googleapis.com"
  status: {
   code:  7
   details: [
    0: {
     @type:  "type.googleapis.com/google.rpc.PreconditionFailure"
     violations: [
      0: {
       type:  "VPC_SERVICE_CONTROLS"
      }
     ]
    }
   ]
   message:  "Request is prohibited by organization's policy"
  }
 }
 receiveTimestamp:  "2018-11-28T23:06:13.579882505Z"
 resource: {
  labels: {
   method:  "bigquery.googleapis.com/bigquery.jobs.create"
   project_id:  "corp-resources-protected"
   service:  "bigquery.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-11-28T23:06:12.799656975Z"
}

In this example, the violationReason is NETWORK_NOT_IN_SAME_SERVICE_PERIMETER. callerNetwork is included in addition to callerIp. The IP address is private and the network is provided to disambiguate it. The relevant resources at issue here are listed in two places: VpcServiceControlAuditMetadata.resourceNames and requestMetadata.callerNetwork (the project that owns the network).

The problem is that the corp-resources-protected project is inside a service perimeter, while data-collector, the project that includes the network that the VM belongs to, is not. In this case, access is denied as expected.

Cross-project BigQuery query

In this example, a VM that belongs to the perimeter-network project attempts to query the BigQuery instances of two different projects: corp-resources-protected, which is in the same service perimeter as perimeter-network, and corp-resources-public, which is not.

The VM uses the following command:

bq query --use_legacy_sql=false \
  'select count(priv.name),count(pub.name) from \
  `corp-resources-protected.babynames.yob2000` as priv, \
  `corp-resources-public.babynames.yob2000` as pub'

The query returns the following:

BigQuery error in query operation: Error processing job
'example:bqjob_r211e6f6eec928ffb_000001675c996aa8_1': VPC Service Controls:
Request is prohibited by organization's policy. Operation ID:
dc4fc177-4850-4fc5-b2e7-8c33f302149a

The following audit log record is generated:

{
 insertId:  "17kg4exd24ag"
 logName:  "projects/perimeter-network/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "projects/117961063178"
    1:  "projects/690885588241"
   ]
   violationReason:  "RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"
  }
  methodName:  "bigquery.googleapis.com/bigquery.tables.getData"
  requestMetadata: {
   callerIp:  "130.211.225.66"
   callerNetwork:  "//compute.googleapis.com/projects/perimeter-network/global/networks/__unknown__"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/927005422713"
  serviceName:  "bigquery.googleapis.com"
  status: {
   code:  7
   details: [
    0: {
     @type:  "type.googleapis.com/google.rpc.PreconditionFailure"
     violations: [
      0: {
       type:  "VPC_SERVICE_CONTROLS"
      }
     ]
    }
   ]
   message:  "Request is prohibited by organization's policy"
  }
 }
 receiveTimestamp:  "2018-11-28T20:48:51.384237810Z"
 resource: {
  labels: {
   method:  "bigquery.googleapis.com/bigquery.tables.getData"
   project_id:  "perimeter-network"
   service:  "bigquery.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-11-28T20:48:50.561884949Z"
}

Looking at callerNetwork and VpcServiceControlAuditMetadata.resourceNames, we can see three projects: perimeter-network, 117961063178 (corp-resources-public), and 690885588241 (corp-resources-protected). Recall that corp-resources-public is not in the same service perimeter as perimeter-network and corp-resources-protected.

The violationReason, RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER indicates that some resource in the request is outside of a perimeter that applies to the request. In this case, that resource is corp-resources-public.

Cloud Storage file move from VM inside perimeter

In this example, a VM in project perimeter-network uses a command to move a file from one Cloud Storage bucket, located in project corp-resources-protected, to another bucket, located in project corp-resources-public.

The VM uses the following command:

gsutil mv gs://corp-resources-private-1/yob2000.txt gs://corp-resources-public-1/babynames/

The command returns the following:

Copying gs://corp-resources-private-1/yob2000.txt [Content-Type=text/plain]...
AccessDeniedException: 403 Request violates VPC Service Controls.

The following audit log record is generated:

{
 insertId:  "1xxnssmd2hqo"
 logName:  "projects/perimeter-network/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
   principalEmail:  "storage-accessing@example.iam.gserviceaccount.com"
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "projects/_/buckets/corp-resources-public-1"
   ]
   violationReason:  "NETWORK_NOT_IN_SAME_SERVICE_PERIMETER"
  }
  methodName:  "google.storage.BillingRequiredRead"
  requestMetadata: {
   callerIp:  "130.211.225.66"
   callerNetwork:  "//compute.googleapis.com/projects/perimeter-network/global/networks/__unknown__"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/927005422713"
  serviceName:  "storage.googleapis.com"
  status: {…}
 }
 receiveTimestamp:  "2018-11-28T00:45:31.531623485Z"
 resource: {
  labels: {
   method:  "google.storage.BillingRequiredRead"
   project_id:  "perimeter-network"
   service:  "storage.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-11-28T00:45:31.351140381Z"
}

In this case, the log is less clear because the method listed is BillingRequiredRead and the action taken is move. This is a limitation of VPC Service Controls's present audit log functionality.

While the reason is less clear, this audit log record indicates that some resource in the request is outside of a perimeter that applies to the request. In this case, that resource is corp-resources-public.

Cloud Storage file move from VM outside perimeter

In this example, a VM in project public-network uses a command to move a file from one Cloud Storage bucket, located in project corp-resources-protected, to another bucket, located in project corp-resources-public.

corp-resources-protected is protected by a service perimeter. public-network and corp-resources-public exist outside the perimeter.

The VM uses the following command:

gsutil mv gs://corp-resources-private-1/yob2000.txt gs://corp-resources-public-1/babynames/

The command returns the following:

Copying gs://corp-resources-private-1/yob2000.txt [Content-Type=text/plain]...
AccessDeniedException: 403 Request violates VPC Service Controls.

The following audit log record is generated:

{
 insertId:  "10moqhsch9v"
 logName:  "projects/corp-resources-private/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
   principalEmail:  "user@example.biz"
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "projects/_/buckets/corp-resources-private-1/objects/yob2000.txt"
    1:  "projects/_/buckets/corp-resources-public-1/objects/out.txt"
   ]
   violationReason:  "RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"
  }
  methodName:  "google.storage.Write"
  requestMetadata: {
   callerIp:  "2620:15c:2c4:203:63d6:5eb8:418d:c034"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/1004338142803"
  serviceName:  "storage.googleapis.com"
  status: {
   code:  7
   details: [
    0: {
     @type:  "type.googleapis.com/google.rpc.PreconditionFailure"
     violations: [
      0: {
       type:  "VPC_SERVICE_CONTROLS"
      }
     ]
    }
   ]
   message:  "Request is prohibited by organization's policy"
  }
 }
 receiveTimestamp:  "2018-11-30T16:34:46.948010626Z"
 resource: {
  labels: {
   method:  "google.storage.Write"
   project_id:  "corp-resources-private"
   service:  "storage.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-11-30T16:34:46.898098978Z"
}

In this example, the audit log indicates that one cannot copy data across the boundary of a service perimeter (both resources are in the audit log record). Recall that the request originates from outside the perimeter (the VM in public-network), and that one of the buckets exists outside the perimeter (corp-resources-public-1).

From outside the perimeter, one is able to write to bucket corp-resources-public-1, so the check that failed in the previous example passes. However, the subsequent check to actually copy the data fails.

This example demonstrates how sometimes a single user operation results in multiple internal operations that must pass VPC Service Controls enforcement.

BigQuery dataset copy from VM inside perimeter

In this example, a VM in project 927005422713 (perimeter-network) tries to copy a BigQuery dataset from project corp-resources-private to corp-resources-public (117961063178). perimeter-network and corp-resources-private share a perimeter, while corp-resources-public exists outside the perimeter.

The VM uses the following command:

bq cp corp-resources-private:babynames.yob2000 \
  corp-resources-public:babynames.yob2000

The command returns the following:

BigQuery error in cp operation: VPC Service Controls: Request is prohibited by
organization's policy. Operation ID: c00dbc44-460f-4bd0-9d09-cda98ac800f9

The following audit log record is generated:

{
 insertId:  "146o5fd2hbp"
 logName:  "projects/perimeter-network/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "projects/117961063178"
   ]
   violationReason:  "RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"
  }
  methodName:  "bigquery.googleapis.com/bigquery.tables.get"
  requestMetadata: {
   callerIp:  "131.201.221.16"
   callerNetwork:  "//compute.googleapis.com/projects/perimeter-network/global/networks/__unknown__"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/927005422713"
  serviceName:  "bigquery.googleapis.com"
  status: {
   code:  7
   details: [
    0: {
     @type:  "type.googleapis.com/google.rpc.PreconditionFailure"
     violations: [
      0: {
       type:  "VPC_SERVICE_CONTROLS"
      }
     ]
    }
   ]
   message:  "Request is prohibited by organization's policy"
  }
 }
 receiveTimestamp:  "2018-11-28T00:27:05.688803777Z"
 resource: {
  labels: {
   method:  "bigquery.googleapis.com/bigquery.tables.get"
   project_id:  "perimeter-network"
   service:  "bigquery.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-11-28T00:27:05.378584819Z"
}

In this example, there is no single underlying API action that shows all of the resources at play in this request due to limitations of the logging mechanism and the distributed architecture of BigQuery.

The audit log record indicates that the operation failed because in order to copy the data, BigQuery must access the target project (corp-resources-public) using the network in project perimeter-network (the source of the request). Recall that corp-resources-public is outside the perimeter that protects perimeter-network. The request is denied as an attempt to exfiltrate data to corp-resources-public.

This example illustrates that one conceptual operation, such as copying data, can trigger multiple attempts to access data from different storage systems, such as Cloud Storage, BigQuery, and Cloud Bigtable. Based on how the operation is executed, the audit log record that is generated may look different than the original user command, especially when multiple checks within a given service are made and potentially fail.

Cloud Dataproc job reading from project

This example shows how to debug indirect VPC Service Controls errors that occur when using data processing services like Cloud Dataproc.

In this example, a Cloud Dataproc cluster is running in a project protected by VPC Service Controls. Hello-world.py is a pyspark job that attempts to access data from Cloud Storage bucket inside the perimeter and then write it to another bucket that exists outside the perimeter, an operation that should be blocked by VPC Service Controls.

The following command is used to execute Hello-world.py:

gcloud dataproc jobs submit pyspark hello-world.py --cluster test-cluster-new2

The command returns the following:

Job [50f16ca8-5102-442b-a545-eed5e4f5f5da] submitted.
Waiting for job output...
18/11/29 00:31:34 INFO org.spark_project.jetty.util.log: Logging initialized @2552ms
18/11/29 00:31:34 INFO org.spark_project.jetty.server.Server: jetty-9.3.z-SNAPSHOT
18/11/29 00:31:34 INFO org.spark_project.jetty.server.Server: Started @2640ms
18/11/29 00:31:34 INFO org.spark_project.jetty.server.AbstractConnector: Started ServerConnector@1f1c18ec{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
18/11/29 00:31:34 INFO com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase: GHFS version: 1.6.4-hadoop2
18/11/29 00:31:35 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at test-cluster-new2-m/10.246.0.3:8032
18/11/29 00:31:37 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1522454176466_0005
Traceback (most recent call last):
  File "/tmp/50f16ca8-5102-442b-a545-eed5e4f5f5da/hello-world.py", line 8, in <module>
    lear.saveAsTextFile("gs://corp-resources-public-1/out.txt")
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 1553, in saveAsTextFile
  File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o49.saveAsTextFile.
: java.io.IOException: Error accessing: bucket: corp-resources-public-1, object: out.txt
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.wrapException(GoogleCloudStorageImpl.java:1767)
$sp(PairRDDFunctions.scala:961)

 (truncated)

Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Request violates VPC Service Controls.",
    "reason" : "vpcServiceControls"
  } ],
  "message" : "Request violates VPC Service Controls."
}
    at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)

 (truncated)

18/11/29 00:31:43 INFO org.spark_project.jetty.server.AbstractConnector: Stopped Spark@1f1c18ec{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
ERROR: (gcloud.dataproc.jobs.submit.pyspark) Job [50f16ca8-5102-442b-a545-eed5e4f5f5da] entered state [ERROR] while waiting for [DONE].

Note the IO exception that occurs when saveAsTextFile is called. Cloud Storage returns a 403 error with the message Request violates VPC Service Controls. The error indicates that the Cloud Storage audit log operation should be reviewed.

In the audit logs for the perimeter-network project, where the command was executed, there is an audit log record for the saveAsTextFile operation:

{
 insertId:  "qdj1o9d1run"
 logName:  "projects/corp-resources-private/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
   principalEmail:  "1004338142803-compute@example.gserviceaccount.com"
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "projects/_/buckets/corp-resources-public-1/objects/out.txt"
   ]
   violationReason:  "RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"
  }
  methodName:  "google.storage.BillingRequiredRead"
  requestMetadata: {
   callerIp:  "10.246.0.3"
   callerNetwork:  "//compute.googleapis.com/projects/corp-resources-private/global/networks/__unknown__"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/1004338142803"
  serviceName:  "storage.googleapis.com"
  status: {
   code:  7
   details: [
    0: {
     @type:  "type.googleapis.com/google.rpc.PreconditionFailure"
     violations: [
      0: {
       type:  "VPC_SERVICE_CONTROLS"
      }
     ]
    }
   ]
   message:  "Request is prohibited by organization's policy"
  }
 }
 receiveTimestamp:  "2018-11-29T00:31:43.666227930Z"
 resource: {
  labels: {
   method:  "google.storage.BillingRequiredRead"
   project_id:  "corp-resources-private"
   service:  "storage.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-11-29T00:31:43.608250320Z"
}

Due to audit log limitations, the methodName for Cloud Storage is listed as Read even though it is actually a write operation. The audit log record indicates that the operation failed because a network in project corp-resources-private was attempting to access the data (writing, in this case) of a resource in bucket corp-resources-public-1. Due to the limitations of the Cloud Storage audit log, it's not clear what project bucket corp-resources-public-1 belongs to.

To identify the project that contains corp-resources-public-1, the following command is used:

gsutil --debug ls -L -b gs://corp-resources-public-1 2>&1 | grep projectNumber

The command returns the following:

projectNumber: u'117961063178'

117961063178 is project corp-resources-public, which is outside perimeter. Thus, the failure is expected.

Unsupported service with restricted VIP

Attempting to access an API that is not supported by the VPC Service Controls restricted VIP will result in a 404 error. For example, because Cloud DNS is not currently supported by VPC Service Controls, the Cloud DNS API is not available when using the restricted VIP.

For example, suppose the following command is used:

gcloud dns managed-zones list

The command returns the following:

ERROR: (gcloud.dns.managed-zones.list) Project [corp-resources-private] not found: <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){ #logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){ #logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That's an error.</ins>
  <p>The requested URL <code>/dns/v1/projects/corp-resources-private/managedZones</code> was not found on this server.  <ins>That's all we know.</ins>

This type of error is expected for services that are not supported by VPC Service Controls and not available on the restricted VIP. If this error is occurs for a service that is supported by VPC Service Controls, then you should check the known service limitations for that service to see if it is a known limitation. Otherwise, the issue should be reported.

Log export to project outside perimeter

In this example, a log export is blocked by VPC Service Controls. The export destination, project corp-resources-public, is outside of the VPC Service Controls perimeter, while the sink is created on project perimeter-network, which is inside the perimeter.

For example, suppose the following command is used:

gcloud logging sinks describe example-sink

The command returns the following:

destination: bigquery.googleapis.com/projects/corp-resources-public/datasets/logs
filter: |-
  resource.type="audited_resource"
  resource.labels.service="bigquery.googleapis.com"
name: example-sink
outputVersionFormat: V2
writerIdentity: serviceAccount:p927005422713-439672@gcp-sa-logging.iam.gserviceaccount.com

The following audit log record is generated:

{
 insertId:  "e5i2i8cbqw"
 logName:  "projects/perimeter-network/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
   principalEmail:  "p927005422713-439672@gcp-sa-logging.iam.gserviceaccount.com"
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "corp-resources-public"
   ]
   violationReason:  "RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"
  }
  methodName:  "google.cloud.bigquery.v2.TableDataService.InsertAll"
  requestMetadata: {
   callerIp:  "2002:a49:8c51::"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/927005422713"
  serviceName:  "bigquery.googleapis.com"
  status: {
   code:  7
   details: [
    0: {
     @type:  "type.googleapis.com/google.rpc.PreconditionFailure"
     violations: [
      0: {
       type:  "VPC_SERVICE_CONTROLS"
      }
     ]
    }
   ]
   message:  "Request is prohibited by organization's policy"
  }
 }
 receiveTimestamp:  "2018-11-29T17:32:19.287138882Z"
 resource: {
  labels: {
   method:  "google.cloud.bigquery.v2.TableDataService.InsertAll"
   project_id:  "perimeter-network"
   service:  "bigquery.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-11-29T17:32:19.054662413Z"
}

The audit log record is generated for BigQuery, not for Logging. This is because BigQuery is the sink service that Logging is attempting to write to.

The export fails because corp-resources-public exists outside the perimeter that protects perimeter-network.

This example shows that for cases where one GCP service calls another using an internal-to-GCP managed service account such as p927005422713-439672@gcp-sa-logging.iam.gserviceaccount.com, the "network project" (in this case, perimeter-network) of the request is derived from that identity. The same identity represents the log export resource itself.

This pattern is common in GCP and applies to numerous cases of service-to-service interaction.

BigQuery extract to Cloud Storage

This example describes how to debug failed BigQuery extractions to Cloud Storage.

In this example, corp-resources-private and perimeter-network are projects protected by a service perimeter. corp-resources-public is a project that exists outside the perimeter.

Suppose the following command was used:

bq extract babynames.yob2000

The command returns the following:

gs://corp-resources-public-1/export.txt
Waiting on bqjob_r47ee34109d02b41_000001676b27157c_1 ... (1s) Current status: DONE
BigQuery error in extract operation: Error processing job 'corp-resources-private:bqjob_r47ee34109d02b41_000001676b27157c_1': Access
Denied: BigQuery BigQuery: Permission denied while writing data.

In this case, the error does not specifically implicate VPC Service Controls (a similar error would be shown if there was a Cloud Identity and Access Management failure).

The following audit log record is generated:

{
 insertId:  "4gbh6pe8jld7"
 logName:  "projects/corp-resources-private/logs/cloudaudit.googleapis.com%2Fdata_access"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
   principalEmail:  "storage-accessing@example.iam.gserviceaccount.com"
  }
  methodName:  "jobservice.jobcompleted"
  requestMetadata: {
   callerIp:  "10.5.0.4"
   callerNetwork:  "//compute.googleapis.com/projects/perimeter-network/global/networks/__unknown__"
   callerSuppliedUserAgent:  "google-api-python-client/1.6.5 (gzip),gzip(gfe)"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/corp-resources-private/jobs/bqjob_r47ee34109d02b41_000001676b27157c_1"
  serviceData: {
   @type:  "type.googleapis.com/google.cloud.bigquery.logging.v1.AuditData"
   jobCompletedEvent: {
    eventName:  "extract_job_completed"
    job: {
     jobConfiguration: {
      extract: {
       destinationUris: [
        0:  "gs://corp-resources-public-1/export.txt"
       ]
       sourceTable: {
        datasetId:  "babynames"
        projectId:  "corp-resources-private"
        tableId:  "yob2000"
       }
      }
     }
     jobName: {
      jobId:  "bqjob_r47ee34109d02b41_000001676b27157c_1"
      location:  "US"
      projectId:  "corp-resources-private"
     }
     jobStatistics: {
      createTime:  "2018-12-01T19:03:03.908Z"
      endTime:  "2018-12-01T19:03:05.494Z"
      startTime:  "2018-12-01T19:03:04.013Z"
     }
     jobStatus: {
      additionalErrors: [
       0: {
        code:  7
        message:  "Access Denied: BigQuery BigQuery: Permission denied while writing data."
       }
      ]
      error: {
       code:  7
       message:  "Access Denied: BigQuery BigQuery: Permission denied while writing data."
      }
      state:  "DONE"
     }
    }
   }
  }
  serviceName:  "bigquery.googleapis.com"
  status: {
   code:  7
   message:  "Access Denied: BigQuery BigQuery: Permission denied while writing data."
  }
 }
 receiveTimestamp:  "2018-12-01T19:03:05.532169998Z"
 resource: {
  labels: {
   project_id:  "corp-resources-private"
  }
  type:  "bigquery_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-12-01T19:03:05.503Z"
}

In this audit log record, storage-accessing@example.iam.gserviceaccount.com is identified as the identity attempting to run the operation. In this example, assume that storage-accessing@example.iam.gserviceaccount.com has the required Cloud IAM permissions to execute the command.

As Cloud IAM permissions are not the issue, the next step is to check for VPC Service Controls failures.

The audit log record for the destination service (Cloud Storage) contains detailed reasons for the failure:

{
 insertId:  "1bq397kcfj1"
 logName:  "projects/corp-resources-private/logs/cloudaudit.googleapis.com%2Fpolicy"
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  authenticationInfo: {
   principalEmail:  "storage-accessing@example.iam.gserviceaccount.com"
  }
  metadata: {
   @type:  "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   resourceNames: [
    0:  "projects/1004338142803"
    1:  "projects/_/buckets/corp-resources-public-1"
   ]
   violationReason:  "RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"
  }
  methodName:  "google.storage.BillingRequiredRead"
  requestMetadata: {
   callerIp:  "10.5.0.4"
   callerNetwork:  "//compute.googleapis.com/projects/perimeter-network/global/networks/__unknown__"
   destinationAttributes: {
   }
   requestAttributes: {
   }
  }
  resourceName:  "projects/1004338142803"
  serviceName:  "storage.googleapis.com"
  status: {
   code:  7
   details: [
    0: {
     @type:  "type.googleapis.com/google.rpc.PreconditionFailure"
     violations: [
      0: {
       type:  "VPC_SERVICE_CONTROLS"
      }
     ]
    }
   ]
   message:  "Request is prohibited by organization's policy"
  }
 }
 receiveTimestamp:  "2018-12-01T19:03:05.617451586Z"
 resource: {
  labels: {
   method:  "google.storage.BillingRequiredRead"
   project_id:  "corp-resources-private"
   service:  "storage.googleapis.com"
  }
  type:  "audited_resource"
 }
 severity:  "ERROR"
 timestamp:  "2018-12-01T19:03:05.420005215Z"
}

From this log, it is clear that the two projects 1004338142803 (corp-resources-private-1) and corp-resources-public are both being used to complete the command. Because those projects do not share a perimeter, the extract job fails.

This example illustrates that in complex multi-service operations, the audit logs for both the source and destination services can contain useful debugging data.

Hai trovato utile questa pagina? Facci sapere cosa ne pensi:

Invia feedback per...

VPC Service Controls