BigQuery audit logs overview

Overview

Cloud Audit Logs are a collection of logs provided by Google Cloud that provide insight into operational concerns related to your use of Google Cloud services. This page provides details about BigQuery specific log information, and it demonstrates how to use BigQuery to analyze logged activity. For more information, see Introduction to audit logs in BigQuery.

Versions

The audit log message system relies on structured logs, and the BigQuery service provides three distinct kinds of messages:

  • AuditData: The old version of logs, which reports API invocations.

  • BigQueryAuditMetadata: The new version of logs, which reports resource interactions such as which tables were read from and written to by a given query job and which tables expired due to having an expiration time configured.

  • AuditLog: The logs that BigQuery Reservations and BigQuery Connections use when reporting requests.

Limitation

Log messages have a size limit of 100K bytes. For more information, see Truncated log entry.

Message Formats

AuditData format

The AuditData messages are communicated within the protoPayload.serviceData submessage within the Cloud Logging LogEntry message. AuditData payload returns resource.type set to bigquery_resource, not bigquery_dataset.

BigQueryAuditMetadata format

You can find BigQueryAuditMetadata details in the protoPayload.metadata submessage that is in the Cloud Logging LogEntry message.

In the Cloud Logging logs, the protoPayload.serviceData information is not set or used. In BigQueryAuditMetadata messages, there is more information:

  • resource.type is set to one of the following values:

    • bigquery_dataset for operations to datasets such as google.cloud.bigquery.v2.DatasetService.*
      • resource.labels.dataset_id contains the encapsulating dataset.
    • bigquery_project for all other called methods, such as jobs
      • resource.labels.location contains the location of the job.
  • protoPayload.methodName is set to one of the following values:

    • google.cloud.bigquery.v2.TableService.InsertTable
    • google.cloud.bigquery.v2.TableService.UpdateTable
    • google.cloud.bigquery.v2.TableService.PatchTable
    • google.cloud.bigquery.v2.TableService.DeleteTable
    • google.cloud.bigquery.v2.DatasetService.InsertDataset
    • google.cloud.bigquery.v2.DatasetService.UpdateDataset
    • google.cloud.bigquery.v2.DatasetService.PatchDataset
    • google.cloud.bigquery.v2.DatasetService.DeleteDataset
    • google.cloud.bigquery.v2.TableDataService.List
    • google.cloud.bigquery.v2.JobService.InsertJob
    • google.cloud.bigquery.v2.JobService.Query
    • google.cloud.bigquery.v2.JobService.GetQueryResults
    • InternalTableExpired
  • protoPayload.resourceName now contains the URI for the referenced resource. For example, a table created by using an insert job reports the resource URI of the table. The earlier format reported the API resource (the job identifier).

  • protoPayload.authorizationInfo only includes information relevant to the specific event. With earlier AuditData messages, you could merge multiple records when source and destination tables were in the same dataset in a query job.

AuditLog format

BigQuery Reservations uses the AuditLog format when reporting requests. Logs contain information such as:

  • resource.type is set to:

    • bigquery_project for jobs
      • resource.labels.location contains the location of the reservation-related resource.
  • protoPayload.methodName is set to one of the following values:

    • google.cloud.bigquery.reservation.v1.ReservationService.CreateReservation
    • google.cloud.bigquery.reservation.v1.ReservationService.DeleteReservation
    • google.cloud.bigquery.reservation.v1.ReservationService.UpdateReservation
    • google.cloud.bigquery.reservation.v1.ReservationService.CreateCapacityCommitment
    • google.cloud.bigquery.reservation.v1.ReservationService.DeleteCapacityCommitment
    • google.cloud.bigquery.reservation.v1.ReservationService.CreateAssignment
    • google.cloud.bigquery.reservation.v1.ReservationService.DeleteAssignment
    • google.cloud.bigquery.reservation.v1.ReservationService.MoveAssignment

BigQuery Connections uses the AuditLog format when reporting requests. Logs contain information such as:

  • resource.type is set to:

    • audited_resource
      • resource.labels.method contains the full method name.
      • resource.labels.project_id contains the project name.
      • resource.service contains service name.
  • protoPayload.methodName is set to one of the following values:

    • google.cloud.bigquery.connection.v1.ConnectionService.CreateConnection
    • google.cloud.bigquery.connection.v1.ConnectionService.DeleteConnection
    • google.cloud.bigquery.connection.v1.ConnectionService.UpdateConnection