Configuring Data Access Logs

This guide explains how to enable and configure aspects of Data Access audit logs in your project or organization. For general information about audit logs, see Cloud Audit Logging.

Admin Activity audit logs are enabled for all Google Cloud services and cannot be configured.

Configuration overview

Data Access audit logs—except for BigQuery—are disabled by default. You can enable and configure certain aspects of Data Access audit logs:

• Organizations. You can enable and configure Data Access audit logs in an organization, which applies to all the existing and new projects and folders in the organization.

• Folders. You can enable and configure Data Access audit logs in a folder, which applies to all the existing and new projects in the folder, but you can't disable a Data Access audit log that was enabled in the project's organization.

• Projects. You can configure Data Access audit logs for an individual project, but you can't disable a Data Access audit log that was enabled in the project's organization or folder.

• Default configurations. You can specify a default Data Access audit log configuration in an organization, folder or project that applies to future Google Cloud services that begin to produce Data Access audit logs.

• Services. You can specify the services whose audit logs you want to receive. For example, you might want audit logs from Compute Engine but not from Cloud SQL. For a list of the Google Cloud services that can generate audit logs, see Google services with audit logs.

• Kinds of information. You can control what kinds of information are in the audit logs. There are three kinds of Data Access audit log information:

  • ADMIN_READ: Records operations that read metadata or configuration information.

    Admin Activity audit logs record writes of metadata and configuration information. They cannot be disabled.

  • DATA_READ: Records operations that read user-provided data.

  • DATA_WRITE: Records operations that write user-provided data.

  For example, you could log only the data-write operations, but log all three kinds of information from Cloud DNS.

• Exempted users. You can exempt specific users or groups from having their data accesses recorded. For example, you can exempt your internal testing accounts from having their Cloud Debugger operations recorded.

You can configure your Data Access audit logs through the Logging console or the API. These methods are explained in the sections below.

Service-specific configurations

If there is both a Google Cloudservice-wide (allServices) configuration and a configuration for a specific Google Cloud service, then the resulting configuration for the service is the union of the two configurations. In other words:

• You can enable Data Access audit logs for specific Google Cloud services, but you cannot disable Data Access audit logs for Google Cloud services that are enabled in the broader configuration.

• You can add additional kinds of information to a Google Cloud service's Data Access audit log, but you cannot remove kinds of information that are specified in the broader configuration.

• You can add users to exemption lists, but you cannot remove users from exemption lists in the broader configuration.

Organization, folder and project configurations

You can configure Data Access audit logs for organizations, folders, and projects. If there is a configuration for a Google Cloud service across the hierarchy, then the resulting configuration is the union of the configurations. In other words, at the project level:

• You can enable logs for a Google Cloud service, but you cannot disable logs for a Google Cloud service that is enabled in the folder or organization.

• You can enable kinds of information, but you cannot disable kinds of information that are enabled in the organization or folder.

• You can add users to exemption lists, but you cannot remove users that are on the organization's or folder's exemption lists.

• At the organization or folder level, you can enable data access logs for a project within that organization or folder, even if Data Access audit logs have not been configured in the project.

Access control

You need the IAM roles roles/resourcemanager.organizationAdmin to configure Data Access audit logs at the organization level, roles/resourcemanager.folderAdmin to configure Data Access audit logs at the folder level, and roles/owner to configure Data Access audit logs at the project level.

Configuring Data Access logs with the Cloud Console

This section explains how to use the Cloud Console to configure Data Access audit logs.

You can also use API or Cloud SDK to perform these tasks programmatically; see Configuring Data Access logs with the API for details.

To access audit log configuration options in the Cloud Console, follow these steps:

1. From the Cloud Console, select IAM & Admin > Audit Logs from the upper left-hand overflow menu.

   Go to the Audit Logs page

2. Select an existing Google Cloud project, folder, or organization at the top of the page.

Enable audit logs

The following steps show you how to enable data access logs:

1. In the main table on the Audit Logs page, select one or more Google Cloud services by clicking on the box to the left of a service name in the Title column.

2. In the Log Type tab in the information panel to the right of the table, select the boxes by the Data Access audit log types that you wish to enable and then click Save.

3. Where you have successfully enabled audit logs, the table includes a checkmark. In the following example, you see that, for the Cloud Composer API service, Admin Read and Data Read audit logs are enabled:

   

You can also enable audit logs for all Google Cloud services that produce Data Access audit logs. In the main table on the Audit Logs page, select all Google Cloud services by clicking on the box to the left of Title.

Note that this "bulk" configuration method applies only to the Google Cloud services that are currently available. If a new Google Cloud service is added, it inherits your default audit configuration.

Disable audit logs

The following steps show you how to disable data access logs:

1. In the main table on the Audit Logs page, select one or more Google Cloud services by clicking on the box to the left of a service name in the Title column.

2. In the Log Type tab in the information panel to the right of the table, select the boxes by the Data Access audit log types that you wish to disable and then click Save.

3. Where you have successfully disabled audit logs, the table indicates this with a gray dash. Any enabled audit logs are indicated with a green checkmark symbol.

Set user exemptions

You can set exemptions to let you control which users generate audit logs. When you add an exempted user, audit logs are not created for that user for the selected log types. Note that Admin Activity logs are always generated regardless of exemption status.

The following steps show you how to set exemptions:

1. In the main table on the Audit Logs configuration page, select one or more Google Cloud services by clicking on the box to the left of a service name in the Title column.

2. Select the Exempted Users tab in the information panel to the right of the table. Under Add exempted user, write the email address of the user you wish add to the exemption list for your selected services. You can add multiple users by selecting the Add exempted user button as many times as needed.

3. Under the email address, select the boxes by the Data Access audit log types that you wish to disable for the user and then click Save.

4. Where you have successfully added exempted users to a service, the table indicates this with a number under the Exemptions column.

Caution: If you add an exempted user to a service for an audit log type, this enables the audit log type if it was previously disabled.

To remove a user from your exemption list:

1. Go to the Exempted Users tab in the information panel.

2. Hover over a user name and select the trash icon that appears.

3. Once the user's name is shown in strikethrough text, click Save.

To edit the information for an exempted user:

1. Go to the Exempted Users tab in the information panel.

2. Click the expander arrow to the right of the user name.

3. Select or de-select the Data Access audit log types as appropriate for the user and then click Save.

Set the default configuration

You can set a configuration that all new and existing Google Cloud services in your project, folder, or organization inherit. Setting this default configuration applies if a new Google Cloud service becomes available that users in your organization start using: the service inherits the audit logging policy that you have already set for other Google Cloud services, ensuring that Data Access audit logs are captured.

1. Click on Set Default Configuration at the top of the page.

2. In the Log Type tab, select the boxes by the Data Access audit log types that you wish to enable or disable, and then click Save.

3. In the Exempted Users tab, write the email address of the users that you wish add to the exemption list and then click Save. Follow the steps in the Set user exemptions section above.

Configuring Data Access logs with the API

This section explains how to use the API and Cloud SDK command-line interface to configure Data Access audit logs programmatically.

Many of these tasks can also be performed by using the Cloud Console; see Configuring Data Access logs with the Cloud Console for details.

IAM policy objects

To configure Data Access audit logs using the API, you must edit the IAM policy associated with your project, folder, or organization. The audit log configuration is in the auditConfigs section of the policy:

"auditConfigs": [
  {
    object(AuditConfig)
  }
]

For details, see the IAM Policy type.

The following sections describe the AuditConfig object in more detail. For the API and Cloud SDK commands used to change the configuration, see getIamPolicy and setIamPolicy

AuditConfig objects

The audit log configuration consists of a list of AuditConfig objects. Each object configures the logs for one service, or it establishes a broader configuration for all services. Each object looks like the following:

{
  "service": [SERVICE],
  "auditLogConfigs": [
    {
      "logType": "ADMIN_READ"
      "exemptedMembers": [ [MEMBER],]
    },
    {
      "logType": "DATA_READ"
      "exemptedMembers": [ [MEMBER],]
    },
    {
      "logType": "DATA_WRITE"
      "exemptedMembers": [ [MEMBER],]
    },
  ]
},

[SERVICE] is service name such as "appengine.googleapis.com", or it is the special value, "allServices". If a configuration doesn't mention a particular service, then the broader configuration is used for that service. If there is no configuration, then Data Access logs are not enabled for that service. For a list of the service names, see Log services.

The auditLogConfigs section of the AuditConfig object is a list of 0 to 3 objects, each of which configures one kind of audit log information. If you omit one of the kinds from the list, then that kind of information is not enabled for the service.

[MEMBER] is a user for whom Data Access audit logs is not collected. You can specify single users, groups, or service accounts. The Binding type describes different kinds of members, but not all of those can be used to configure Data Access audit logs.

Following is an example of an audit configuration in both JSON and YAML formats. The YAML format is the default when using the Cloud SDK commands.

JSON

"auditConfigs": [
  {
    "auditLogConfigs": [
      {
        "logType": "ADMIN_READ"
      },
      {
        "logType": "DATA_WRITE"
      },
      {
        "logType": "DATA_READ"
      }
    ],
    "service": "allServices"
  },
  {
    "auditLogConfigs": [
      {
        "exemptedMembers": [
          "[e-mail]"
        ],
        "logType": "ADMIN_READ"
      }
    ],
    "service": "cloudsql.googleapis.com"
  }
],

YAML

auditConfigs:
- auditLogConfigs:
  - logType: ADMIN_READ
  - logType: DATA_WRITE
  - logType: DATA_READ
  service: allServices
- auditLogConfigs:
  - exemptedMembers:
    - [e-mail]
    logType: ADMIN_READ
  service: cloudsql.googleapis.com

Common configurations

Following are some common audit log configurations for projects.

These configurations do not take into account any audit log configuration in the project's organization or folder. For more information, see Organization and project configurations.

Enable all access logs

The following auditConfigs section enables Data Access logs for all services and users:

JSON

"auditConfigs": [
      {
        "service": "allServices",
        "auditLogConfigs": [
          { "logType": "ADMIN_READ" },
          { "logType": "DATA_READ"  },
          { "logType": "DATA_WRITE" },
        ]
      },
    ]

YAML

auditConfigs:
- auditLogConfigs:
  - logType: ADMIN_READ
  - logType: DATA_WRITE
  - logType: DATA_READ
  service: allServices

Enable one service and information kind

The following configuration enables Data Access audit logs for Cloud SQL. The logs record only the writes of user-defined data:

JSON

"auditConfigs": [
  {
    "service": "cloudsql.googleapis.com",
    "auditLogConfigs": [
      { "logType": "DATA_WRITE" },
    ]
  },
]

YAML

auditConfigs:
- auditLogConfigs:
  - logType: DATA_WRITE
  service: cloudsql.googleapis.com

Disable all logs

To disable all Data Access audit logs (except BigQuery) in a project, include an empty auditConfigs: section in your new IAM policy:

JSON

"auditConfigs": [],

YAML

auditConfigs:

If you remove the auditConfigs section entirely from your new policy, then setIamPolicy does not change the existing Data Access audit logs configuration. For more information, see The setIamPolicy update mask.

BigQuery Data Access audit logs cannot be disabled.

getIamPolicy and setIamPolicy

You use the Cloud Resource Manager's getIamPolicy and setIamPolicy methods to read and write your IAM policy. You have several choices for the specific methods to use:

• The Cloud Resource Manager API has the following methods:

  projects.getIamPolicy
  projects.setIamPolicy
  organizations.getIamPolicy
  organizations.setIamPolicy
  

• The Cloud SDK has the following Cloud Resource Manager commands:

  gcloud projects get-iam-policy
  gcloud projects set-iam-policy
  gcloud organizations get-iam-policy
  gcloud organizations set-iam-policy
  

Note: When you call projects.getIamPolicy or projects get-iam-policy, the result shows only the policies set in the project, not the policies inherited from the parent folder or organization.

Regardless of your choice, follow these three steps:

1. Read the current policy using one of the getIamPolicy methods. Save the policy to a temporary file.
2. Edit the policy in the temporary file. Change (or add) only the auditConfigs section.
3. Write the edited policy in the temporary file, using one of the setIamPolicy methods.

setIamPolicy fails if Cloud Resource Manager detects that someone else changed the policy after you read it in the first step. If this happens, then repeat the three steps.

Caution: Your edited IAM policy replaces the current policy. Changing parts of your policy not related to audit logging might make your project inaccessible.

Examples

The following examples demonstrate how to configure your project's Data Access audit logs using the gcloud command and the Cloud Resource Manager API.

To configure organization Data Access audit logs, replace the "projects" version of the commands and API methods with the "organizations" version.

Caution: Follow the instructions carefully. You are editing a policy object that contains critical information about who can access your project or organization. Accidentally altering that information could make your project or organization unusable.

Cloud SDK

To configure your Data Access audit logs using the gcloud projects command, do the following:

1. Read your project's IAM policy and store it in a file:

   gcloud projects get-iam-policy [PROJECT_ID] > /tmp/policy.yaml
   

   The returned policy is shown below. This policy does not yet have an auditConfigs section:

   bindings:
   - members:
     - user:[e-mail]
     role: roles/editor
   - members:
     - user:[e-mail]
     role: roles/owner
   etag: BwVM-FDzeYM=
   version: 1
   

2. Edit your policy in /tmp/policy.yaml, adding or changing only the Data Access audit logs configuration.

   Caution: You must preserve the bindings: and etag: sections without changes. Failure to do so might cause your project to become unusable. For more information, see the following section, The setIamPolicy update mask.

   An example of your edited policy, which enables Cloud SQL data-write Data Access audit logs, is shown below. Four lines have been added to the beginning:

   auditConfigs:
   - auditLogConfigs:
     - logType: DATA_WRITE
     service: cloudsql.googleapis.com
   bindings:
   - members:
     - user:[e-mail]
     role: roles/editor
   - members:
     - user:[e-mail]
     role: roles/owner
   etag: BwVM-FDzeYM=
   version: 1
   

3. Write your new IAM policy:

   gcloud projects set-iam-policy [PROJECT_ID] /tmp/policy.yaml
   

   If the preceding command reports a conflict with another change, then repeat these steps, starting with the first step.

JSON

To work with your IAM policy in JSON format instead of YAML, substitute the following gcloud commands into the example:

gcloud projects get-iam-policy [PROJECT_ID] --format=json >/tmp/policy.json
gcloud projects set-iam-policy [PROJECT_ID] /tmp/policy.json

API

To configure your Data Access audit logs using the Cloud Resource Manager API, do the following:

1. Read your project's IAM policy, specifying the following parameters to the getIamPolicy API method:

   • resource: projects/[PROJECT_ID]
   • Request body: empty

   The method returns the current policy object, shown below. This project's policy does not yet have an auditConfigs section:

   {
     "bindings": [
     {
       "members": [
         "user:[e-mail]"
       ],
       "role": "roles/editor"
     },
     {
       "members": [
         "user:[e-mail]"
       ],
       "role": "roles/owner"
     }
   ],
   "etag": "BwUsv2gimRs=",
   "version": 1
   

   }

2. Edit the current policy:

   • Change or add the auditConfigs section.

     If you wish to disable your Data Access audit logs, then include an empty value for the section: auditConfigs:[].

   • Preserve the value of etag.

   You may, if you wish, remove all other information from the new policy object, as long as you are careful to set updateMask in the next step. The edited policy, which enables Cloud SQL data-write audit logs, is shown below:

   {
     "auditConfigs": [
       {
         "auditLogConfigs": [
           {
             "logType": "DATA_WRITE"
           }
         ],
         "service": "cloudsql.googleapis.com"
       }
     ],
     "etag": "BwVM-FDzeYM="
   }
   

3. Write the new policy using the setIamPolicy API method, specifying the following parameters:

   • resource: projects/[PROJECT_ID]
   • Request body:
     • updateMask: "auditConfigs,etag"
     • policy: your edited policy object

Caution: You must specify the updateMask parameter as shown. If you do not specify updateMask, you might lose all access to your project. For more information, see the following section, The setIamPolicy update mask.

The setIamPolicy update mask

This section explains the importance of the updateMask parameter in the setIamPolicy method, and explains why you must be careful with the SDK's set-iam-policy command so that you do not cause accidental harm to your project or organization.

The setIamPolicy API method uses an updateMask parameter to control which policy fields are updated. For example, if the mask does not contain bindings, then you cannot accidentally change that policy section. On the other hand, if the mask does contain bindings, then that section is always updated. If you do not include an updated value for bindings, then that section is removed entirely from the policy.

The gcloud projects set-iam-policy command, which calls setIamPolicy, does not let you specify the updateMask parameter. Instead, the command computes a value for updateMask in the following way:

• The updateMask always contains the fields bindings and etag.
• If the policy object supplied in set-iam-policy contains any other top-level fields—such as auditConfigs—then those fields are added to updateMask.

As a consequence of these rules, the set-iam-policy command has the following behaviors:

• If you omit the auditConfigs section in your new policy, then the previous value of auditConfigs section (if any) is not changed, because that section is not in the update mask. This is harmless but might be confusing.

• If you omit bindings in your new policy object, then the bindings section is removed from your policy, since this section appears in the update mask. This is very harmful, and causes all users to lose access to your project.

• If you omit etag in your new policy object, this disables the checking for concurrent changes to your policy and might result in your changes accidentally overwriting someone else's changes.