Configure default settings for organizations and folders

This document describes how to configure default resource settings for Logging by using the Google Cloud CLI. Default resource settings, which can be applied to an organization or to a folder, can determine the following:

  • Whether CMEK is required for new log buckets.

  • The storage location for the _Default and _Required log buckets.

  • Whether the _Default sink is enabled or disabled.

  • The filter that is applied to the _Default sink of new resources.

Overview

The organization resource is at the highest level of the Google Cloud resource hierarchy. The organization resource is the parent of these child resources: Google Cloud projects, folders, billing accounts and, regarding Logging, buckets.

You can configure Logging to use default resource settings for a Google Cloud organization and for folders. When you create new resources, those resources inherit the default resource settings of their parent.

Cloud Logging supports the following default resource settings:

  • Whether or not new log buckets in a resource are to be encrypted with a customer-managed key, and if so, the default Cloud KMS key to use for encryption.

    If you configure CMEK for a resource, then you must also set the default storage location for new _Default and _Required buckets that are created by child resources.

  • The storage location for new _Default and _Required buckets. This storage location lets you control where your logs are stored.

    If you set a default storage location for a resource and don't configure CMEK for that resource, then new log buckets in the resource don't require CMEK.

  • Whether the _Default log sink is enabled or disabled for new projects in the resource.

  • The inclusion filters or exclusion filters that are applied to all new _Default sinks in the child resources.

Example configurations:

  • You configure a default storage location for an organization. For new projects in the organization, the _Default and _Required buckets are created in the specified location.

  • You configure a default storage location for an organization and you configure a default storage location for each folder in that organization. For new projects that are in a folder, the _Default and _Required buckets are created in the location specified by the folder's settings. For projects that aren't in a folder, their _Default and _Required buckets are created in the location specified by the organization's settings.

  • You configure CMEK for an organization, and for the folder named Non-CMEK you only set the default storage location. If you create a project that isn't in the folder named Non-CMEK, then the _Default and _Required buckets are created in the same location as the Cloud Key Management Service key, and these log buckets are encrypted by that key. However, if you create a new project in the folder named Non-CMEK, their log buckets are created in the locations specified by that folder's setting, and those log buckets aren't encrypted by CMEK.

  • You configure an exclusion filter that applies to new _Default sinks at an organization level. The filter excludes Data Access audit logs from being routed through the _Default sink in all child resources, which prevents the Data Access audit logs from being stored in the _Default bucket.

Before you begin

This document doesn't contain information about how to configure CMEK as a default resource setting for Logging. For information about that topic, see Configure CMEK for Logging.

To get started with configuring default resource settings for Logging, do the following:

  1. Install the Google Cloud CLI, then initialize it by running the following command: 

    gcloud init

  2. Ensure that you have the following Cloud Logging permissions for the organization:

    • logging.settings.get
    • logging.settings.update

  3. Understand the LogBucket formatting requirements, including the supported locations which you can store your logs. For a list of the supported storage locations for log buckets, see Data regionality: Supported regions.

  4. Find the identifiers for the organization or the folder for which you want to configure default resource settings:

    • ORGANIZATION_ID is the unique numeric identifier of the Google Cloud organization. You don't need this value if you only plan to configure a default resource setting for a folder. For information about getting this identifier, see Getting your organization ID.
    • FOLDER_ID is the unique numeric identifier of the Google Cloud folder. You don't need this value if you only plan to configure a default resource setting for an organization. For information about using folders, see Creating and managing folders.
    • LOCATION is the location where you want to store your log data.

View default resource settings for Logging

To view the default resource settings for Logging, including the default storage location, use the gcloud logging settings describe command:

FOLDER 

 gcloud logging settings describe --folder=FOLDER_ID

ORGANIZATION 

gcloud logging settings describe --organization=ORGANIZATION_ID

The previous command returns information about the default resource settings. For example, the following shows the default resource settings for a particular organization:

name: organizations/ORGANIZATION_ID/settings
kmsKeyName: KMS_KEY_NAME
kmsServiceAccountId: SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com
storageLocation: europe-west1
disableDefaultSink: false

The value of the SERVICE_ACCT_NAME might have the format cmek-12345 or service-12345@.... If you can't use the Google Cloud CLI, then run the Cloud Logging API method getSettings.

Set the default storage location

Log buckets are the containers in your Google Cloud projects, billing accounts, folders, and organizations that store and organize your log data. For each Google Cloud project, billing account, folder, and organization, Logging automatically creates two log buckets: _Required and _Default, which are automatically stored in an unspecified global location.

You can specify a storage location for the _Required and _Default buckets that are contained by an organization or a folder by modifying the default resource settings for Logging. For a list of the supported storage locations, see Supported regions.

After you configure the default storage location for an organization, the following happens:

  • Existing _Required and _Default buckets in that organization or folder maintain the storage location that was assigned to them at the time they were created.

  • For child resources created in the organization or the folder after the default storage location is configured, their _Required and _Default buckets inherit the default storage location.

The default storage location for Cloud Logging applies only to the _Default and _Required log buckets. It doesn't apply to user-defined log buckets.

Configure the organization policies

Logging supports organization policies that can restrict where data can be stored. If such a policy exists for your organization, then you can only create log buckets in locations that are allowed by the policy.

When an organization policy that specifies a location constraint exists, the policy values for the constraint must include the location specified in the default resource settings for Logging. Further, if you plan to modify your default resource settings, before you update the default resource settings, review and, if necessary, update the organization policies.

To view or update organization policies, do the following:

  1. In the navigation panel of the Google Cloud console, select IAM & Admin, and then select Organization Policies:

    Go to Organization Policies

  2. Select your organization.

  3. View, and if necessary, update the constraint with ID constraints/gcp.resourceLocations. If this constraint isn't configured, then an update isn't required.

    For information about how to view specific constraints and how to edit these constraints, see Creating and editing policies.

Configure the default storage location for Logging

To configure the default storage location for Cloud Logging, run the gcloud logging settings update command and include the --storage-location flag:

FOLDER 

gcloud logging settings update --folder=FOLDER_ID--storage-location=LOCATION

ORGANIZATION 

gcloud logging settings update --organization=ORGANIZATION_ID --storage-location=LOCATION

If you can't use the Google Cloud CLI, then run the Cloud Logging API method updateSettings.

For information about resolving errors when updating the default storage location, see Troubleshoot setting the default resource location.

Configure the _Default sink

Logging provides a predefined _Default sink for each Google Cloud project, billing account, folder, and organization resource. Any log that is generated in the resource that matches the inclusion filter and that isn't excluded, is routed to the resource's predefined, correspondingly named _Default bucket.

You can configure default resource settings for the _Default sink for your organization and folders with the following options:

  • You can disable the _Default sink for all child resources.

  • You can configure an inclusion filter or several exclusion filters that apply to the _Default sinks of new projects.

Disable the _Default sink

You can disable the creation of _Default sinks for all new resources in an organization or folder; disabling the _Default sinks prevents logs from being stored in the resource's _Default bucket. If you stop storing logs in a resource's _Default bucket, then the logs that would have been routed to that bucket are excluded from storage in Logging, unless those logs are explicitly included in another user-defined sink for that resource.

To disable the _Default sinks for a resource and any of its child resources, run the following gcloud logging settings update command:

FOLDER 

gcloud logging settings update --folder=FOLDER_ID--disable-default-sink

ORGANIZATION 

gcloud logging settings update --organization=ORGANIZATION_ID --disable-default-sink

The disable-default-sink flag applies only to the _Default sink that routes logs into the _Default bucket.

You can re-enable the _Default sinks by running the following gcloud logging settings update command:

FOLDER 

gcloud logging settings update --folder=FOLDER_ID--no-disable-default-sink

ORGANIZATION 

gcloud logging settings update --organization=ORGANIZATION_ID --no-disable-default-sink

Configure default filter of _Default sinks

The predefined _Default sink routes any logs that match the sink criteria to the corresponding _Default bucket. You can use inclusion filters and exclusion filters to configure which logs are included and excluded for new _Default sinks in an organization or folder.

The inclusion filter can either override or be appended to the _Default sink filter, and the exclusion filters are appended as the _Default sink has no exclusion filters by default.

To specify an inclusion filter or exclusion filter that is applied to all _Default sinks of new resources in an organization or folder, run the Cloud Logging API method updateSettings with the defaultSinkConfig object. You can only set the default filter of _Default sinks by using the Logging API.

You can execute the updateSettings method by using the APIs Explorer widget on the method's reference page. The following example illustrates sample parameters:

  • name (URL): organizations/ORGANIZATION_ID/settings
  • updateMask: "default_sink_config"

  • Request body, which contains an instance of Settings:

    "defaultSinkConfig": {
  {
  "filter": "NOT LOG_ID(\"externalaudit.googleapis.com/activity\") "
  "AND NOT LOG_ID(\"cloudaudit.googleapis.com/system_event\") "
  "AND NOT LOG_ID(\"externalaudit.googleapis.com/system_event\") "
  "AND NOT LOG_ID(\"cloudaudit.googleapis.com/access_transparency\") "
  "AND NOT LOG_ID(\"externalaudit.googleapis.com/access_transparency\") ",
  "exclusions": [
     {
        "name": "exclude-data-access",
        "description": "Prevents Data Access audit logs from being routed",
        "filter": "log_id(\"cloudaudit.googleapis.com/data_access\")",
     }
  ],
  "mode": OVERWRITE
  }
}

The previous example does the following:

  • Overwrites the _Default sink's inclusion filter to include Admin Activity audit logs, which are excluded by default.

  • Appends an exclusion filter that prevents Data Access audit logs from being routed to the _Default bucket.

Troubleshoot configuration errors

For troubleshooting information, see Troubleshoot CMEK and default setting errors.