This document describes how to configure default resource settings for Logging by using the Google Cloud CLI. Default resource settings, which can be applied to an organization or to a folder, can determine the following:
Whether CMEK is required for new log buckets.
The storage location, which determines the following:
Where
_Default
and_Required
log buckets are stored.Where queries in the Logs Explorer or Log Analytics pages are stored, specifically recent queries, and queries saved by a member of the Google Cloud project.
Whether the
_Default
sink is enabled or disabled.The filter that is applied to the
_Default
sink of new resources.
Overview
The organization resource is at the highest level of the Google Cloud resource hierarchy. The organization resource is the parent of these child resources: Google Cloud projects, folders, billing accounts and, regarding Logging, buckets.
You can configure Logging to use default resource settings for a Google Cloud organization and for folders. When you create new resources, those resources inherit the default resource settings of their parent.
Cloud Logging supports the following default resource settings:
Whether or not new log buckets in a resource are to be encrypted with a customer-managed key, and if so, the default Cloud KMS key to use for encryption.
If you configure CMEK for a resource, then you must also set the default storage location for new
_Default
and_Required
buckets that are created by child resources.The storage location for new
_Default
and_Required
buckets, and queries in the Logs Explorer or Log Analytics pages. This storage location lets you control where your logs are stored.If you set a default storage location for a resource and don't configure CMEK for that resource, then new log buckets in the resource don't require CMEK.
Whether the
_Default
log sink is enabled or disabled for new projects in the resource.The inclusion filters or exclusion filters that are applied to all new
_Default
sinks in the child resources.
Example configurations:
You configure a default storage location for an organization. For new projects in the organization, the
_Default
and_Required
buckets are created in the specified location.You configure a default storage location for an organization and you configure a default storage location for each folder in that organization. For new projects that are in a folder, the
_Default
and_Required
buckets are created in the location specified by the folder's settings. For projects that aren't in a folder, their_Default
and_Required
buckets are created in the location specified by the organization's settings.You configure a default storage location where all queries in the Logs Explorer are stored. This includes recent queries that are automatically saved after being run, and queries saved by members of the Google Cloud project.
You configure CMEK for an organization, and for the folder named
Non-CMEK
you only set the default storage location. If you create a project that isn't in the folder namedNon-CMEK
, then the_Default
and_Required
buckets are created in the same location as the Cloud Key Management Service key, and these log buckets are encrypted by that key. However, if you create a new project in the folder namedNon-CMEK
, their log buckets are created in the locations specified by that folder's setting, and those log buckets aren't encrypted by CMEK.You configure an exclusion filter that applies to new
_Default
sinks at an organization level. The filter excludes Data Access audit logs from being routed through the_Default
sink in all child resources, which prevents the Data Access audit logs from being stored in the_Default
bucket.
Before you begin
This document doesn't contain information about how to configure CMEK as a default resource setting for Logging. For information about that topic, see Configure CMEK for Logging.
To get started with configuring default resource settings for Logging, do the following:
Install the Google Cloud CLI, then initialize it by running the following command:
gcloud init
Ensure that you have the following Cloud Logging permissions for the organization:
logging.settings.get
logging.settings.update
Understand the
LogBucket
formatting requirements, including the supported locations which you can store your logs. For a list of the supported storage locations for log buckets, see Data regionality: Supported regions.Find the identifiers for the organization or the folder for which you want to configure default resource settings:
- ORGANIZATION_ID is the unique numeric identifier of the Google Cloud organization. You don't need this value if you only plan to configure a default resource setting for a folder. For information about getting this identifier, see Getting your organization ID.
- FOLDER_ID is the unique numeric identifier of the Google Cloud folder. You don't need this value if you only plan to configure a default resource setting for an organization. For information about using folders, see Creating and managing folders.
- LOCATION is the location where you want to store your log data.
View default resource settings for Logging
To view the default resource settings for Logging,
including the default storage location, use the
gcloud logging settings describe
command:
FOLDER
gcloud logging settings describe --folder=FOLDER_ID
ORGANIZATION
gcloud logging settings describe --organization=ORGANIZATION_ID
The previous command returns information about the default resource settings. For example, the following shows the default resource settings for a particular organization:
name: organizations/ORGANIZATION_ID/settings kmsKeyName: KMS_KEY_NAME kmsServiceAccountId: SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com storageLocation: europe-west1 disableDefaultSink: false
The value of the SERVICE_ACCT_NAME might have the format cmek-12345
or
service-12345@...
. If you can't use the Google Cloud CLI, then run the
Cloud Logging API method getSettings
.
Set the default storage location
Log buckets are the containers in your
Google Cloud projects, billing accounts, folders, and organizations that store
and organize your log data. For each Google Cloud project, billing account,
folder, and organization, Logging automatically creates two log
buckets: _Required
and _Default
, which are automatically stored in
an unspecified global
location.
You can specify a storage location for the _Required
and _Default
buckets that are contained by an organization or a folder by modifying the
default resource settings for Logging. This storage location also
determines where queries in the Logs Explorer and Log Analytics pages
are stored. These queries include recent queries that are automatically saved
after being run, and queries saved by members of the Google Cloud project.
For a list of the supported storage locations, see Supported regions.
After you configure the default storage location for an organization, the following happens:
Existing
_Required
and_Default
buckets in that organization or folder maintain the storage location that was assigned to them at the time they were created.For child resources created in the organization or the folder after the default storage location is configured, their
_Required
and_Default
buckets inherit the default storage location.Existing Logs Explorer or Log Analytics queries maintain their current storage location.
New Logs Explorer or Log Analytics queries that you save after the default storage location is configured use the default storage location. This location also applies to recent queries that are automatically saved.
The default storage location for Cloud Logging applies
only to the _Default
and _Required
log buckets, and to queries
in the Logs Explorer or Log Analytics pages. These queries include
queries that are automatically saved after being run, and queries saved by
members of the Google Cloud project. It doesn't apply to user-defined log
buckets, or to queries saved by using the Logging API, as a location
must be specified in the request.
Configure the organization policies
Logging supports organization policies that can restrict where data can be stored. If such a policy exists for your organization, then you can only create log buckets in locations that are allowed by the policy.
When an organization policy that specifies a location constraint exists, the policy values for the constraint must include the location specified in the default resource settings for Logging. Further, if you plan to modify your default resource settings, before you update the default resource settings, review and, if necessary, update the organization policies.
To view or update organization policies, do the following:
-
In the Google Cloud console, go to the Organization Policies page:
If you use the search bar to find this page, then select the result whose subheading is IAM & Admin.
Select your organization.
View, and if necessary, update the constraint with the ID
constraints/gcp.resourceLocations
. If this constraint isn't configured, then an update isn't required.For information about how to view specific constraints and how to edit these constraints, see Creating and editing policies.
Configure the default storage location for Logging
To configure the default storage location for Cloud Logging, run the
gcloud logging settings update
command and include the --storage-location
flag:
FOLDER
gcloud logging settings update --folder=FOLDER_ID--storage-location=LOCATION
ORGANIZATION
gcloud logging settings update --organization=ORGANIZATION_ID --storage-location=LOCATION
If you can't use the Google Cloud CLI, then run the
Cloud Logging API method updateSettings
.
For information about resolving errors when updating the default storage location, see Troubleshoot setting the default resource location.
Configure the _Default
sink
Logging provides a predefined
_Default
sink for each
Google Cloud project, billing account, folder, and organization resource. Any
log that is generated in the resource that matches the inclusion filter and
that isn't excluded, is routed to the resource's predefined, correspondingly
named _Default
bucket.
You can configure default resource settings for the _Default
sink for your
organization and folders with the following options:
You can disable the
_Default
sink for all child resources.You can configure an inclusion filter or several exclusion filters that apply to the
_Default
sinks of new projects.
Disable the _Default
sink
You can disable the creation of _Default
sinks for all new resources in
an organization or folder; disabling the _Default
sinks prevents
logs from being stored in the resource's _Default
bucket.
If you stop storing logs in a
resource's _Default
bucket, then the logs that would have been routed to that
bucket are excluded from storage in Logging, unless those logs
are explicitly included in another user-defined sink for that resource.
To disable the _Default
sinks for a resource and any of its child
resources, run the following
gcloud logging settings update
command:
FOLDER
gcloud logging settings update --folder=FOLDER_ID--disable-default-sink
ORGANIZATION
gcloud logging settings update --organization=ORGANIZATION_ID --disable-default-sink
The disable-default-sink
flag applies only to the _Default
sink that routes
logs into the _Default
bucket.
You can re-enable the _Default
sinks by running the following
gcloud logging settings update
command:
FOLDER
gcloud logging settings update --folder=FOLDER_ID--no-disable-default-sink
ORGANIZATION
gcloud logging settings update --organization=ORGANIZATION_ID --no-disable-default-sink
Configure default filter of _Default
sinks
The predefined _Default
sink routes any logs that match the sink criteria
to the corresponding _Default
bucket. You can use
inclusion filters
and exclusion filters to configure
which logs are included and excluded for new _Default
sinks in an organization
or folder.
The inclusion filter can either override or be
appended to the _Default
sink filter, and the exclusion filters are appended
as the _Default
sink has no exclusion filters by default.
To specify an inclusion filter or exclusion filter that is applied to all
_Default
sinks of new resources in an organization or folder,
run the Cloud Logging API method updateSettings
with
the defaultSinkConfig
object. You can only set the default filter of
_Default
sinks by using the Logging API.
You can execute the updateSettings
method by using the
APIs Explorer widget on the method's reference page. The
following example illustrates sample parameters:
- name (URL):
organizations/ORGANIZATION_ID/settings
- updateMask:
"default_sink_config"
Request body, which contains an instance of
Settings
:"defaultSinkConfig": { { "filter": "NOT LOG_ID(\"externalaudit.googleapis.com/activity\") " "AND NOT LOG_ID(\"cloudaudit.googleapis.com/system_event\") " "AND NOT LOG_ID(\"externalaudit.googleapis.com/system_event\") " "AND NOT LOG_ID(\"cloudaudit.googleapis.com/access_transparency\") " "AND NOT LOG_ID(\"externalaudit.googleapis.com/access_transparency\") ", "exclusions": [ { "name": "exclude-data-access", "description": "Prevents Data Access audit logs from being routed", "filter": "log_id(\"cloudaudit.googleapis.com/data_access\")", } ], "mode": OVERWRITE } }
The previous example does the following:
Overwrites the
_Default
sink's inclusion filter to include Admin Activity audit logs, which are excluded by default.Appends an exclusion filter that prevents Data Access audit logs from being routed to the
_Default
bucket.
Troubleshoot configuration errors
For troubleshooting information, see Troubleshoot CMEK and default setting errors.