This document describes how to enable and use Cloud Audit Logs for Google Distributed Cloud. Google Distributed Cloud uses Kubernetes Audit Logging, to keep a chronological record of calls made to a cluster's Kubernetes API server. Audit logs are useful for investigating suspicious API requests and for collecting statistics.
If Cloud Audit Logs is enabled, audit logs are written to Cloud Audit Logs in your Google Cloud project. Writing to Cloud Audit Logs has several benefits over writing to disk, or even capturing logs in an on-premises logging system:
- Audit logs for all GKE Enterprise clusters can be centralized.
- Log entries written to Cloud Audit Logs are immutable.
- Cloud Audit Logs entries are retained for 400 days.
- Cloud Audit Logs feature is included in the price of GKE Enterprise.
- You can configure Google Distributed Cloud to write logs to disk or to Cloud Audit Logs.
Disk-based audit logging
By default, audit logs in Google Distributed Cloud are written to a persistent disk so that cluster restarts and upgrades don't cause the logs to disappear. Google Distributed Cloud retains up to 1 GB of audit log entries.
You can access the disk-based audit logs by logging into control plane Nodes.
The logs are located in the /var/log/apiserver/
directory.
Cloud Audit Logs
If Cloud Audit Logs is enabled, then Admin Activity audit log entries from
all Kubernetes API servers are sent to Google Cloud. These log entries are
stored under the cluster's project name and location. To buffer and write
log entries to Cloud Audit Logs, Google Distributed Cloud deploys an
audit-proxy
Pod to the admin cluster. This Pod is also available as a sidecar
container on user clusters.
Limitations
Cloud Audit Logs for Google Distributed Cloud is a preview feature and has the following limitations:
- Data access logging isn't supported.
- Modifying the Kubernetes audit policy isn't supported.
- Cloud Audit Logs isn't resilient to extended network outages. If the log entries cannot be exported to Google Cloud, they are cached in a 10-GB disk buffer. If that buffer fills, then subsequent entries are dropped.
- Cloud Audit Logs can be enabled when creating new 1.8.0 clusters only. Enabling Cloud Audit Logs on existing clusters through upgrading isn't supported for the feature preview.
Create a service account for Cloud Audit Logs
Before you can enable Cloud Logging and Cloud Monitoring with Google Distributed Cloud, you must first configure the following:
Create a Cloud Monitoring Workspace within the Google Cloud project, if you don't have one already.
This is done in the Google Cloud console. Click the following button and follow the workflow.
Click the following buttons to enable the required APIs:
Assign the following IAM roles to the service account used by the Stackdriver agents:
logging.logWriter
monitoring.metricWriter
stackdriver.resourceMetadata.writer
monitoring.dashboardEditor
Enable Cloud Audit Logs when creating a cluster
To use Cloud Audit Logs with Google Distributed Cloud, follow the regular
cluster creation
instructions, but make the following edit to the cluster config file before
executing the bmctl create cluster
command:
Uncomment the
disableCloudAuditLogging
field in the cluster config file and ensure that it is set tofalse
as shown in the following example:... clusterOperations: # Cloud project for logs and metrics. projectID: `PROJECT_ID` # Cloud location for logs and metrics. location: us-central1 # Enable Cloud Audit Logging if uncommented and set to false. disableCloudAuditLogging: false ...
The rest of the cluster creation process is the same. For more information and links to instructions, see Creating clusters: overview.
Access Cloud Audit Logs
Console
In the Google Cloud console, go to the Logs page in the Logging menu.
In the Filter by label or text search box, click the down arrow to open the drop-down menu. From the menu, choose Convert to advanced filter.
Fill the text box with the following filter:
resource.type="k8s_cluster" logName="projects/PROJECT_ID/logs/externalaudit.googleapis.com%2Factivity" protoPayload.serviceName="anthosgke.googleapis.com"
Replace
PROJECT_ID
with your project ID.Click Submit Filter to display all audit logs from Google Distributed Cloud that were configured to log in to this project.
gcloud
List the first two log entries in your project's Admin Activity log that
apply to the k8s_cluster
resource type:
gcloud logging read \ 'logName="projects/PROJECT_ID/logs/externalaudit.googleapis.com%2Factivity" \ AND resource.type="k8s_cluster" \ AND protoPayload.serviceName="anthosgke.googleapis.com" ' \ --limit 2 \ --freshness 300d
Replace PROJECT_ID
with your project ID.
The output shows two log entries. Notice that for each log entry, the
logName
field has the value
projects/<var>PROJECT_ID</var>/logs/externalaudit.googleapis.com%2Factivity
and protoPayload.serviceName
is equal to anthosgke.googleapis.com
.
Audit policy
Cloud Audit Logs behavior is determined by a statically-configured Kubernetes audit logging policy. Changing this policy isn't supported currently, but will be available in a future release.