This document describes how to enable and use Cloud Audit Logs for Anthos clusters on bare metal. Anthos clusters on bare metal makes use of Kubernetes Audit Logging, which keeps a chronological record of calls made to a cluster's Kubernetes API server. Audit logs are useful for investigating suspicious API requests and for collecting statistics.
If Cloud Audit Logs is enabled, audit logs are written to Cloud Audit Logs in your Google Cloud project. Writing to Cloud Audit Logs has several benefits over writing to disk, or even capturing logs in an on-premises logging system:
- Audit logs for all Anthos clusters can be centralized.
- Log entries written to Cloud Audit Logs are immutable.
- Cloud Audit Logs entries are retained for 400 days.
- Cloud Audit Logs feature is included in the price of Anthos.
- You can configure Anthos clusters on bare metal to write logs to disk or to Cloud Audit Logs.
Disk-based audit logging
By default, audit logs in Anthos clusters on bare metal are written to a persistent disk so that cluster restarts and upgrades don't cause the logs to disappear. Anthos clusters on bare metal retains up to 1 GB of audit log entries.
You can access the disk-based audit logs by logging into control plane Nodes.
The logs are located in the
Cloud Audit Logs
If Cloud Audit Logs is enabled, then Admin Activity audit log entries from
all Kubernetes API servers are sent to Google Cloud, using the project and
location that you specify when you create a user cluster. To buffer and write
log entries to Cloud Audit Logs, Anthos clusters on bare metal deploys an
audit-proxy Pod to the admin cluster. This Pod is also available as a sidecar
container on user clusters.
Cloud Audit Logs for Anthos clusters on bare metal is a preview feature and has the following limitations:
- Data access logging is not supported.
- Modifying the Kubernetes audit policy is not supported.
- Cloud Audit Logs is not resilient to extended network outages. If the log entries cannot be exported to Google Cloud, they are cached in a 10G disk buffer. If that buffer fills, then subsequent entries are dropped.
- Cloud Audit Logs can be enabled when creating new 1.8.0 clusters only. Enabling Cloud Audit Logs on existing clusters through upgrading is not supported for the feature preview.
Creating a service account for Cloud Audit Logs
Before you can enable Cloud Logging and Cloud Monitoring with Anthos clusters on bare metal, you must first configure the following:
Create a Cloud Monitoring Workspace within the Google Cloud project, if you don't have one already.
This is done in the Cloud Console. Click the following button and follow the workflow.
Click the following buttons to enable the required APIs:
Assign the following IAM roles to the service account used by the Stackdriver agents:
Enabling Cloud Audit Logging when creating a cluster
To use Cloud Audit Logs with Anthos clusters on bare metal, follow the regular
instructions, but make the following edit to the cluster config file before
bmctl create cluster command:
disableCloudAuditLoggingfield in the cluster config file and ensure it is set to
falseas shown in the following example:
... clusterOperations: # Cloud project for logs and metrics. projectID: <GCP project ID> # Cloud location for logs and metrics. location: us-central1 # Enable Cloud Audit Logging if uncommented and set to false. disableCloudAuditLogging: false ...
The rest of the cluster creation process is the same. For more information and links to instructions, see Creating clusters: overview.
Accessing Cloud Audit Logs
In the Cloud Console, go to the Logs page in the Logging menu.
In the Filter by label or text search box, just above the drop-down menus discussed above, click the down arrow to open the drop-down menu. From the menu, choose Convert to advanced filter.
Fill the text box with the following filter:
resource.type="k8s_cluster" logName="projects/[PROJECT_ID]/logs/externalaudit.googleapis.com%2Factivity" protoPayload.serviceName="anthosgke.googleapis.com"
Replace [PROJECT_ID] with your project ID.
Click Submit Filter to display all audit logs from Anthos clusters on VMware clusters that were configured to log in to this project.
List the first two log entries in your project's Admin Activity log that
apply to the
k8s_cluster resource type:
gcloud logging read \ 'logName="projects/[PROJECT_ID]/logs/externalaudit.googleapis.com%2Factivity" \ AND resource.type="k8s_cluster" \ AND protoPayload.serviceName="anthosgke.googleapis.com" ' \ --limit 2 \ --freshness 300d
Replace [PROJECT_ID] with your project ID.
The output shows two log entries. Notice that for each log entry, the
logName field has the value
protoPayload.serviceName is equal to
Cloud Audit Logs behavior is determined by a statically-configured Kubernetes audit logging policy. Changing this policy isn't supported currently, but will be available in a future release.