Enable audit logging

This document describes how to enable and use Cloud Audit Logs for GKE on Bare Metal. GKE on Bare Metal uses Kubernetes Audit Logging, to keep a chronological record of calls made to a cluster's Kubernetes API server. Audit logs are useful for investigating suspicious API requests and for collecting statistics.

If Cloud Audit Logs is enabled, audit logs are written to Cloud Audit Logs in your Google Cloud project. Writing to Cloud Audit Logs has several benefits over writing to disk, or even capturing logs in an on-premises logging system:

  • Audit logs for all GKE Enterprise clusters can be centralized.
  • Log entries written to Cloud Audit Logs are immutable.
  • Cloud Audit Logs entries are retained for 400 days.
  • Cloud Audit Logs feature is included in the price of GKE Enterprise.
  • You can configure GKE on Bare Metal to write logs to disk or to Cloud Audit Logs.

Disk-based audit logging

By default, audit logs in GKE on Bare Metal are written to a persistent disk so that cluster restarts and upgrades don't cause the logs to disappear. GKE on Bare Metal retains up to 1 GB of audit log entries.

You can access the disk-based audit logs by logging into control plane Nodes. The logs are located in the /var/log/apiserver/ directory.

Cloud Audit Logs

If Cloud Audit Logs is enabled, then Admin Activity audit log entries from all Kubernetes API servers are sent to Google Cloud. These log entries are stored under the cluster's project name and location. To buffer and write log entries to Cloud Audit Logs, GKE on Bare Metal deploys an audit-proxy Pod to the admin cluster. This Pod is also available as a sidecar container on user clusters.

Limitations

Cloud Audit Logs for GKE on Bare Metal is a preview feature and has the following limitations:

  • Data access logging isn't supported.
  • Modifying the Kubernetes audit policy isn't supported.
  • Cloud Audit Logs isn't resilient to extended network outages. If the log entries cannot be exported to Google Cloud, they are cached in a 10-GB disk buffer. If that buffer fills, then subsequent entries are dropped.
  • Cloud Audit Logs can be enabled when creating new 1.8.0 clusters only. Enabling Cloud Audit Logs on existing clusters through upgrading isn't supported for the feature preview.

Create a service account for Cloud Audit Logs

Before you can enable Cloud Logging and Cloud Monitoring with GKE on Bare Metal, you must first configure the following:

  1. Create a Cloud Monitoring Workspace within the Google Cloud project, if you don't have one already.

    This is done in the Google Cloud console. Click the following button and follow the workflow.

    Go to Monitoring

  2. Click the following buttons to enable the required APIs:

    Enable the Anthos Audit API

    Enable the Stackdriver API

    Enable the Monitoring API

    Enable the Logging API

  3. Assign the following IAM roles to the service account used by the Stackdriver agents:

    • logging.logWriter
    • monitoring.metricWriter
    • stackdriver.resourceMetadata.writer
    • monitoring.dashboardEditor

Enable Cloud Audit Logs when creating a cluster

To use Cloud Audit Logs with GKE on Bare Metal, follow the regular cluster creation instructions, but make the following edit to the cluster config file before executing the bmctl create cluster command:

  1. Uncomment the disableCloudAuditLogging field in the cluster config file and ensure that it is set to false as shown in the following example:

    ...
clusterOperations:
    # Cloud project for logs and metrics.
    projectID: `PROJECT_ID`
    # Cloud location for logs and metrics.
    location: us-central1
    # Enable Cloud Audit Logging if uncommented and set to false.
    disableCloudAuditLogging: false
...

    The rest of the cluster creation process is the same. For more information and links to instructions, see Creating clusters: overview.

Access Cloud Audit Logs

Console

  1. In the Google Cloud console, go to the Logs page in the Logging menu.

    Go to the Logs page

  2. In the Filter by label or text search box, click the down arrow to open the drop-down menu. From the menu, choose Convert to advanced filter.

  3. Fill the text box with the following filter:

    resource.type="k8s_cluster"
logName="projects/PROJECT_ID/logs/externalaudit.googleapis.com%2Factivity"
protoPayload.serviceName="anthosgke.googleapis.com"

    Replace PROJECT_ID with your project ID.

  4. Click Submit Filter to display all audit logs from GKE on VMware that were configured to log in to this project.

gcloud

List the first two log entries in your project's Admin Activity log that apply to the k8s_cluster resource type:

gcloud logging read \
    'logName="projects/PROJECT_ID/logs/externalaudit.googleapis.com%2Factivity" \
    AND resource.type="k8s_cluster" \
    AND protoPayload.serviceName="anthosgke.googleapis.com" ' \
    --limit 2 \
    --freshness 300d

Replace PROJECT_ID with your project ID.

The output shows two log entries. Notice that for each log entry, the logName field has the value
projects/<var>PROJECT_ID</var>/logs/externalaudit.googleapis.com%2Factivity and protoPayload.serviceName is equal to anthosgke.googleapis.com.

Audit policy

Cloud Audit Logs behavior is determined by a statically-configured Kubernetes audit logging policy. Changing this policy isn't supported currently, but will be available in a future release.