Version 1.8. This version is supported as outlined in the Anthos version support policy, offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on bare metal. For more details, see the release notes 1.8. This is the most recent version. For a complete list of each minor and patch release in chronological order, see the combined release notes.

Available versions: 1.8  |   1.7  |   1.6

Enable Audit Logging

Overview

This document describes how to enable and use Cloud Audit Logs for Anthos clusters on bare metal. Anthos clusters on bare metal makes use of Kubernetes Audit Logging, which keeps a chronological record of calls made to a cluster's Kubernetes API server. Audit logs are useful for investigating suspicious API requests and for collecting statistics.

If Cloud Audit Logs is enabled, audit logs are written to Cloud Audit Logs in your Google Cloud project. Writing to Cloud Audit Logs has several benefits over writing to disk, or even capturing logs in an on-premises logging system:

  • Audit logs for all Anthos clusters can be centralized.
  • Log entries written to Cloud Audit Logs are immutable.
  • Cloud Audit Logs entries are retained for 400 days.
  • Cloud Audit Logs feature is included in the price of Anthos.
  • You can configure Anthos clusters on bare metal to write logs to disk or to Cloud Audit Logs.

Disk-based audit logging

By default, audit logs in Anthos clusters on bare metal are written to a persistent disk so that cluster restarts and upgrades don't cause the logs to disappear. Anthos clusters on bare metal retains up to 1 GB of audit log entries.

You can access the disk-based audit logs by logging into control plane Nodes. The logs are located in the /var/log/apiserver/ directory.

Cloud Audit Logs

If Cloud Audit Logs is enabled, then Admin Activity audit log entries from all Kubernetes API servers are sent to Google Cloud, using the project and location that you specify when you create a user cluster. To buffer and write log entries to Cloud Audit Logs, Anthos clusters on bare metal deploys an audit-proxy Pod to the admin cluster. This Pod is also available as a sidecar container on user clusters.

Limitations

Cloud Audit Logs for Anthos clusters on bare metal is a preview feature and has the following limitations:

  • Data access logging is not supported.
  • Modifying the Kubernetes audit policy is not supported.
  • Cloud Audit Logs is not resilient to extended network outages. If the log entries cannot be exported to Google Cloud, they are cached in a 10G disk buffer. If that buffer fills, then subsequent entries are dropped.
  • Cloud Audit Logs can be enabled when creating new 1.8.0 clusters only. Enabling Cloud Audit Logs on existing clusters through upgrading is not supported for the feature preview.

Creating a service account for Cloud Audit Logs

Before you can enable Cloud Logging and Cloud Monitoring with Anthos clusters on bare metal, you must first configure the following:

  1. Create a Cloud Monitoring Workspace within the Google Cloud project, if you don't have one already.

    This is done in the Cloud Console. Click the following button and follow the workflow.

    Go to Monitoring

  2. Click the following buttons to enable the required APIs:

    Enable the Anthos Audit API

    Enable the Stackdriver API

    Enable the Monitoring API

    Enable the Logging API

  3. Assign the following IAM roles to the service account used by the Stackdriver agents:

    • logging.logWriter
    • monitoring.metricWriter
    • stackdriver.resourceMetadata.writer
    • monitoring.dashboardEditor

Enabling Cloud Audit Logging when creating a cluster

To use Cloud Audit Logs with Anthos clusters on bare metal, follow the regular cluster creation instructions, but make the following edit to the cluster config file before executing the bmctl create cluster command:

  1. Uncomment the disableCloudAuditLogging field in the cluster config file and ensure it is set to false as shown in the following example:

    ...
    clusterOperations:
        # Cloud project for logs and metrics.
        projectID: <GCP project ID>
        # Cloud location for logs and metrics.
        location: us-central1
        # Enable Cloud Audit Logging if uncommented and set to false.
        disableCloudAuditLogging: false
    ...
    

    The rest of the cluster creation process is the same. For more information and links to instructions, see Creating clusters: overview.

Accessing Cloud Audit Logs

Console

  1. In the Cloud Console, go to the Logs page in the Logging menu.

    Go to the Logs page

  2. In the Filter by label or text search box, just above the drop-down menus discussed above, click the down arrow to open the drop-down menu. From the menu, choose Convert to advanced filter.

  3. Fill the text box with the following filter:

    resource.type="k8s_cluster"
    logName="projects/[PROJECT_ID]/logs/externalaudit.googleapis.com%2Factivity"
    protoPayload.serviceName="anthosgke.googleapis.com"
    

    Replace [PROJECT_ID] with your project ID.

  4. Click Submit Filter to display all audit logs from Anthos clusters on VMware clusters that were configured to log in to this project.

gcloud

List the first two log entries in your project's Admin Activity log that apply to the k8s_cluster resource type:

gcloud logging read \
    'logName="projects/[PROJECT_ID]/logs/externalaudit.googleapis.com%2Factivity" \
    AND resource.type="k8s_cluster" \
    AND protoPayload.serviceName="anthosgke.googleapis.com" ' \
    --limit 2 \
    --freshness 300d

Replace [PROJECT_ID] with your project ID.

The output shows two log entries. Notice that for each log entry, the logName field has the value projects/[PROJECT_ID]/logs/externalaudit.googleapis.com%2Factivity and protoPayload.serviceName is equal to anthosgke.googleapis.com.

Audit policy

Cloud Audit Logs behavior is determined by a statically-configured Kubernetes audit logging policy. Changing this policy isn't supported currently, but will be available in a future release.