Audit logging

Overview

Google Distributed Cloud (software only) for VMware supports audit logging at both the Google Cloud API and Kubernetes cluster level. This document provides information about Kubernetes cluster audit logging. For information about Google Cloud API audit logging, see Cloud API audit logging information.

Google Distributed Cloud makes use of Kubernetes Audit Logging, which keeps a chronological record of calls made to a cluster's Kubernetes API server. Audit logs are useful for investigating suspicious API requests and for collecting statistics.

You can configure a cluster to write audit logs to disk or to Cloud Audit Logs in a Google Cloud project. Writing to Cloud Audit Logs has several benefits over writing to disk, or even capturing logs in an on-premises logging system:

Audit logs for all GKE clusters can be centralized.
Log entries written to Cloud Audit Logs are immutable.
Cloud Audit Logs entries are retained for 400 days.
Cloud Audit Logs is included in the price of Anthos.

Disk-based audit logging

By default, audit logs are written to a persistent disk so that VM restarts and upgrades don't cause the logs to disappear.

If advanced cluster isn't enabled:

Google Distributed Cloud retains up to 12 GB of audit log entries.
If advanced cluster is enabled

Google Distributed Cloud retains up to 1 GB of audit log entries.

Cloud Audit Logs

If you enable Cloud Audit Logs for a cluster, then Admin Activity audit log entries from the cluster's Kubernetes API server are sent to Google Cloud, using the Google Cloud project that you specify in the cloudAuditLogging.projectID field of your cluster configuration file. This Google Cloud project is called your audit logging project.

Your audit logging project must be the same as your fleet host project.

To buffer and write log entries to Cloud Audit Logs, Google Distributed Cloud deploys an audit-proxy Pod to the admin cluster. This component is also available as a sidecar container on user clusters.

Limitations

The current version of Cloud Audit Logs for Google Distributed Cloud has several limitations:

Data access (get, list, watch requests) logging is not supported.
Modifying the Kubernetes audit policy is not supported.
Cloud Audit Logs is not resilient to extended network outages. If the log entries cannot be exported to Google Cloud, they are cached in a 10G disk buffer. If that buffer fills, then subsequent entries are dropped.
One project can support up to approximately 1000 service accounts for use with Cloud Audit Logs. Multiple clusters can use the same service account.

Enable the Anthos Audit API

Enable the Anthos Audit API in your audit logging project.

Enable the Anthos Audit API

Create a service account for Cloud Audit Logs

You already have one or more service accounts that you created to use with Google Distributed Cloud. For this feature, you need to create an additional service account called the audit logging service account.

Create your audit logging service account:

gcloud iam service-accounts create audit-logging-service-account

Create a JSON key file for your Cloud Audit Logs service account:
```
gcloud iam service-accounts keys create audit-logging-key.json \
   --iam-account AUDIT_LOGGING_SERVICE_ACCOUNT_EMAIL
```
where AUDIT_LOGGING_SERVICE_ACCOUNT_EMAIL is the email address of your service account.
Save audit-logging-key.json on the admin workstation in the same location as your other service account keys.

WARNING Before deleting this service account, be sure to replace it with the new service account in the cluster configuration first! The process is the similar to Enable Cloud Audit Logs for an existing user cluster. If you forget to do this, follow the guide to clean up.

Create an admin cluster with Cloud Audit Logs enabled

You can enable Cloud Audit Logs for an admin cluster only when you first create the admin cluster. You cannot modify an existing admin cluster to enable Cloud Audit Logs.

Refer to Creating an admin cluster.
In your admin cluster configuration file, fill in the cloudAuditLogging section.
Set cloudAuditLogging.projectID to the ID of your audit logging project.
Set cloudAuditLogging.clusterLocation to a Google Cloud region where you want to store audit logs. For improved latency, choose a region that is near your on-premises data center.
Set cloudAuditLogging.serviceAccountKeyPath to the path of the JSON key file for your audit logging service account.

For example:

cloudAuditLogging:
  projectID: "my-project"
  clusterLocation: "us-west1"
  serviceAccountKeyPath: "/my-key-folder/audit-logging-key.json"

Continue the cluster creation as usual.

Create a user cluster with Cloud Audit Logs enabled

Refer to Creating a user cluster.
In your user cluster configuration file, fill in the cloudAuditLogging section.
Set cloudAuditLogging.projectID to the ID of your audit logging project.
Set cloudAuditLogging.clusterLocation to a Google Cloud region where you want to store audit logs. For improved latency, choose a region that is near your on-premises data center.
Set cloudAuditLogging.serviceAccounKeyPath to the path of the JSON key file for your Cloud Audit Logs service account.
Ensure that the gkeConnect section is filled in and gkeConnect.projectID is the same as cloudAuditLogging.projectID.

For example:

gkeConnect:
  projectID: "my-project"
  registerServiceAccountKeyPath: "/my-key-fokder/connect-register-key.json"

cloudAuditLogging:
  projectID: "my-project"
  clusterLocation: "us-west1"
  serviceAccountKeyPath: "/my-key-folder/audit-logging-key.json"

Continue the cluster creation as usual.

Enable Cloud Audit Logs for an existing user cluster

Cloud Audit Logs can be enabled only in the Google Cloud project where the user cluster is registered.

If an existing user cluster is not registered, register it by following these steps before you enable Cloud Audit Logs:

Add a gkeConnect section to the user cluster configuration file. For example:

gkeConnect:
  projectID: "my-project"
  registerServiceAccountKeyPath: "/my-key-fokder/connect-register-key.json"

Update the cluster:

gkectl update cluster --config USER_CLUSTER_CONFIG --kubeconfig ADMIN_CLUSTER_KUBECONFIG

After the user cluster is registered, follow these steps to enable Cloud Audit Logs:

Fill in the cloudAuditLogging section of your user cluster configuration file. See Create a user cluster with Cloud Audit Logs enabled for details on the individual fields. The projectID in the cloudAuditLogging section must be the same as that in the gkeConnect section.

Update the cluster:

gkectl update cluster --config USER_CLUSTER_CONFIG --kubeconfig ADMIN_CLUSTER_KUBECONFIG

Disable Cloud Audit Logs for an existing user cluster

In the user cluster configuration file, delete the cloudAuditLogging section.
Update the user cluster:

gkectl update cluster --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] --config [USER_CLUSTER_CONFIG]

Access audit logs

Disk-based audit logging

You can find the audit logs for the admin cluster on the control-plane nodes under /var/log/kube-audit/kube-apiserver-audit.log. The audit logs for the user cluster are in the PersistentVolumeClaim named kube-audit-kube-apiserver-0. You can access this data within your own Pods via volumes entries:

Add this entry for the admin cluster:

    volumes:
    - name: kube-audit
      hostPath:
        path: /var/log/kube-audit
        type: ""

Add this entry for the user cluster:

    volumes:
    - name: kube-audit
      persistentVolumeClaim:
        claimName: kube-audit-kube-apiserver-0

To schedule your Pod on the appropriate admin cluster node (and only this node) you will need to add nodeSelector and tolerations sections to your Pod spec, like this:

    spec:
      nodeSelector:
        node-role.kubernetes.io/master: ''
      tolerations:
      - key: node-role.kubernetes.io/master
        value: ""
        effect: NoSchedule

For the user cluster, set namespace as the user cluster name, then set nodeName to the same as kube-apiserver-0:

   spec:
     nodeName: NODE_NAME

To point out the nodeName of kube-apiserver-0, run the following command:

kubectl get pod kube-apiserver-0 -n USER_CLUSTER_NAME --kubeconfig kubeconfig -o jsonpath='{.spec.nodeName}'

Each audit log's filename has a timestamp that indicates when the file was rotated. A file contains audit logs up to that time and date.

Cloud Audit Logs

Console

In the Google Cloud console, go to the Logs page in the Logging menu.

Go to the Logs page
In the Filter by label or text search box, just above the drop-down menus discussed above, click the down arrow to open the drop-down menu. From the menu, choose Convert to advanced filter.

Fill the field with the following filter:

resource.type="k8s_cluster"
logName="projects/PROJECT_ID/logs/externalaudit.googleapis.com%2Factivity"
protoPayload.serviceName="anthosgke.googleapis.com"

Click Submit Filter to display all audit logs from clusters that were configured to log in to this project.

gcloud

List the first two log entries in your project's Admin Activity log that apply to the k8s_cluster resource type:

gcloud logging read \
    'logName="projects/PROJECT_ID/logs/externalaudit.googleapis.com%2Factivity" \
    AND resource.type="k8s_cluster" \
    AND protoPayload.serviceName="anthosgke.googleapis.com" ' \
    --limit 2 \
    --freshness 300d

where PROJECT_ID is your project ID.

The output shows two log entries. Notice that for each log entry, the logName field has the value projects/PROJECT_ID/logs/externalaudit.googleapis.com%2Factivity and protoPayload.serviceName is equal to anthosgke.googleapis.com.

Audit policy

Cloud Audit Logs behavior is determined by a statically-configured Kubernetes audit logging policy. Changing this policy is currently not supported, but will be available in a future release.