Audit Logging

This page explains how to use audit logging in your Google Kubernetes Engine clusters.

Overview

All Kubernetes-powered clusters have Kubernetes Audit Logging, which keeps a chronological record of calls that have been made to the Kubernetes API server. Kubernetes audit log entries are useful for investigating suspicious API requests, for collecting statistics, or for creating monitoring alerts for unwanted API calls.

GKE clusters integrate Kubernetes Audit Logging with Cloud Audit Logs and Stackdriver Logging. You can see Kubernetes audit log entries in your Google Cloud project.

In addition to entries written by Kubernetes, your project's audit logs have entries written by Kubernetes Engine.

Audit Logging GA is available in GKE 1.11.4 and later.

Before you begin

To prepare for this task, perform the following steps:

• Ensure that you have enabled the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• Ensure that you have installed the Cloud SDK.
• Set your default project ID:

  gcloud config set project [PROJECT_ID]

• If you are working with zonal clusters, set your default compute zone:

  gcloud config set compute/zone [COMPUTE_ZONE]

• If you are working with regional clusters, set your default compute region:

  gcloud config set compute/region [COMPUTE_REGION]

• Update gcloud to the latest version:

  gcloud components update

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region operational flags.

You need to have a Kubernetes Engine cluster in your project. You can use an existing cluster, or you can create a new one for the exercises in this topic. If you choose to use an existing cluster, make sure the cluster has had some recent activity. For example, if you haven't created a Deployment recently, you could create a Deployment now by entering this command:

kubectl run log-exercise --image nginx

Read about Kubernetes Audit Logging.

Audit logs in your project

Your Google Cloud project has these audit logs:

• Admin Activity log
• Data Access log

Admin Activity logging is enabled by default and has no extra cost.

Data Access logging is disabled by default, and enabling it can result in extra billing. To learn more about enabling Data Access logging, and the associated costs, see Configuring Data Access Logs.

Kubernetes Engine does not support Access Transparency logging.

Various Google Cloud Platform services write entries to your project's logs. The Kubernetes service also writes entries to your project's audit logs. For Kubernetes Engine clusters, log entries written by these services are the most relevant:

Service	Display name
k8s.io	Kubernetes
container.googleapis.com	Kubernetes Engine

Viewing your project's Admin Activity log

Console

1. In the Cloud Console, go to the Logs page in the Logging menu.

   Go to the Logs page

2. Near the top of the page, locate the drop-down menu for selecting a resource type. From the drop-down menu, select Kubernetes Cluster.

3. Optionally, specify the Location or choose All location. If you select a location, you can select a specific cluster in that location.

4. The next menu to the right is for selecting a log. From the drop-down menu, select activity, and click OK.

5. The display shows all log levels by default. To specify a log level, such as Warning, select it from the drop-down menu representing log levels.

6. The display might only show log entries from the last hour. If you don't see any log entries from the last hour, click Load older logs.

7. In the Filter by label or text search box, just above the drop-down menus discussed above, click the down arrow to open the drop-down menu. From the menu, choose Convert to advanced filter.

8. The text box displays a filter similar to this:

   resource.type="k8s_cluster"
   logName="projects/my-project/logs/cloudaudit.googleapis.com%2Factivity"
   

9. Open one of the log entries by clicking the arrow at the beginning of the line. The logName field of the entry has the value projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity.

gcloud

List the first two log entries in your project's Admin Activity log:

gcloud logging read \
    'logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"' \
    --limit 2 \
    --freshness 300d

where [PROJECT_ID] is your project ID.

The output shows two log entries. Notice that for each log entry, the logName field has the value projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity.

insertId: 18yao5jem14og
labels:
  cluster_version: 1.8.8-gke.0
logName: projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity
...

Basic and advanced filter interfaces

In the Cloud Console, the Logs page has two filtering interfaces: basic and advanced. For information about the two filtering interfaces, see Logs Viewer filter interfaces.

Viewing services that write to your Admin Activity log

Console

1. In the Cloud Console, go to the Logs page in the Logging menu.

   Go to the Logs page

2. If the Logs page is not already in advanced mode, switch to advanced mode. In the filter box, click the down arrow at the right, and select Convert to advanced filter.

3. In the filter box, delete any existing text, and enter this filter.

   logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
   protoPayload.serviceName="k8s.io"
   

   where [PROJECT_ID] is your project ID.

   Click Submit filter.

   The display lists all entries in your Admin Activity log that were written by the k8s.io service. That is, the entries were written by the Kubernetes control plane.

4. Now look for log entries that were written by services other than k8s.io. In the filter box, change serviceName="k8s.io" to serviceName!="k8s.io". Click Submit Filter.

   The display lists log entries in your Admin Activity log that were not written by the k8s.io service.

5. Open one of the log entries, and expand the protoPayload field. Look at the value of serviceName to see which service wrote the log entry.

6. Now look for log entries that were written by the container.googleapis.com. service. That is, the entries were written by the Kubernetes Engine control plane. In the filter box, change serviceName!="k8s.io" to serviceName="container.googleapis.com". Click Submit Filter.

gcloud

List the first two log entries in your project's Admin Activity log that were written by the k8s.io service. That is the entries were written by the Kubernetes control plane.

gcloud logging read \
    'logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
        AND protoPayload.serviceName="k8s.io"' \
    --limit 2 \
    --freshness 300d

where [PROJECT_ID] is your project ID.

In the output, you can see that protoPayload:serviceName has the value k8s.io:

protoPayload:
  ...
  serviceName: k8s.io

Now look for log entries that were written by services other than k8s.io:

gcloud logging read \
    'logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
        AND protoPayload.serviceName!="k8s.io"' \
    --limit 2 \
    --freshness 300d

The output lists log entries in your Admin Activity log that were not written by the k8s.io service.

logName: projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity
...
protoPayload:
  ...
  serviceName: compute.googleapis.com
...

In one of the log entries, look at the value of protoPayload.serviceName to see which service wrote the log entry.

Now look for log entries that were written by the container.googleapis.com service. That is, the entries were written by the Kubernetes Engine control plane:

gcloud logging read \
    'logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
        AND protoPayload.serviceName="container.googleapis.com"' \
    --limit 2 \
    --freshness 300d

where [PROJECT_ID] is your project ID.

Filtering your Admin Activity log by resource type

Each log entry in your Admin Activity log applies to a certain type of resource. These are the resource types that are the most relevant to Kubernetes clusters:

Resource type	Display name
k8s_cluster	Kubernetes Cluster
gke_cluster	GKE Cluster Operations

Log entries written by the Kubernetes API server apply to the k8s_cluster resource type. These log entries describe operations on Kubernetes resources in your cluster, for example, Pods, Deployments, and Secrets.

Log entries written by the Kubernetes Engine API server apply to the gke_cluster resource. These log entries describe operations like cluster creation and deletion.

Console

1. In the Cloud Console, go to the Logs page in the Logging menu.

   Go to the Logs page

2. If the Logs page is not already in basic mode, switch to basic mode. In the filter box, click the down arrow at the right, and select Clear filters and return to basic mode.

3. From the drop-down menu for selecting a resource type, select Kubernetes Cluster. This is the display name for the k8s_cluster resource type.

4. From the drop-down menu for selecting a log, select activity, and click OK.

   The display lists all log entries in your Admin Activity log that apply to the k8s_cluster resource type.

5. Open one of the log entries, and expand the resource field. Verify that the type field has the value k8s_cluster.

6. Now from the drop-down menu, select GKE Cluster Operations. This is the display name for the gke_cluster resource type. From the drop-down menu for selecting a log, select activity, and click OK.

   The display lists all log entries in your Admin Activity log that apply to the gke_cluster resource type.

gcloud

List the first two log entries in your project's Admin Activity log that apply to the k8s_cluster resource type:

gcloud logging read \
    'logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
        AND resource.type="k8s_cluster"' \
    --limit 2 \
    --freshness 300d

where [PROJECT_ID] is your project ID.

In the output, you can see that the resource:type field has the value k8s_cluster:

resource:
  ...
  type: k8s_cluster

Next list the first two log entries in your project's Admin Activity log that apply to the gke_cluster resource type:

gcloud logging read \
    'logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
        AND resource.type="gke_cluster"' \
    --limit 2 \
    --freshness 300d

In the output, you can see that the resource:type field has the value gke_cluster:

resource:
  ...
  type: gke_cluster

Example filters for your Admin Activity log

Here are some examples of filters that you can try in the Cloud Console. In each case, replace [PROJECT_ID] with your project ID.

Find changes to Role Based Access Control, excluding automated system changes.

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.methodName:"io.k8s.authorization.rbac.v1"
NOT protoPayload.authenticationInfo.principalEmail:"system"

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.methodName:"io.k8s.authorization.rbac.v1.roles"
NOT protoPayload.authenticationInfo.principalEmail:"system"

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.methodName:"io.k8s.authorization.rbac.v1.rolebindings"
NOT protoPayload.authenticationInfo.principalEmail:"system"

You can use similar queries to find changes to clusterroles and clusterrolebindings.

Find certificate signing requests.

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.resourceName:"certificates.k8s.io/v1beta1/certificatesigningrequests"

Find unauthenticated web requests.

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.authenticationInfo.principalEmail:"system:anonymous"

Find kubelet bootstrap identity calls.

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.authenticationInfo.principalEmail:"kubelet"

Find node authenticated requests.

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.authenticationInfo.principalEmail:"system:node"

Find calls outside an IP range.

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.requestMetadata.callerIp!="127.0.0.1"
protoPayload.requestMetadata.callerIp!="::1"
NOT protoPayload.requestMetadata.callerIp:"[IP Prefix]"

Find entries in your Admin Activity log that apply to the k8s_cluster resource type and describe creating a Deployment.

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.methodName:"deployments.create"

Find entries in your Admin Activity log that apply to the k8s_cluster resource type and have a principalEmail value of system:anonymous. These entries probably represent failed attempts to authenticate.

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.authenticationInfo.principalEmail="system:anonymous"

Find entries in your Admin Activity log that apply to the gke_cluster resource type and describe cluster creation:

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="gke_cluster"
protoPayload.methodName="google.container.v1.ClusterManager.CreateCluster"

Find entries in your Admin Activity log that apply to the gke_cluster resource type and have a severity value of ERROR:

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="gke_cluster"
severity="ERROR"

Find entries in your Admin Activity log that apply to the k8s_cluster resource type and describe a write request to a Secret:

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.methodName:"io.k8s.core.v1.secrets"
NOT protoPayload.methodName:"get"
NOT protoPayload.methodName:"list"
NOT protoPayload.methodName:"watch"

Find entries in your Admin Activity log that apply to the k8s_cluster resource type and describe a Pod request from a particular user:

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.methodName:"io.k8s.core.v1.pods"
protoPayload.authenticationInfo.principalEmail="[e-mail]"

For more information about how to construct filters, see Advanced Logs Filters.

Structure of a log entry

Every log entry is an object of type LogEntry. For more information, go to Understanding audit logs.

Enabling data access logs

Get the Cloud Identity and Access Management (Cloud IAM) policy for your project:

gcloud projects get-iam-policy [PROJECT_ID] > my-policy.yaml

where [PROJECT_ID] is your project ID.

Open my-policy.yaml to view your Cloud IAM policy. Your policy probably contains a bindings object similar to this:

bindings:
- members:
  - serviceAccount:xxx.gserviceaccount.com
  - serviceAccount:yyy.gserviceaccount.com
  role: roles/container.clusterAdmin
- members:
  ...

In my-policy.yaml, create an auditConfigs object, or add to your existing auditConfigs object, so that ADMIN_READ, DATA_WRITE, AND DATA_READ are listed under auditLogConfigs.

auditConfigs:
- auditLogConfigs:
  - logType: ADMIN_READ
  - logType: DATA_WRITE
  - logType: DATA_READ
  service: allServices

Do not change the value of etag.

Save your updated file as my-policy-2.yaml.

Caution: Do not change any part of your policy that is not related to audit logging. You are editing a policy object that contains critical information about who can access your project or organization. Accidentally altering that information could make your project or organization unusable.

Set the Cloud IAM policy for your project:

gcloud projects set-iam-policy [PROJECT_ID] my-policy-2.yaml

where [PROJECT_ID] is your project ID.

Viewing your project's Data Access log

Console

1. In the Cloud Console, go to the Logs page in the Logging menu.

   Go to the Logs page

2. If the Logs page is not already in basic mode, switch to basic mode. In the filter box, click the down arrow at the right, and select Clear filters and return to basic mode.

3. Near the top of the page, locate the drop-down menu for selecting a resource type. From the drop-down menu, select Kubernetes Cluster.

4. In the menu for selecting a log, select data_access, and click OK.

5. In the Filter by label or text search box, at the right side, click the down arrow to open the drop-down menu. From the menu, choose Convert to advanced filter.

6. The text box displays a filter similar to this:

   resource.type="k8s_cluster"
   logName="projects/my-project/logs/cloudaudit.googleapis.com%2Fdata_access"

7. Open one of the log entries, and notice that the logName field of the entry has the value projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Fdata_access.

gcloud

List the first two log entries in your project's Data Access log:

gcloud logging read \
    'logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Fdata_access"' \
    --limit 2 \
    --freshness 30d

where [PROJECT_ID] is your project ID.

The output shows two log entries. Notice that for each log entry, the logName field has the value projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Fdata_access.

insertId:  "x0vy9eej0j54"
labels: {…}
logName:  "projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Fdata_access"
...

Exporting and storing log entries

Log entries are held in Stackdriver Logging for a limited time known as the retention period. After that, the entries are deleted.

If you want to keep your log entries longer, you can export them to a Google service like Cloud Storage, BigQuery, or Cloud Pub/Sub.

Setting up metrics and alerts

You can use Stackdriver Monitoring to set up metrics based on your log entries. And you can use log-based metrics to set up charts and alerts.

Audit policy

The Kubernetes audit policy determines which log entries are exported by the Kubernetes API server. The Kubernetes Engine audit policy determines which entries go to your Admin Activity log and which entries go to your Data Access log.

For more information about audit policies in Kubernetes Engine, see Kubernetes Engine Audit Policy.

What's next

• Kubernetes Audit Logging
• Kubernetes Engine Audit Policy
• Kubernetes Engine Security Overview
• [Cloud Audit Logging]