Audit Logging

This page explains how to use audit logging in your Kubernetes Engine clusters.

Overview

All Kubernetes-powered clusters have Kubernetes Audit Logging, which keeps a chronological record of calls that have been made to the Kubernetes API server. Kubernetes audit log entries are useful for investigating suspicious API requests, for collecting statistics, or for creating monitoring alerts for unwanted API calls.

Kubernetes Engine clusters integrate Kubernetes Audit Logging with Cloud Audit Logging and Stackdriver Logging. You can see Kubernetes audit log entries in your Google Cloud Platform project.

In addition to entries written by Kubernetes, your project's audit logs have entries written by Kubernetes Engine.

Before you begin

To prepare for this task, perform the following steps:

  • Ensure that you have enabled the Kubernetes Engine API.
  • Enable Kubernetes Engine API
  • Ensure that you have installed the Cloud SDK.
  • Set your default project ID:
    gcloud config set project [PROJECT_ID]
  • Set your default compute zone:
    gcloud config set compute/zone [COMPUTE_ZONE]
  • Update all gcloud commands to the latest version:
    gcloud components update

You need to have a Kubernetes Engine cluster in your project. You can use an existing cluster, or you can create a new one for the exercises in this topic. If you choose to use an existing cluster, make sure the cluster has had some recent activity. For example, if you haven't created a Deployment recently, you could create a Deployment now by entering this command:

kubectl run log-exercise --image nginx

Read about Kubernetes Audit Logging.

Audit logs in your project

Your GCP project has these audit logs:

  • Admin Activity log
  • Data Access log

Admin Activity logging is enabled by default and has no extra cost.

Data Access logging is disabled by default, and enabling it can result in extra billing. To learn more about enabling Data Access logging, and the associated costs, see Configuring Data Access Logs.

Kubernetes Engine does not support Access Transparency logging.

Various Google Cloud Platform services write entries to your project's logs. The Kubernetes service also writes entries to your project's audit logs. For Kubernetes Engine clusters, log entries written by these services are the most relevant:

Service Display name
k8s.io Kubernetes
container.googleapis.com Kubernetes Engine

Viewing your project's Admin Activity log

Console

  1. In the GCP Console, go to the Logs page.

    Go to the Logs page

  2. Near the top of the page, locate the drop-down menu for selecting a resource type. From the drop-down menu, select Kubernetes Cluster.

  3. The next menu to the right is for selecting a log. From the drop-down menu, select activity, and click OK.

  4. The display might only show log entries from the last hour. If you don't see any log entries from the last hour, click Load older logs.

  5. In the Filter by label or text search box, at the right side, click the down arrow to open the drop-down menu. From the menu, choose Convert to advanced filter.

  6. The text box displays a filter similar to this:

    resource.type="k8s_cluster"
    logName="projects/my-project/logs/cloudaudit.googleapis.com%2Factivity"
    

  7. Open one of the log entries. Note that the logName field of the entry has the value projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity.

gcloud

List the first two log entries in your project's Admin Activity log:

gcloud logging read \
    'logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"' \
    --limit 2 \
    --freshness 300d

where [PROJECT_ID] is your project ID.

The output shows two log entries. Notice that for each log entry, the logName field has the value projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity.

insertId: 18yao5jem14og
labels:
  cluster_version: 1.8.8-gke.0
logName: projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity
...

Basic and advanced filtering modes

In the GCP Console, the Logs page has two filtering modes: basic and advanced.

In basic mode, you can use drop-down menus to filter the display according to resource type, log name, and log level. In advanced mode, you can do fine-grained filtering by entering a custom filter in a text box.

Determining which mode is active

On the Logs page, in the filter box, click the down arrow at the right. In the menu items, if you see Convert to advanced filter, the page is in basic mode. If you see Clear filters and return to basic mode, the page is in advanced mode.

Changing from one filtering mode to the other

To change from one mode to the other, click the down arrow at the right of the filter box, and select either Convert to advance filter or Clear filters and return to basic mode.

Viewing services that write to your Admin Activity log

Console

  1. In the GCP Console, go to the Logs page.

    Go to the Logs page

  2. If the Logs page is not already in advanced mode, switch to advanced mode. In the filter box, click the down arrow at the right, and select Convert to advanced filter.

  3. In the filter box, delete any existing text, and enter this filter.

    logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
    protoPayload.serviceName="k8s.io"
    

    where [PROJECT_ID] is your project ID.

    Click Submit filter.

    The display lists all entries in your Admin Activity log that were written by the k8s.io service. That is, the entries were written by the Kubernetes control plane.

  4. Now look for log entries that were written by services other than k8s.io. In the filter box, change serviceName="k8s.io" to serviceName!="k8s.io". Click Submit Filter.

    The display lists log entries in your Admin Activity log that were not written by the k8s.io service.

  5. Open one of the log entries, and expand the protoPayload field. Look at the value of serviceName to see which service wrote the log entry.

  6. Now look for log entries that were written by the container.googleapis.com. service. That is, the entries were written by the Kubernetes Engine control plane. In the filter box, change serviceName!="k8s.io" to serviceName="container.googleapis.com". Click Submit Filter.

gcloud

List the first two log entries in your project's Admin Activity log that were written by the k8s.io service. That is the entries were written by the Kubernetes control plane.

gcloud logging read \
    'logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
        AND protoPayload.serviceName="k8s.io"' \
    --limit 2 \
    --freshness 300d

where [PROJECT_ID] is your project ID.

In the output, you can see that protoPayload:serviceName has the value k8s.io:

protoPayload:
  ...
  serviceName: k8s.io

Now look for log entries that were written by services other than k8s.io:

gcloud logging read \
    'logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
        AND protoPayload.serviceName!="k8s.io"' \
    --limit 2 \
    --freshness 300d

The output lists log entries in your Admin Activity log that were not written by the k8s.io service.

logName: projects/steve-70/logs/cloudaudit.googleapis.com%2Factivity
...
protoPayload:
  ...
  serviceName: compute.googleapis.com
...

In one of the log entries, look at the value of protoPayload.serviceName to see which service wrote the log entry.

Now look for log entries that were written by the container.googleapis.com service. That is, the entries were written by the Kubernetes Engine control plane:

gcloud logging read \
    'logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
        AND protoPayload.serviceName="container.googleapis.com"' \
    --limit 2 \
    --freshness 300d

where [PROJECT_ID] is your project ID.

Filtering your Admin Activity log by resource type

Each log entry in your Admin Activity log applies to a certain type of resource. These are the resource types that are the most relevant to Kubernetes clusters:

Resource type Display name
k8s_cluster Kubernetes Cluster
gke_cluster GKE Cluster Operations

Log entries written by the Kubernetes API server apply to the k8s_cluster resource type. These log entries describe operations on Kubernetes resources in your cluster, for example, Pods, Deployments, and Secrets.

Log entries written by the Kubernetes Engine API server apply to the gke_cluster resource. These log entries describe operations like cluster creation and deletion.

Console

  1. In the GCP Console, go to the Logs page.

    Go to the Logs page

  2. If the Logs page is not already in basic mode, switch to basic mode. In the filter box, click the down arrow at the right, and select Clear filters and return to basic mode.

  3. From the drop-down menu for selecting a resource type, select Kubernetes Cluster. This is the display name for the k8s_cluster resource type.

  4. From the drop-down menu for selecting a log, select activity, and click OK.

    The display lists all log entries in your Admin Activity log that apply to the k8s_cluster resource type.

  5. Open one of the log entries, and expand the resource field. Verify that the type field has the value k8s_cluster.

  6. Now from the drop-down menu, select GKE Cluster Operations. This is the display name for the gke_cluster resource type. From the drop-down menu for selecting a log, select activity, and click OK.

    The display lists all log entries in your Admin Activity log that apply to the gke_cluster resource type.

gcloud

List the first two log entries in your project's Admin Activity log that apply to the k8s_cluster resource type:

gcloud logging read \
    'logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
        AND resource.type="k8s_cluster"' \
    --limit 2 \
    --freshness 300d

where [PROJECT_ID] is your project ID.

In the output, you can see that the resource:type field has the value k8s_cluster:

resource:
  ...
  type: k8s_cluster

Next list the first two log entries in your project's Admin Activity log that apply to the gke_cluster resource type:

gcloud logging read \
    'logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
        AND resource.type="gke_cluster"' \
    --limit 2 \
    --freshness 300d

In the output, you can see that the resource:type field has the value gke_cluster:

resource:
  ...
  type: gke_cluster

Example filters for your Admin Activity log

Here are some examples of filters that you can try in the GCP Console. In each case, replace [PROJECT_ID] with your project ID.

Find entries in your Admin Activity log that apply to the k8s_cluster resource type and describe creating a Deployment.

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.methodName:"deployments.create"

Find entries in your Admin Activity log that apply to the k8s_cluster resource type and have a principalEmail value of system:anonymous. These entries probably represent failed attempts to authenticate.

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.authenticationInfo.principalEmail="system:anonymous"

Find entries in your Admin Activity log that apply to the gke_cluster resource type and describe cluster creation:

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="gke_cluster"
protoPayload.methodName="google.container.v1.ClusterManager.CreateCluster"

Find entries in your Admin Activity log that apply to the gke_cluster resource type and have a severity value of ERROR:

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="gke_cluster"
severity="ERROR"

Find entries in your Admin Activity log that apply to the k8s_cluster resource type and describe a write request to a Secret:

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.methodName:"io.k8s.core.v1.secrets"
NOT protoPayload.methodName:"get"
NOT protoPayload.methodName:"list"
NOT protoPayload.methodName:"watch"

Find entries in your Admin Activity log that apply to the k8s_cluster resource type and describe a Pod request from a particular user:

logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.methodName:"io.k8s.core.v1.pods"
protoPayload.authenticationInfo.principalEmail="dev@example.com"

For more information about how to construct filters, see Advanced Logs Filters.

Structure of a log entry

Every log entry is an object of type LogEntry. For more information, see Audit Log Datatypes.

Enabling data access logs

Get the Identity and Access Management (IAM) policy for your project:

gcloud projects get-iam-policy [PROJECT_ID] > my-policy.yaml

where [PROJECT_ID] is your project ID.

Open my-policy.yaml to view your IAM policy. Your policy probably contains a bindings object similar to this:

bindings:
- members:
  - serviceAccount:xxx.gserviceaccount.com
  - serviceAccount:yyy.gserviceaccount.com
  role: roles/container.clusterAdmin
- members:
  ...

In my-policy.yaml, create an auditConfigs object, or add to your existing auditConfigs object, so that ADMIN_READ, DATA_WRITE, AND DATA_READ are listed under auditLogConfigs.

auditConfigs:
- auditLogConfigs:
  - logType: ADMIN_READ
  - logType: DATA_WRITE
  - logType: DATA_READ
  service: allServices

Do not change the value of etag.

Save your updated file as my-policy-2.yaml.

Set the IAM policy for your project:

gcloud projects set-iam-policy [PROJECT_ID] my-policy-2.yaml

where [PROJECT_ID] is your project ID.

Viewing your project's Data Access log

Console

  1. In the GCP Console, go to the Logs page.

    Go to the Logs page

  2. If the Logs page is not already in basic mode, switch to basic mode. In the filter box, click the down arrow at the right, and select Clear filters and return to basic mode.

  3. Near the top of the page, locate the drop-down menu for selecting a resource type. From the drop-down menu, select Kubernetes Cluster.

  4. In the menu for selecting a log, select data_access, and click OK.

  5. In the Filter by label or text search box, at the right side, click the down arrow to open the drop-down menu. From the menu, choose Convert to advanced filter.

  6. The text box displays a filter similar to this:

    resource.type="k8s_cluster"
    logName="projects/my-project/logs/cloudaudit.googleapis.com%2Fdata_access"
    
  7. Open one of the log entries, and notice that the logName field of the entry has the value projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Fdata_access.

gcloud

List the first two log entries in your project's Data Access log:

gcloud logging read \
    'logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Fdata_access"' \
    --limit 2 \
    --freshness 30d

where [PROJECT_ID] is your project ID.

The output shows two log entries. Notice that for each log entry, the logName field has the value projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Fdata_access.

insertId:  "x0vy9eej0j54"
labels: {…}
logName:  "projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Fdata_access" 
...

Exporting and storing log entries

Log entries are held in Stackdriver Logging for a limited time known as the retention period. After that, the entries are deleted.

If you want to keep your log entries longer, you can export them to a Google service like Cloud Storage, BigQuery, or Cloud Pub/Sub.

Setting up metrics and alerts

You can use Stackdriver Monitoring to set up metrics based on your log entries. And you can use log-based metrics to set up charts and alerts.

Audit policy

The Kubernetes audit policy determines which log entries are exported by the Kubernetes API server. The Kubernetes Engine audit policy determines which entries go to your Admin Activity log and which entries go to your Data Access log.

For more information about audit policies in Kubernetes Engine, see Kubernetes Engine Audit Policy.

What's next

Was this page helpful? Let us know how we did:

Send feedback about...

Kubernetes Engine