Accessing audit logs

This page explains how to use audit logging in your Google Kubernetes Engine (GKE) clusters.

All Kubernetes-powered clusters have Kubernetes Audit Logging, which keeps a chronological record of calls that have been made to the Kubernetes API server. Kubernetes audit log entries are useful for investigating suspicious API requests, for collecting statistics, or for creating monitoring alerts for unwanted API calls.

GKE clusters integrate Kubernetes Audit Logging with Cloud Audit Logs and Cloud Logging. You can see Kubernetes audit log entries in your Google Cloud project.

In addition to entries written by Kubernetes, your project's audit logs have entries written by GKE.

Audit Logging GA is available in GKE 1.11.4 and later.

Before you begin

Before you start, make sure you have performed the following tasks:

• Ensure that you have enabled the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• Ensure that you have installed the Cloud SDK.

Set up default gcloud settings using one of the following methods:

• Using gcloud init, if you want to be walked through setting defaults.
• Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

If you receive the error One of [--zone, --region] must be supplied: Please specify location, complete this section.

1. Run gcloud init and follow the directions:

   gcloud init

   If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

   gcloud init --console-only

2. Follow the instructions to authorize gcloud to use your Google Cloud account.
3. Create a new configuration or select an existing one.
4. Choose a Google Cloud project.
5. Choose a default Compute Engine zone.

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Using gcloud config

• Set your default project ID:

  gcloud config set project project-id

• If you are working with zonal clusters, set your default compute zone:

  gcloud config set compute/zone compute-zone

• If you are working with regional clusters, set your default compute region:

  gcloud config set compute/region compute-region

• Update gcloud to the latest version:

  gcloud components update

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

You need to have a GKE cluster in your project. You can use an existing cluster, or you can create a new one for the exercises in this topic. If you choose to use an existing cluster, make sure the cluster has had some recent activity. For example, if you haven't created a Deployment recently, you could create a Deployment now by entering this command:

kubectl run log-exercise --image nginx

Read about Kubernetes Audit Logging.

Audit logs in your project

Your Cloud project has these audit logs:

• Admin Activity log
• Data Access log

Admin Activity logging is enabled by default and has no extra cost.

Data Access logging is disabled by default, and enabling it can result in extra billing. To learn more about enabling Data Access logging, and the associated costs, see Configuring Data Access Logs.

GKE does not support Access Transparency logging.

Various Google Cloud services write entries to your project's logs. The Kubernetes service also writes entries to your project's audit logs. For GKE clusters, log entries written by these services are the most relevant:

Service	Display name	Description
k8s.io	Kubernetes	The k8s.io service is used for Kubernetes audit logs. These logs are generated by the Kubernetes API Server component and they contain information about actions performed using the Kubernetes API. For example, any changes you make on a Kubernetes resource by using the kubectl command are recorded by the k8s.io service. For more information, see Auditing in the Kubernetes documentation.
container.googleapis.com	Kubernetes Engine	The container.googleapis.com service is used for GKE control plane audit logs. These logs are generated by the GKE internal components and they contain information about actions performed using the GKE API. For example, any changes you perform on a GKE cluster configuration using a gcloud command are recorded by the container.googleapis.com service.

Viewing your project's Admin Activity log

Console

1. In the Cloud Console, go to the Logs page in the Logging menu.

   Go to the Logs page

2. Near the top of the page, locate the drop-down menu for selecting a resource type. From the drop-down menu, select Kubernetes Cluster.

3. Optionally, specify the Location or choose All location. If you select a location, you can select a specific cluster in that location.

4. The next menu to the right is for selecting a log. From the drop-down menu, select activity, and click OK.

5. The display shows all log levels by default. To specify a log level, such as Warning, select it from the drop-down menu representing log levels.

6. The display might only show log entries from the last hour. If you don't see any log entries from the last hour, click Load older logs.

7. In the Filter by label or text search box, above the drop-down menus discussed in the preceding steps, click the down arrow to open the drop-down menu. From the menu, choose Convert to advanced filter.

8. The text box displays a filter similar to this:

   resource.type="k8s_cluster"
   logName="projects/my-project/logs/cloudaudit.googleapis.com%2Factivity"
   

9. Open one of the log entries by clicking the arrow at the beginning of the line. The logName field of the entry has the value projects/project-id/logs/cloudaudit.googleapis.com%2Factivity.

gcloud

List the first two log entries in your project's Admin Activity log:

gcloud logging read \
    'logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"' \
    --limit 2 \
    --freshness 300d

where project-id is your project ID.

The output shows two log entries. Notice that for each log entry, the logName field has the value projects/project-id/logs/cloudaudit.googleapis.com%2Factivity.

insertId: 18yao5jem14og
labels:
  cluster_version: 1.8.8-gke.0
logName: projects/project-id/logs/cloudaudit.googleapis.com%2Factivity
...

Basic and advanced filter interfaces

In the Cloud Console, the Logs page has two filtering interfaces: basic and advanced. For information about the two filtering interfaces, see Logs Viewer filter interfaces.

Viewing services that write to your Admin Activity log

Console

1. In the Cloud Console, go to the Logs page in the Logging menu.

   Go to the Logs page

2. If the Logs page is not already in advanced mode, switch to advanced mode. In the filter box, click the down arrow at the right, and select Convert to advanced filter.

3. In the filter box, delete any existing text, and enter this filter.

   logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
   protoPayload.serviceName="k8s.io"
   

   where project-id is your project ID.

   Click Submit filter.

   The display lists all entries in your Admin Activity log that were written by the k8s.io service, that is, the entries written by the Kubernetes control plane.

4. Look for log entries that were written by services other than k8s.io. In the filter box, change serviceName="k8s.io" to serviceName!="k8s.io". Click Submit Filter.

   The display lists log entries in your Admin Activity log that were not written by the k8s.io service.

5. Open one of the log entries, and expand the protoPayload field. Look at the value of serviceName to see which service wrote the log entry.

6. Look for log entries that were written by the container.googleapis.com service, that is, the entries written by the Kubernetes Engine control plane. In the filter box, change serviceName!="k8s.io" to serviceName="container.googleapis.com". Click Submit Filter.

gcloud

1. List the first two log entries in your project's Admin Activity log that were written by the k8s.io service, that is, the entries written by the Kubernetes control plane.

   gcloud logging read \
      'logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
          AND protoPayload.serviceName="k8s.io"' \
      --limit 2 \
      --freshness 300d
   

   where project-id is your project ID.

   In the output, you can see that protoPayload:serviceName has the value k8s.io:

   protoPayload:
    ...
    serviceName: k8s.io
   

2. Look for log entries that were written by services other than k8s.io:

   gcloud logging read \
      'logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
          AND protoPayload.serviceName!="k8s.io"' \
      --limit 2 \
      --freshness 300d
   

   The output lists log entries in your Admin Activity log that were not written by the k8s.io service.

   logName: projects/project-id/logs/cloudaudit.googleapis.com%2Factivity
   ...
   protoPayload:
    ...
    serviceName: compute.googleapis.com
   ...
   

3. In one of the log entries, look at the value of protoPayload.serviceName to see which service wrote the log entry.

4. Look for log entries that were written by the container.googleapis.com service, that is, the entries written by the GKE control plane:

   gcloud logging read \
      'logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
          AND protoPayload.serviceName="container.googleapis.com"' \
      --limit 2 \
      --freshness 300d
   

   where project-id is your project ID.

Filtering your Admin Activity log by resource type

Each log entry in your Admin Activity log applies to a certain type of resource. These are the resource types that are the most relevant to Kubernetes clusters:

Resource type	Display name	Description
k8s_cluster	Kubernetes Cluster	Log entries written by the Kubernetes API server apply to the k8s_cluster resource type. These log entries describe operations on Kubernetes resources in your cluster, for example, Pods, Deployments, and Secrets.
gke_cluster	GKE Cluster Operations	Log entries written by the Kubernetes Engine API server apply to the gke_cluster resource. These log entries describe operations like cluster creation and deletion.

You can use these filters in the Cloud Console or by using the gcloud command line tool.

Console

1. In the Cloud Console, go to the Logs page in the Logging menu.

   Go to the Logs page

2. If the Logs page is not already in basic mode, switch to basic mode. In the filter box, click the down arrow at the right, and select Clear filters and return to basic mode.

3. From the drop-down menu for selecting a resource type, select Kubernetes Cluster. This is the display name for the k8s_cluster resource type.

4. From the drop-down menu for selecting a log, select activity, and click OK.

   The display lists all log entries in your Admin Activity log that apply to the k8s_cluster resource type.

5. Open one of the log entries, and expand the resource field. Verify that the type field has the value k8s_cluster.

6. From the drop-down menu, select GKE Cluster Operations. This is the display name for the gke_cluster resource type. From the drop-down menu for selecting a log, select activity, and click OK.

   The display lists all log entries in your Admin Activity log that apply to the gke_cluster resource type.

gcloud

1. List the first two log entries in your project's Admin Activity log that apply to the k8s_cluster resource type:

   gcloud logging read \
      'logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
          AND resource.type="k8s_cluster"' \
      --limit 2 \
      --freshness 300d
   

   where project-id is your project ID.

   In the output, you can see that the resource:type field has the value k8s_cluster:

   resource:
    ...
    type: k8s_cluster
   

2. List the first two log entries in your project's Admin Activity log that apply to the gke_cluster resource type:

   gcloud logging read \
      'logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
          AND resource.type="gke_cluster"' \
      --limit 2 \
      --freshness 300d
   

   In the output, you can see that the resource:type field has the value gke_cluster:

   resource:
    ...
    type: gke_cluster
   

Example filters for your Admin Activity log

Here are some examples of filters that you can try in the Cloud Console. In each case, replace project-id with your project ID.

Find changes to Role Based Access Control, excluding automated system changes.

logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.methodName:"io.k8s.authorization.rbac.v1"
NOT protoPayload.authenticationInfo.principalEmail:"system"

logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.methodName:"io.k8s.authorization.rbac.v1.roles"
NOT protoPayload.authenticationInfo.principalEmail:"system"

logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.methodName:"io.k8s.authorization.rbac.v1.rolebindings"
NOT protoPayload.authenticationInfo.principalEmail:"system"

You can use similar queries to find changes to clusterroles and clusterrolebindings.

Find certificate signing requests.

logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.resourceName:"certificates.k8s.io/v1beta1/certificatesigningrequests"

Find unauthenticated web requests.

logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.authenticationInfo.principalEmail:"system:anonymous"

Find kubelet bootstrap identity calls.

logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.authenticationInfo.principalEmail:"kubelet"

Find node authenticated requests.

logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.authenticationInfo.principalEmail:"system:node"

Find calls outside an IP range.

logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.requestMetadata.callerIp!="127.0.0.1"
protoPayload.requestMetadata.callerIp!="::1"
NOT protoPayload.requestMetadata.callerIp:"ip-prefix"

Find entries in your Admin Activity log that apply to the k8s_cluster resource type and describe creating a Deployment.

logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.methodName:"deployments.create"

Find entries in your Admin Activity log that apply to the k8s_cluster resource type and have a principalEmail value of system:anonymous. These entries probably represent failed attempts to authenticate.

logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.authenticationInfo.principalEmail="system:anonymous"

Find entries in your Admin Activity log that apply to the gke_cluster resource type and describe cluster creation:

logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="gke_cluster"
protoPayload.methodName="google.container.v1.ClusterManager.CreateCluster"

Find entries in your Admin Activity log that apply to the gke_cluster resource type and have a severity value of ERROR:

logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="gke_cluster"
severity="ERROR"

Find entries in your Admin Activity log that apply to the k8s_cluster resource type and describe a write request to a Secret:

logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.methodName:"io.k8s.core.v1.secrets"
NOT protoPayload.methodName:"get"
NOT protoPayload.methodName:"list"
NOT protoPayload.methodName:"watch"

Find entries in your Admin Activity log that apply to the k8s_cluster resource type and describe a Pod request from a particular user:

logName="projects/project-id/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.methodName:"io.k8s.core.v1.pods"
protoPayload.authenticationInfo.principalEmail="[e-mail]"

For more information about how to construct filters, see Advanced Logs Filters.

Structure of a log entry

Every log entry is an object of type LogEntry. For more information, go to Understanding audit logs.

Enabling data access logs

1. Get the Identity and Access Management (IAM) policy for your project:

   gcloud projects get-iam-policy project-id > my-policy.yaml
   

   where project-id is your project ID.

2. Open my-policy.yaml to view your IAM policy. Your policy probably contains a bindings object similar to this:

   bindings:
   - members:
     - serviceAccount:xxx.gserviceaccount.com
     - serviceAccount:yyy.gserviceaccount.com
     role: roles/container.clusterAdmin
   - members:
     ...
   

3. In my-policy.yaml, create an auditConfigs object, or add to your existing auditConfigs object, so that ADMIN_READ, DATA_WRITE, AND DATA_READ are listed under auditLogConfigs.

   auditConfigs:
   - auditLogConfigs:
     - logType: ADMIN_READ
     - logType: DATA_WRITE
     - logType: DATA_READ
     service: allServices
   

   Do not change the value of etag.

4. Save your updated file as my-policy-2.yaml.

   Caution: Do not change any part of your policy that is not related to audit logging. You are editing a policy object that contains critical information about who can access your project or organization. Accidentally altering that information could make your project or organization unusable.

5. Set the IAM policy for your project:

   gcloud projects set-iam-policy project-id my-policy-2.yaml
   

   where project-id is your project ID.

Viewing your project's Data Access log

Console

1. In the Cloud Console, go to the Logs page in the Logging menu.

   Go to the Logs page

2. If the Logs page is not already in basic mode, switch to basic mode. In the filter box, click the down arrow at the right, and select Clear filters and return to basic mode.

3. Near the top of the page, locate the drop-down menu for selecting a resource type. From the drop-down menu, select Kubernetes Cluster.

4. In the menu for selecting a log, select data_access, and click OK.

5. In the Filter by label or text search box, at the right side, click the down arrow to open the drop-down menu. From the menu, choose Convert to advanced filter.

6. The text box displays a filter similar to this:

   resource.type="k8s_cluster"
   logName="projects/my-project/logs/cloudaudit.googleapis.com%2Fdata_access"

7. Open one of the log entries, and notice that the logName field of the entry has the value projects/project-id/logs/cloudaudit.googleapis.com%2Fdata_access.

gcloud

List the first two log entries in your project's Data Access log:

gcloud logging read \
    'logName="projects/project-id/logs/cloudaudit.googleapis.com%2Fdata_access"' \
    --limit 2 \
    --freshness 30d

where project-id is your project ID.

The output shows two log entries. Notice that for each log entry, the logName field has the value projects/project-id/logs/cloudaudit.googleapis.com%2Fdata_access.

insertId:  "x0vy9eej0j54"
labels: {…}
logName:  "projects/project-id/logs/cloudaudit.googleapis.com%2Fdata_access"
...

Exporting and storing log entries

Log entries are held in Cloud Logging for a limited time known as the retention period. After that, the entries are deleted.

If you want to keep your log entries longer, you can export them to a Google service like Cloud Storage, BigQuery, or Pub/Sub.

Setting up metrics and alerts

You can use Cloud Monitoring to set up metrics based on your log entries. And you can use log-based metrics to set up charts and alerts.

Audit policy

The Kubernetes audit policy determines which log entries are exported by the Kubernetes API server. The Kubernetes Engine audit policy determines which entries go to your Admin Activity log and which entries go to your Data Access log.

For more information about audit policies in Kubernetes Engine, see Kubernetes Engine Audit Policy.

What's next

• Kubernetes Audit Logging
• Kubernetes Engine Audit Policy
• Kubernetes Engine Security Overview
• Cloud Audit Logs