Overview
GKE on-prem makes use of Kubernetes Audit Logging, which keeps a chronological record of calls made to a cluster's Kubernetes API server. Audit logs are useful for investigating suspicious API requests and for collecting statistics.
Prior to GKE Enterprise 1.2, GKE on-prem wrote audit logs only to disk. GKE Enterprise 1.2 introduces an alpha feature that enables audit logs to be written to Cloud Audit Logs in a Google Cloud project. Writing to Cloud Audit Logs has several benefits over writing to disk, or even capturing logs in an on-premises logging system:
- Audit logs for all GKE clusters can be centralized.
- Log entries written to Cloud Audit Logs are immutable.
- Cloud Audit Logs entries are retained for 400 days.
- Cloud Audit Logs is included in the price of Anthos.
You can configure GKE on-prem to write logs to disk or to Cloud Audit Logs.
Disk-based audit logging
By default, audit logs in GKE on-prem are written to a persistent disk so that VM restarts and upgrades don't cause the logs to disappear. GKE on-prem retains up to 12 GB of audit log entries.
Cloud Audit Logs
If Cloud Audit Logs is enabled, then Admin Activity audit log entries from all Kubernetes API servers are sent to Google Cloud, using the project and location that you specify when you create a user cluster. When you enable Cloud Audit Logs, GKE on-prem disables disk-based audit logging.
To buffer and write log entries to Cloud Audit Logs,
GKE on-prem deploys an audit-proxy
Pod to the admin cluster.
This Pod is also available as a sidecar container on user clusters.
Limitations
Cloud Audit Logs for GKE on-prem is an alpha feature. This alpha release has several limitations:
Data access logging is not supported.
Modifying the Kubernetes audit policy is not supported.
Cloud Audit Logs is currently not resilient to network outages. If the log entries cannot be exported to Google Cloud, they are dropped.
Enabling Anthos GKE API and Anthos Audit API
To use Cloud Audit Logs with GKE on-prem, do the following:
If you are on GKE Enterprise 1.4.x or earlier, enable the Anthos GKE API.
If you are on GKE Enterprise 1.5 or later, enable the Anthos Audit API.
If you are not sure, or you're planning to upgrade to GKE Enterprise 1.5, enable both.
Creating a service account for Cloud Audit Logs
You already have several service accounts that you created to use with GKE on-prem. For this alpha feature, you need to create an additional service account and have it allowlisted.
Create your Cloud Audit Logs service account:
gcloud iam service-accounts create audit-logging-service-account
Create a JSON key file for your Cloud Audit Logs service account:
gcloud iam service-accounts keys create audit-logging-key.json \ --iam-account AUDIT_LOGGING_SERVICE_ACCOUNT_EMAIL
where AUDIT_LOGGING_SERVICE_ACCOUNT_EMAIL is the email address of your service account.
Save
audit-logging-key.json
on the admin workstation in the same location as your other service account keys.
Allowlisting your service account
To request that your Cloud Audit Logs service account be added to the allowlist, fill in the Cloud Audit Logs for GKE on-prem Alpha form. You will receive a notification email when the allowlisting is done. Your service account must be allowlisted before you create an admin cluster or a user cluster that enables Cloud Audit Logs.
Creating an admin cluster with Cloud Audit Logs enabled
You can enable Cloud Audit Logs for an admin cluster only when you first create the admin cluster. You cannot modify an existing admin cluster to enable Cloud Audit Logs.
Refer to the instructions for Creating an admin cluster.
After you run
gkectl create-config
, fill in your configuration file,admin-cluster.yaml
, as usual, but also fill in the newcloudAuditLogging
section.Set
cloudAuditLogging.projectId
to the project ID of the Google Cloud project in which you want to view audit logs that pertain to your admin cluster.Set
cloudAuditLogging.clusterLocation
to a Google Cloud region where you want to store audit logs. For improved latency, choose a region that is near your on-premises data center.Set
cloudAuditLogging.serviceAccountKeyPath
to the path of the JSON key file for your Cloud Audit Logs service account.
For example:
cloudAuditLogging: projectId: "my-project" clusterLocation: "us-west1" serviceAccountKeyPath: "/my-key-folder/audit-logging-key.json"
Continue the cluster creation as usual.
Creating a user cluster with Cloud Audit Logs enabled
If you have not yet created an admin cluster, refer to the instructions for Creating an admin cluster.
If you already have an admin cluster, create a new user cluster by following the instructions in Creating additional user clusters.
After you run
gkectl create-config
, fill in your configuration file,user-cluster.yaml
, as usual, but also fill in the newcloudAuditLogging
section.Set
cloudAuditLogging.projectId
to the project ID of the Google Cloud project where you want to view audit logs that pertain to your user cluster.Set
cloudAuditLogging.clusterLocation
to a Google Cloud region where you want to store audit logs. It is a good idea to choose a region that is near your on-premises data center.Set
cloudAuditLogging.serviceAccounKeyPath
to the path of the JSON key file for your Cloud Audit Logs service account.
For example:
cloudAuditLogging: projectId: "my-project" clusterLocation: "us-west1" serviceAccountKeyPath: "/my-key-folder/audit-logging-key.json"
Continue the cluster creation as usual.
Enabling Cloud Audit Logs on an existing user cluster
Cloud Audit Logs can be enabled on an existing user cluster via the
gkectl update cluster
command.
Fill in the cloudAuditLogging
section of your
user-cluster.yaml
file. (See
Creating a user cluster with Cloud Audit Logs enabled
for details on the individual fields.)
Then run:
gkectl update cluster --config [USER_CLUSTER_YAML] --kubeconfig [ADMIN_CLUSTER_KUBECONFIG]
Disabling Cloud Audit Logs on an existing user cluster
Open the
user-cluster.yaml
file that describes your user cluster.Delete or comment out the
cloudAuditLogging
section, and save the file.Run this command to update the user cluster:
gkectl update cluster --config [USER_CLUSTER_YAML] --kubeconfig [ADMIN_CLUSTER_KUBECONFIG]
Accessing GKE on-prem audit logs
Disk-based audit logging
View the Kubernetes API servers running in your admin cluster and all of its associated user clusters:
kubectl --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] get pods --all-namespaces -l component=kube-apiserver
where [ADMIN_CLUSTER_KUBECONFIG] is the kubeconfig file of your admin cluster.
Download the API server's audit logs:
kubectl cp -n [NAMESPACE] [APISERVER_POD_NAME]:/var/log/kube-audit/kube-apiserver-audit.log /tmp/kubeaudit.log
This command fetches the latest log file, which can contain up to 1GB of data for the admin cluster and up to 850GB for user clusters.
You can also find the audit logs for the admin cluster on the control-plane nodes under
/var/log/kube-audit/kube-apiserver-audit.log
. The audit logs for the user cluster are in thePersistentVolumeClaim
namedkube-audit-kube-apiserver-0
. You can access this data within your own Pods viavolume
entries like this:volumes: ‐ name: kube-audit hostPath: path: /var/log/kube-audit type: ""
volumes: ‐ name: kube-audit persistentVolumeClaim: claimName: kube-audit-kube-apiserver-0 readOnly: true
To schedule your Pod on the appropriate admin cluster node (and only this node) you will need to add
nodeSelector
andtolerations
sections to your Pod spec, like this:spec: nodeSelector: node-role.kubernetes.io/master: '' tolerations: ‐ key: node-role.kubernetes.io/master value: "" effect: NoSchedule
For the user cluster, use this
nodeSelector
:spec: nodeSelector: kubernetes.googleapis.com/cluster-name: [USER_CLUSTER_NAME]
Older audit records are kept in separate files. To view those files:
kubectl exec -n [NAMESPACE] [APISERVER_POD_NAME] -- ls /var/log/kube-audit -la
Each audit log's filename has a timestamp that indicates when the file was rotated. A file contains audit logs up to that time and date.
Cloud Audit Logs
Console
In the Google Cloud console, go to the Logs page in the Logging menu.
In the Filter by label or text search box, just above the drop-down menus discussed above, click the down arrow to open the drop-down menu. From the menu, choose Convert to advanced filter.
Fill the text box with the following filter:
resource.type="k8s_cluster" logName="projects/[PROJECT_ID]/logs/externalaudit.googleapis.com%2Factivity" protoPayload.serviceName="anthosgke.googleapis.com"
Click Submit Filter to display all audit logs from GKE on-prem clusters that were configured to log in to this project.
gcloud
List the first two log entries in your project's Admin Activity log that
apply to the k8s_cluster
resource type:
gcloud logging read \ 'logName="projects/[PROJECT_ID]/logs/externalaudit.googleapis.com%2Factivity" \ AND resource.type="k8s_cluster" \ AND protoPayload.serviceName="anthosgke.googleapis.com" ' \ --limit 2 \ --freshness 300d
where [PROJECT_ID] is your project ID.
The output shows two log entries. Notice that for each log entry, the
logName
field has the value
projects/[PROJECT_ID]/logs/externalaudit.googleapis.com%2Factivity
and protoPayload.serviceName
is equal to anthosgke.googleapis.com
.
Audit policy
Cloud Audit Logs behavior is determined by a statically-configured Kubernetes audit logging policy. Changing this policy is currently not supported, but will be available in a future release.