Google Cloud Platform

Cloud Audit Logging for Kubernetes Engine: Answer the who, what, when of admin accesses

Sometimes, in Google Kubernetes Engine, you want to investigate what’s happened in your cluster. Thankfully, Stackdriver Cloud Audit Logging now supports Kubernetes Engine in beta, and can help you answer “Who did what, where and when?” to your cluster.

To showcase Cloud Audit Logging, we’d like to introduce you to Alice, a DevOps engineer running her environment on Kubernetes Engine. Alice logs into her Kubernetes cluster to inspect the workloads and notices something odd – an instance of FoobarDB. This is peculiar, as to the best of her knowledge, her team is not using FoobarDB.

audit-logging-4fuow.PNG

She examines the pod and realizes it’s actually a pod running a reverse shell that's only masquerading as FoobarDB. Alice decides to mitigate this vulnerability by deleting the offending pod, but wants to understand how it got there in the first place, to prevent future breaches. She turns to Cloud Audit Logging, which is a feature of Stackdriver Logging (our real-time log management and analysis tool) that writes and stores admin activity logs about your project.

audit-logging-2xkv7.PNG

While searching the audit logs, Alice finds a “create pod” request for the rogue pod coming from one of the service accounts.

audit-logging-3qqf5.PNG

She sees that this service account is associated with the deployment running the company’s PHP frontend (by investigating the “principalEmail” field highlighted by the top red box in the screenshot above) – however, there are hundreds of replicas of that deployment. Fortunately, audit logs contain the IP address of the node where pod is located. Alice can use that information to find the instance that created the rogue pod. Cloud Audit Logging contains logs about Admin Activity and Data Access but not the application-level logs so Alice uses her regular logging tool (also Stackdriver in this case) to look at the PHP frontend logs and to understand exactly what happened. She manages to track the breach to a vulnerability in the PHP version that the frontend is using so she applies a patch that upgrades the application.

Alice decides to try to prevent similar attacks in the future by using Stackdriver Monitoring features to create an alert based on a log-based metric that will fire if any identity other than the controller manager creates pods.

audit-logging-169mc.PNG

These are just some of the scenarios that Cloud Audit Logging can help you with. For more information on using Cloud Audit Logging, check out the documentation. And watch this space for future posts about how to use BigQuery and Cloud Audit Logging together to further identify and minimize unauthorized access.

Take it for a spin

Try Kubernetes Engine today with our generous 12-month free trial of $300 credits. Spin up a cluster (or a dozen) and experience the difference of running Kubernetes on the cloud build for containers.