Cloud Audit Logging for Kubernetes Engine: Answer the who, what, when of admin accesses
Sometimes, in Google Kubernetes Engine, you want to investigate what’s happened in your cluster. Thankfully, Stackdriver Cloud Audit Logging now supports Kubernetes Engine in beta, and can help you answer “Who did what, where and when?” to your cluster.
To showcase Cloud Audit Logging, we’d like to introduce you to Alice, a DevOps engineer running her environment on Kubernetes Engine. Alice logs into her Kubernetes cluster to inspect the workloads and notices something odd – an instance of FoobarDB. This is peculiar, as to the best of her knowledge, her team is not using FoobarDB.
She examines the pod and realizes it’s actually a pod running a reverse shell that's only masquerading as FoobarDB. Alice decides to mitigate this vulnerability by deleting the offending pod, but wants to understand how it got there in the first place, to prevent future breaches. She turns to Cloud Audit Logging, which is a feature of Stackdriver Logging (our real-time log management and analysis tool) that writes and stores admin activity logs about your project.
While searching the audit logs, Alice finds a “create pod” request for the rogue pod coming from one of the service accounts.
She sees that this service account is associated with the deployment running the company’s PHP frontend (by investigating the “principalEmail” field highlighted by the top red box in the screenshot above) – however, there are hundreds of replicas of that deployment. Fortunately, audit logs contain the IP address of the node where pod is located. Alice can use that information to find the instance that created the rogue pod. Cloud Audit Logging contains logs about Admin Activity and Data Access but not the application-level logs so Alice uses her regular logging tool (also Stackdriver in this case) to look at the PHP frontend logs and to understand exactly what happened. She manages to track the breach to a vulnerability in the PHP version that the frontend is using so she applies a patch that upgrades the application.
Alice decides to try to prevent similar attacks in the future by using Stackdriver Monitoring features to create an alert based on a log-based metric that will fire if any identity other than the controller manager creates pods.
These are just some of the scenarios that Cloud Audit Logging can help you with. For more information on using Cloud Audit Logging, check out the documentation. And watch this space for future posts about how to use BigQuery and Cloud Audit Logging together to further identify and minimize unauthorized access.