This page describes how to verify connections made by Google personnel to your Google Kubernetes Engine (GKE) cluster control plane by correlating GKE logs with Access Transparency logs.
Access Transparency logs record the actions that Google personnel take when accessing your content. This guide is intended for security administrators who want additional verification of the contents of Access Transparency logs and associated Access Approval approvals by correlating with additional logging sources from GKE. This verification is entirely optional and is not required to secure your control plane.
Ensure that you're familiar with the following concepts:
This page describes one part of a set of optional control plane features in GKE that lets you perform tasks like verifying your control plane security posture or configuring encryption and credential signing in the control plane using keys that you manage. For details, see About GKE control plane authority.
By default, Google Cloud applies various security measures to the managed control plane. This page describes optional capabilities that give you more visibility or control over the GKE control plane.
About Google access to cluster control plane instances
During troubleshooting sessions or for other justified business reasons, Google personnel like site reliability engineers and Cloud Customer Care employees might need administrative access to the Compute Engine instances that host your control plane. Depending on your Customer Care support package and configuration, Access Transparency provides detailed audit logging for this administrative access. Access Approval lets you require explicit approval before any Google personnel can access your resources. To learn more about administrative access and the tooling that you can use to authorize access and to record changes, see Administrative access for Google employees.
Control plane access logs
When you enable GKE control plane authority, GKE
generates control plane access logs that you can optionally use to
cross-reference the audit logs generated by Access Transparency and by
Access Approval. GKE adds control plane access
logs to the
_Default
bucket in Logging
to record incoming network connections and specific SSH events
in your control plane instances. You must enable
GKE control plane authority in your project to generate control
plane access logs for your clusters.
GKE generates the following access logs for the control plane:
The volume of control plane connection logs depends on factors like the number of nodes in the cluster, the number of control plane instances (regional clusters have more control plane instances than zonal clusters), and how often your workloads call the Kubernetes API server. The volume of SSH logs is small and depends on the number of node restarts.
To verify the connections to your control plane, you find the control plane access logs for your cluster and match those logs to audit logs from Access Transparency and Access Approval. This lets you confirm that all of the SSH connections to your control plane instances were as a result of authorized administrative access by Google personnel. When you enable GKE control plane authority for your cluster, all SSH access by Google personnel to your control plane is non-interactive, which means that every SSH connection runs a single command that you authorize.
Pricing
The following pricing considerations apply:
- Control plane access logs are subject to Logging pricing.
- Access Transparency is included with specific Customer Care subscriptions. For details, see Access Transparency pricing.
Before you begin
Before you start, make sure you have performed the following tasks:
- Enable the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- If you want to use the Google Cloud CLI for this task,
install and then
initialize the
gcloud CLI. If you previously installed the gcloud CLI, get the latest
version by running
gcloud components update
.
Enable the Cloud Logging API.
Enable Access Transparency for your organization. For instructions, see Enabling Access Transparency.
Optionally, enable Access Approval for your project and select the GKE service. For instructions, see Review and approve access requests using the Google-managed signing key.
Ensure that your environment is eligible to use GKE control plane authority features. To opt-in to these features, contact your Google Cloud sales team.
Requirements
Control plane access logs require GKE version 1.31.1-gke.1846000 or later.
Required roles and permissions
To get the permissions that you need to enable log generation and to access and process logs, ask your administrator to grant you the following IAM roles:
-
Enable control plane connection logging in your cluster:
Kubernetes Engine Cluster Admin (
roles/container.clusterAdmin
) on your project -
Access logs and to use Logs Explorer and Log Analytics:
Logs Viewer (
roles/logging.viewer
) on your project -
Enable Access Transparency for the organization:
Access Transparency Admin (
roles/axt.admin
) on your organization
For more information about granting roles, see Manage access to projects, folders, and organizations.
You might also be able to get the required permissions through custom roles or other predefined roles.
Enable GKE cluster control plane access logs
You can enable control plane access log generation for Autopilot mode and Standard mode clusters by enabling the corresponding logging component. For more information about control plane log types, see View GKE logs.
The supported logging component names for control plane access logs are as follows:
- Control plane SSH logs:
KCP_SSHD
- Control plane connection logs:
KCP_CONNECTION
Enable control plane access logs on a new cluster
The following example creates an Autopilot mode cluster with both types of control plane access logs enabled. To only enable one type of control plane access log, omit the corresponding component name from the command.
gcloud container clusters create-auto CLUSTER_NAME \
--location=LOCATION \
--logging=SYSTEM,KCP_SSHD,KCP_CONNECTION
Replace the following:
CLUSTER_NAME
: the name for the new cluster.LOCATION
: the location in which to create the cluster.
To specify logging components when you create a cluster using the
GKE API, in the
projects.locations.clusters.create
method, set the corresponding values in the
LoggingConfig
object
of the Cluster
resource.
Enable control plane access logs on an existing cluster
To update the logging configuration of an existing cluster to enable control plane access logs, you must do the following:
- Find the existing logging components that your cluster uses.
- Identify the corresponding values to specify in the
--logging
flag in the gcloud CLI to keep those logging components enabled. - Update your cluster logging configuration to enable the control plane access logs alongside your existing logging configuration.
The values that you specify for the --logging
flag in the
gcloud container clusters update
command are different from the values that
you see when you describe your cluster.
Check the existing logging configuration of the cluster:
gcloud container clusters describe CLUSTER_NAME \ --location=LOCATION \ --flatten=loggingConfig \ --format='csv[delimiter=",",no-heading](componentConfig.enableComponents)'
The output is similar to the following:
SYSTEM_COMPONENTS,WORKLOADS,APISERVER,SCHEDULER,CONTROLLER_MANAGER
Identify the gcloud CLI values for the
--logging
flag that correspond to the logging component configuration from the output of the preceding step. For a list of the gcloud CLI values that correspond to specific logging components, see the Available logs table.Update the logging configuration with control plane access logs:
gcloud container clusters update CLUSTER_NAME \ --location=LOCATION \ --logging=SYSTEM,EXISTING_LOGS,KCP_ACCESS_LOGS
Replace the following:
EXISTING_LOGS
: a comma-separated list of logging components that your cluster already uses. Ensure that you specify the gcloud CLI values that correspond to these logging components, taken from the Available logs table.KCP_ACCESS_LOGS
: a comma-separated list of the control plane access log types to enable for the cluster, as follows:- For control plane SSH logs, specify
KCP_SSHD
. - For control plane connection logs, specify
KCP_CONNECTION
.
- For control plane SSH logs, specify
To specify logging components when you update a cluster using the
GKE API, in the
projects.locations.clusters.update
method, set the existing and new logging component values in the
LoggingConfig
object
of the
ClusterUpdate
resource.
Example cluster update to enable control plane access logs
Consider a cluster for which the gcloud container clusters describe
command
has the following logging configuration:
SYSTEM_COMPONENTS,WORKLOADS,APISERVER,SCHEDULER,CONTROLLER_MANAGER
The following cluster update command enables both types of control plane access logs while retaining the existing log configuration for this example cluster:
gcloud container clusters update example-cluster \
--location=us-central1 \
--logging=SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,KCP_SSHD,KCP_CONNECTION
Cross-reference control plane access logs with Access Transparency logs
To verify control plane access for a cluster, get the control plane connection logs, the control plane SSH logs, and the Access Transparency logs for that cluster:
In the Google Cloud console, open the Logs Explorer page.
To get all of the logs for a specific cluster, including the control plane access logs and Access Transparency logs, run the following query:
(logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fkcp-connection" resource.labels.cluster_name="CLUSTER_NAME" jsonPayload.connection.dest_port="22") OR (logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fkcp-sshd" resource.labels.cluster_name="CLUSTER_NAME") OR (logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Faccess_transparency" json_payload.accesses.methodName="GoogleInternal.SSH.Master" json_payload.accesses.resourceName="//container.googleapis.com/projects/PROJECT_NUMBER/locations/LOCATION/clusters/CLUSTER_NAME")
The output should display all of the following types of logs for your cluster:
- Access Transparency log
- Control plane connection log
- SSH logs for each SSH session
Perform verification checks
Your primary verification check is whether you see all of the log types for any SSH connections when you run the Logging query from the preceding section. Every Access Transparency log should have a corresponding control plane connection log and one or more SSH logs. These logs are for actions that humans perform in your control plane instances, so the log volume should be small.
You can optionally perform the following additional checks of the log contents:
- For each control plane SSH log, check whether an Access Transparency log exists in a 15-minute window before the timestamp of the SSH log. This window of time accounts for the final SSH session closure happening several minutes after the initial connection was logged by Access Transparency.
- For each control plane connection log, check whether an Access Transparency log exists in a 5-minute window before the timestamp of the control plane connection log.
If you use Access Approval for your cluster, check whether each Access Transparency log has a corresponding
accessApprovals
field. Cross-reference this field against Access Approval requests for your cluster.To get Access Approval requests for your project, see View historical Access Approval requests. Access Approval might be subject to exclusions.
Optionally, validate the signature of the signed Access Approval associated with the Access Transparency log.
Control plane access log details
This section provides details and examples of the control plane access logs that GKE generates when Google personnel connect to your control plane instances.
Control plane connection logs
GKE adds a control plane connection log for every new incoming network connection to a control plane instance. These logs include specific details like the following:
- Source and destination IP addresses and ports
- Connection direction and protocol
The following example is of a control plane connection log:
{
insertId: "z1eq8wonio335a5h",
jsonPayload: {
instance: {
vm_name: "gke-dee49f0d6fa34ce3a2ac-f513-d195-vm",
zone: "us-central1-c"
},
cluster: {
cluster_id: "CLUSTER_ID",
cluster_urn: "//container.googleapis.com/projects/PROJECT_NUMBER/locations/us-central1-c/clusters/CLUSTER_NAME"
},
connection: {
state: "NEW",
src_ip: "192.0.2.100",
src_port: 32774,
dest_ip: "203.0.113.12",
dest_port: 22,
direction: "INGRESS"
protocol: "TCP"
},
}
logName: "projects/PROJECT_ID/logs/container.googleapis.com%2Fkcp-connection",
receiveTimestamp: "2024-04-11T04:08:01.883070399Z",
resource: {
labels: {
cluster_name: "CLUSTER_NAME",
location: "us-central1-c",
project_id: "PROJECT_ID"
}
type: "gke_cluster",
}
severity: "NOTICE",
timestamp: "2024-04-11T04:07:59.019330Z"
}
The following fields in the log entry are relevant to verifying Google's actions:
cluster.cluster_urn
: The fully-qualified resource identifier of the cluster. This identifier has the format//container.googleapis.com/projects/PROJECT_NUMBER/locations/LOCATION/clusters/CLUSTER_NAME
, with the following variables:PROJECT_NUMBER
: the numerical project number of your clusters project.LOCATION
: the Google Cloud location of your cluster.CLUSTER_NAME
: the name of your cluster.
connection
: Details about the connection attempt. This field has the following information:state
: the state of the connection. For new connections, the value isNEW
.src_ip
: the IP address of the connection source.src_port
: the port number of the connection source.dest_ip
: the internal IP address of your control plane VM.dest_port
: the destination port number.direction
: the connection direction. The value is alwaysINGRESS
.protocol
: the IP protocol, likeTCP
.
Control plane SSH logs
GKE adds control plane SSH logs for events related to SSH connections to control plane instances. GKE records the following events:
- SSH key accepted for a user
- Session changed status from 0 to 1, indicating that the user successfully logged in
- SSH session opened
- SSH session closed
- Session changed status from 1 to 0, indicating that the user logged out
- SSH session failed
For example, the following control plane SSH log is for an SSH session being opened:
{
insertId: "8llczemdulwbbwpa",
jsonPayload: {
instance: {
vm_name: "gke-06cb920c609941c0a5ce-6840-40e9-vm",
zone: "us-central1-c"
},
cluster: {
cluster_id: "891e6d12889747748c1ac16ffcc6cb7c0a96450b36864eb680917c119fd801d0",
cluster_urn: "//container.googleapis.com/projects/PROJECT_NUMBER/locations/us-central1/clusters/CLUSTER_NAME",
},
message: "pam_unix(sshd:session): session opened for user REDACTED by (uid=0)",
},
logName: "projects/PROJECT_ID/logs/container.googleapis.com%2Fkcp-ssh",
receiveTimestamp: "2024-04-09T13:21:55.231436462Z"
resource: {
type: "gke_cluster",
labels: {
cluster_name: "CLUSTER_NAME",
location: "us-central1",
project_id: "PROJECT_ID"
}
},
severity: "NOTICE",
timestamp: "2024-04-09T13:21:50.742246Z"
}
The following fields in the log entry are relevant to verifying Google's actions:
cluster.cluster_urn
: The fully-qualified resource identifier of the cluster. This identifier has the format//container.googleapis.com/projects/PROJECT_NUMBER/locations/LOCATION/clusters/CLUSTER_NAME
, with the following variables:PROJECT_NUMBER
: the numerical project number of your cluster project.LOCATION
: the Google Cloud location of your cluster.CLUSTER_NAME
: the name of your cluster.
message
: Details about the SSH connection.
Disable control plane access logs
To view the specific log types that your cluster uses, run the following command:
gcloud container clusters describe CLUSTER_NAME \ --location=LOCATION \ --flatten=loggingConfig \ --format='csv[delimiter=",",no-heading](componentConfig.enableComponents)'
The output is similar to the following:
SYSTEM_COMPONENTS,WORKLOADS,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,KCP_SSHD,KCP_CONNECTION
To disable the control plane access logs for a cluster, run the following command:
gcloud container clusters update CLUSTER_NAME \ --location=LOCATION \ --logging=SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER
In the --logging
flag, specify the logging components from the output of the
previous command. This example command disables control plane access logs but
keeps other control plane component logs enabled.
To disable logging components using the GKE API, set the
corresponding values in the LoggingConfig
object of the ClusterUpdate
resource in
the
projects.locations.clusters.update
method.
What's next
- Learn about control plane security.
- Learn about administrative access for Google employees.
- Learn how to configure logging and monitoring for GKE.
- Learn how to configure field-level access to logs.
- Learn about logging usage limits.