This page shows you how to use logs to check the issuance and usage of Kubernetes identities, in the form of certificates and service account tokens, by the Google Kubernetes Engine (GKE) cluster control plane. This verification is entirely optional and is not required to secure your control plane.
This guide is intended for security administrators and platform owners who have specific organizational compliance or policy requirements for control over credential issuance and signing. You should already have set up self-managed keys and CAs with GKE control plane authority.
You should already be familiar with the following concepts:
This page describes one part of a set of optional control plane features in GKE that lets you perform tasks like verifying your control plane security posture or configuring encryption and credential signing in the control plane using keys that you manage. For details, see About GKE control plane authority.
By default, Google Cloud applies various security measures to the managed control plane. This page describes optional capabilities that give you more visibility or control over the control plane.
About identity issuance logs
The GKE identity issuance logs are control plane audit logs that record when the control plane issues credentials and when those credentials are used in the cluster. You can use these logs to track the lifetime of a credential, including issuance and usage, by correlating the identity issuance logs with Cloud KMS, Certificate Authority Service, and Kubernetes API logs. GKE identity issuance logs are enabled when using GKE control plane authority. These logs track the issuance and usage of the following types of credentials:
- X.509 certificates
- Cluster JSON Web Tokens (JWTs)
X.509 certificates
Kubernetes uses X.509 certificates for client certificate authentication. To issue certificates, the Kubernetes control plane sends an approved CertificateSigningRequest to a certificate authority (CA) in CA Service. The CA then issues a certificate using the corresponding key in Cloud KMS to sign the certificate digest.
The Kubernetes API server logs contain certificate signature details for any Kubernetes API call that was authenticated with a certificate. The credential ID entry in the log has the following form:
"authentication.k8s.io/credential-id": "X509SHA256=CERTIFICATE_HASH"
The CERTIFICATE_HASH
value is the SHA256 hash for the
certificate, which you can use to trace the lifecycle of the certificate.
You can use Kubernetes API server certificate logs to trace the lifecycle of the certificate by correlating logs from the following services:
- GKE identity issuance logs: Use the
protoPayload.metadata.credentialId
query to find specific GKE identity issuance logs based on the credential ID from the Kubernetes API server logs. Then use theprotoPayload.metadata.certificateFingerprint
field from the GKE identity issuance log to correlate the identity issuance logs with the CA Service logs. - CA Service logs: Find the certificate issuance log entry,
which contains the following IDs:
cert_fingerprint.sha256_hash
: the SHA256 hash of the signed certificate. Use this ID to match logs with GKE and Kubernetes API events.tbs_certificate_digest
: a hash of the certificate contents that were sent to be signed by a Cloud KMS key. Use this ID to match logs with Cloud KMS logs.
- Cloud KMS signing logs: Use the
tbs_certificate_digest
value from the CA Service log to confirm that the expected Cloud KMS key signed the certificate.
JSON Web Tokens (JWTs)
Signed JWTs (JSON Web Tokens) are used as bearer tokens for entities in the cluster, like Kubernetes service accounts, when authenticating requests to the Kubernetes API. When a Pod is created that uses a specific service account, Kubernetes creates a JWT and mounts it in the Pod. When you use GKE control plane authority to run your own keys and CAs, this JWT is signed and later verified using the service account signing key in Cloud KMS.
The Kubernetes API server logs contain token signature details for any Kubernetes API call that was authenticated with a JWT. The token signature entry in the log has the following form:
"authentication.kubernetes.io/credential-id":"JTI=JWT_ID"
The JWT_ID
value is a string that identifies the JWT that
was used in the Kubernetes API call.
You can use the JWT ID from Kubernetes API server logs to trace the lifecycle of a JWT by correlating the following logs:
- GKE identity issuance logs: Use the JWT ID from the
Kubernetes API server logs to find specific JWT issuance entries. Each entry
also contains the
toBeSignedDigest
field, the value of which can match Cloud KMS logs. - Cloud KMS signing logs: Use the value of the
toBeSignedDigest
field from the GKE identity issuance logs to confirm that the expected Cloud KMS key signed the JWT.
Pricing
In this document, you use the following billable components of Google Cloud:
To generate a cost estimate based on your projected usage,
use the pricing calculator.
Before you begin
Configure your GKE cluster control plane with self-managed CAs or keys.
Enable the following Cloud Audit Logging Service APIs:
- For Cloud KMS, turn on Data Access audit logs of type Data Read.
- For CA Service, turn on Data Access audit logs of types Admin Read and Data Write.
Required roles
To get the permissions that you need to access identity issuance logs, ask your administrator to grant you the following IAM roles on your project:
-
Perform all actions in Logging:
Logging Admin (
roles/logging.admin
). -
View logs:
Private Logs Viewer (
roles/logging.privateLogViewer
).
For more information about granting roles, see Manage access to projects, folders, and organizations.
You might also be able to get the required permissions through custom roles or other predefined roles.
Requirements and limitations
The following requirements and limitations apply:
- You must use GKE version 1.31.1-gke.1846000 or later.
- Identity issuance logs are recorded as Cloud Audit Logs logs and have a set retention period of 400 days. The retention period is not configurable, but you can route your system event audit logs to other destinations for longer retention periods.
Verify certificates
You can use GKE control plane authority identity issuance logs to confirm that a certificate was successfully issued or used. You can use any of the following logs, or a combination of logs, to confirm information about certificate issuance and use:
Certificate logs |
|
---|---|
Kubernetes API log for certificate use |
Logs the certificate signature details whenever the certificate is used against the Kubernetes API. |
GKE log for certificate issuance operations |
Logs all certificate issuance operations as a System Event audit log. These logs are enabled by default on any cluster that uses GKE control plane authority user-managed keys or CAs. |
CA Service audit log |
Logs an entry whenever a certificate is issued. |
Cloud KMS audit log |
Logs an entry whenever a digest is signed, in response to a signing request from CA Service. |
Verify certificate usage with Kubernetes API logs
To find log entries for API calls that were authenticated using certificates, complete the following steps:
In the Google Cloud console, go to the Logs Explorer page:
Paste the following expression into the query editor field:
log_id("cloudaudit.googleapis.com/activity") resource.type="k8s_cluster" labels."authentication.kubernetes.io/credential-id":"X509SHA256="
Click Run query.
This query returns all Kubernetes API server logs that have an associated X.509 certificate. Find specific log entries to investigate by using your security tooling or by checking the logs manually.
To correlate these logs with other log types, find the following field:
"authentication.k8s.io/credential-id":"CREDENTIAL_ID"
The CREDENTIAL_ID
value is the identifier that you can use to correlate logs from
GKE, CA Service, and Cloud KMS. The
CREDENTIAL_ID
has the form "X509SHA256=CERTIFICATE_HASH"
.
Verify certificate issuance with GKE logs
To find GKE log entries for certificate issuance events, complete the following steps:
In the Google Cloud console, go to the Logs Explorer page:
Paste the following expression into the query editor field:
logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event" resource.type="gke_cluster" protoPayload.serviceName="container.googleapis.com" protoPayload.metadata.credentialId="CREDENTIAL_ID"
Replace the following:
PROJECT_ID
: your project ID.CREDENTIAL_ID
: theCREDENTIAL_ID
from the Verify certificate usage with Kubernetes API logs section.
Click Run query.
Verify certificate issuance with CA Service logs
To find CA Service logs that match the GKE certificate issuance events, complete the following steps:
In the Google Cloud console, go to the Logs Explorer page:
Paste the following expression into the query editor field:
resource.type="audited_resource" protoPayload.serviceName="privateca.googleapis.com" protoPayload.methodName="google.cloud.security.privateca.v1.CertificateAuthorityService.CreateCertificate" protoPayload.response.certificate_description.cert_fingerprint.sha256_hash="CERTIFICATE_HASH"
Replace
CERTIFICATE_HASH
with theCERTIFICATE_HASH
from the Verify certificate usage with Kubernetes API logs section. Ensure that you omit theX509SHA256=
prefix from the hash.Click Run query.
This query returns a log that contains a
tbs_certificate_digest: DIGEST_VALUE
field in the certificate description response section.
You can use the DIGEST_VALUE
to
match Cloud KMS signing logs for the certificate.
Verify certificate issuance with Cloud KMS signing logs
To find Cloud KMS signing events for the CA Service certificate issuance events, complete the following steps:
In the Google Cloud console, go to the Logs Explorer page:
Paste the following expression into the query editor field:
protoPayload.request.digest.sha256="DIGEST_VALUE" protoPayload.@type="type.googleapis.com/google.cloud.audit.AuditLog" protoPayload.methodName="AsymmetricSign" protoPayload.serviceName="cloudkms.googleapis.com"
Replace
DIGEST_VALUE
with the digest value from the Verify certificate issuance with CA Service logs section.Click Run query.
This query returns the logs for certificate issuance events. Cloud KMS logs don't distinguish between certificates and JWTs, so the log entries for both are identical.
Verify tokens
You can use GKE control plane authority identity issuance logs and Cloud KMS signing logs to confirm that a JSON Web Token (JWT) was successfully issued.
Initiating the traceback of the token issuance event typically starts with monitoring the Kubernetes API log for service account activity. After you identity an activity log needing further investigation, you can use the following logs to confirm information about credential issuance and usage:
JWT logs |
|
---|---|
Kubernetes API log for JWT use |
Logs the JWT signature details whenever a JWT is used against the Kubernetes API. |
GKE log for JWT issuance operations |
Logs all token issuance operations as a System Event audit log. These logs are enabled by default on any cluster using GKE control plane authority user-managed keys or CAs. |
Cloud KMS audit log |
Logs an entry whenever a token is signed and issued. |
Verify token usage with Kubernetes API server logs
To find log entries for token usage events, complete the following steps:
In the Google Cloud console, go to the Logs Explorer page:
Paste the following expression into the query editor field:
log_id("cloudaudit.googleapis.com/activity") resource.type="k8s_cluster" labels."authentication.kubernetes.io/credential-id":"JTI="
Click Run query.
This query returns all Kubernetes API server logs that have an associated JWT. Find specific log entries to investigate by using your security tooling or by checking the logs manually.
To correlate these logs with other log types, find the following field:
"authentication.k8s.io/credential-id": "JTI=JWT_ID"
The JWT_ID
is the token identifier that you can use to correlate logs from GKE
and Cloud KMS.
Verify token issuance with GKE logs
To find log entries for token issuance events, complete the following steps:
In the Google Cloud console, go to the Logs Explorer page:
Paste the following expression into the query editor field:
resource.type="gke_cluster" logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event" protoPayload.methodName="google.cloud.gkeauth.v1.Auth.SignServiceAccountJWT" protoPayload.metadata.credentialId="JTI=JWT_ID"
Replace the following:
PROJECT_ID
: your project ID.JWT_ID
: the JWT ID from the Verify token usage with Kubernetes API server logs section.
Click Run query.
This query returns logs that contain a
toBeSignedDigest
field.
You can use the toBeSignedDigest
value to find Cloud KMS signing events.
Verify token issuance with Cloud KMS signing logs
To find log entries for signed digests, complete the following steps:
In the Google Cloud console, go to the Logs Explorer page:
Paste the following expression into the query editor field:
protoPayload.request.digest.sha256="DIGEST_VALUE" protoPayload.@type="type.googleapis.com/google.cloud.audit.AuditLog" protoPayload.methodName="AsymmetricSign" protoPayload.serviceName="cloudkms.googleapis.com"
Replace
DIGEST_VALUE
with the value in thetoBeSignedDigest
field, from the Verify token issuance with GKE logs section.Click Run query.
This query returns the logs for certificate issuance events. Cloud KMS logs don't distinguish between certificates and JWTs, so the log entries for both are identical.
What's next
- Learn about control plane security.
- Learn about administrative access for Google employees.
- Learn how to configure logging and monitoring for GKE.
- Learn how to configure field-level access to logs.
- Learn about logging usage limits.