Google Distributed Cloud air-gapped 1.14.3 hotfixes

Hotfix 25

The 1.14.3-gdch.9425-25 hotfix is available.
This hotfix fixes the following issues:

DNS:

  • There are failures in deploying dns-mesh probes when zones have duplicate names.
  • DNS resolution takes a few minutes.
  • There is a problem where deleting a ManagedDNSZone before removing its associated ResourceRecordSets results in orphaned records that persist in the system.

Logging:

  • The SyslogCollectorDroppedLogs alert includes a fingerprint, which results in a new incident created each time that the alert fires.
  • The pod logs in the unet-system namespace are missing.

Managed Kubernetes Service:

  • Database instances are stuck in a deletion state caused by a race condition where virtual machines were deleted before their corresponding InventoryMachine resources.
  • User cluster deletion is stuck due to a race condition during resource deletion.
  • There is an issue with cluster scaling where downscaling a node pool followed by immediate upscaling can lead to IP address allocation failures, such as unable to assign IP.

Virtual machines:

  • VM startups are delayed until a certificate is created.
  • Random errors and delays occur when creating VMs in parallel. These issues manifest as CPI errors, agent timeouts, and network connectivity problems.
  • Windows BYO image imports fail due to kubevirt version bump.

Hotfix 24

The 1.14.3-gdch.9425-24 hotfix is available.
This hotfix fixes the following issues:

Resource manager:

  • The project and project service account performance degrades over time.

Hotfix 23

The 1.14.3-gdch.9425-23 hotfix is available.
This hotfix fixes the following issues:

Identity and access management:

  • Role bindings are enqueued frequently in the global IAM controllers, causing a propagation delay when assigning permissions.

Networking:

  • StatefulSet pod communication fails.
  • There is a missing node_cpu_seconds_total metric.
  • Networking controllers are stuck, resulting in load balancer and project network policy APIs not reconciling.

OCLCM:

  • The oclcm configrunner stops working due to a certificate renewal failure.

Hotfix 22

The 1.14.3-gdch.9425-22 hotfix is available.
This hotfix fixes the following issues:

Identity and access management:

  • Login config request times can take minutes to complete.
  • Application operators can't grant themselves access to roles in the infra cluster.
  • Existing service accounts tokens become invalid.
  • The namespace-admin project role is missing in a project.

Managed Kubernetes Service:

  • There is a timeout waiting for a shared service cluster to become ready.

Monitoring:

  • Project deletion is stuck due to dashboard and data source pending finalizers.
  • Recurring usage fails to emit metrics.
  • Metrics from KSM are not visible to PAs.
  • Prometheus pods are scheduled on control plane nodes, leading to resource exhaustion and instability.
  • KUB dashboards show incorrect data values.

Hotfix 21

The 1.14.3-gdch.9425-21 hotfix is available.
This hotfix fixes the following issues:

CLI:

  • The gdcloud compute images import command fails with QCOW images due to a file command version mismatch.
  • The gdcloud resource-support get-report command produces a failed to get support information from cluster error.
  • The gdcloud auth activate-service-account command resets the gdcloud config.
  • Added gdcloud get-credentials support for vanilla clusters.
  • Role bindings created with gdcloud bind to the platform namespace by default.
  • The gdcloud resource-support get-report command fails with an error indicating a missing schema.
  • Added SKUS to support more country-specific pricing.
  • The gdcloud CLI doesn't use the correct zone when listing clusters, so the result of gdcloud clusters list shows one zone.

Hotfix 20

The 1.14.3-gdch.9425-20 hotfix is available.
This hotfix fixes the following issues:

Console:

  • Role bindings created for service accounts are not configured correctly.
  • Project IAM Admins can't assign roles from the UI.
  • When creating a custom role, the source role of a permission changes when you view it.
  • The Roles Overview and Role Details pages show an error about not having permissions.
  • The Access page is blocked even though users have some of the required permissions.

Virtual machines:

  • The vmm-vm-controller subcomponent fails reconciliation due to a large config file exceeding the size limit (1MB).

Hotfix 19

The 1.14.3-gdch.9425-19 hotfix is available.
This hotfix fixes the following issues:

Backup and restore:

  • When deleting a snapshot in a user cluster, the corresponding snapshot in the infra cluster is not deleted.
  • The backup subcomponent deployment uses a variable that prepends characters to the cluster name when creating a Kubernetes label which can sometimes violate the Kubernetes 63 character limit.

Billing:

  • Added durable pricing SKUs.
  • The prebuy calculator is not working.
  • Recurring usage metrics are not emitted.
  • Partner billing is not enabled.
  • The prebuy calculator cannot access a community network.

Identity and access management:

  • Service account creation fails due to a project in a deleting state causing role template reconciliation failures.
  • Added P4SA support for vanilla clusters.
  • Creating a service identity from the GDC console fails.

Hotfix 18

The 1.14.3-gdch.9425-18 hotfix is available.
This hotfix fixes the following issues:

Object storage:

  • Added support for S3 GetBucketVersioning.
  • Cannot upload to sync dual-zone buckets using signed URLs.
  • DeleteObject returns 500 for non-current versioned deletes.
  • With dual zone buckets, S3 secrets are not generated after binding the project-bucket-object-admin role to a service account.

Hotfix 17

The 1.14.3-gdch.9425-17 hotfix is available.
This hotfix fixes the following issues:

File storage:

  • The Trident CSI driver deletes NetApp ONTAP volumes when they are offline, potentially leading to data loss.
  • Multi-attach errors occur for volumes after cold reboot or node de-provision scenarios.
  • The project-fileshare-admin role is missing patch and update access.
  • Snapshots are not deleted in infra clusters when deleted in a user cluster.

Managed Kubernetes Service:

  • Revert moving vanilla cluster VMs to user projects.

Networking:

  • Invalid error code affects project network policies.
  • A large CT ebpf map leads to create endpoint and delete endpoint failures.
  • Leaked services might cause service IP duplication.

Hotfix 16

The 1.14.3-gdch.9425-16 hotfix is available.
This hotfix fixes the following issues:

Identity and access management:

  • There are forbidden errors when accessing vanilla Kubernetes clusters using kubeconfig from gdcloud.

Endpoint detection response:

  • The endpoint detection response subcomponent gets stuck in a reconciliation error state.

Managed Kubernetes Service:

  • Cluster validation should use the cluster's pod density when validating nodes.

Networking:

  • The subnet predefined roles are missing verbs.

Platform authentication:

  • CSRs for intermediary CAs are missing the basic constraint for CA.
  • Added support for reusing a system domain in managed public DNS.

Ticketing system:

  • An alert is not fired when the ticketing system is unavailable.

Hotfix 15

The 1.14.3-gdch.9425-15 hotfix is available.
This hotfix fixes the following issues:

Console:

  • There is an error when creating a role binding with a non-existent role.
  • You can't add multiple role bindings to a service account in the Console.

Identity and access management:

  • Added gdcloud get-credentials support for vanilla clusters.
  • Custom roles should generate templates with the same name for global and zonal APIs.
  • Exposed CertificateAuthority data on the well-known server.
  • The identity and access management page is broken.
  • There is an error when creating a role binding to a custom role.
  • You can't attach user roles in the Console.

Managed Kubernetes Service:

  • Move the vanilla cluster VMs to a user project.
  • There are missing machine types for n3 type.

Hotfix 14

The 1.14.3-gdch.9425-14 hotfix is available.
This hotfix fixes the following issues:

Console:

  • The identity and access management page is broken.

Multizone:

  • A pod in a vanilla cluster in zone1 cannot access the Management API server in zone2.

Hotfix 13

The 1.14.3-gdch.9425-13 and 1.14.3-gdch.8490-13 hotfixes are available.
This hotfix fixes the following issues:

Console:

  • Custom role creation does not work.

  • Custom role creation from project scope shouldn't show the Limit to selected projects checkbox.

DNS:

  • A pod in a vanilla cluster in zone1 cannot access the Management API server in zone2.

Monitoring:

  • A pod in a vanilla cluster in zone1 cannot access the Management API server in zone2.

Networking:

  • Controllers are stuck for hours in the unet-cm-backend-controller pod.

  • Multiple clustermesh API servers reached their defined CPU limits.

  • Data exfiltration protection (DEP) cannot be enabled on a global project that has DEP disabled.

Object storage:

  • GetBucketVersioning for S3 is not supported.

  • There is an error while initiating cp between different folders in a bucket.

Platform authentication:

  • Cert Manager fails to issue certificates.

SIEM:

  • You can't connect to a Splunk host from a user cluster.

Hotfix 12

The 1.14.3-gdch.9425-12 and 1.14.3-gdch.8490-12 hotfixes are available.
This hotfix fixes the following issues:

Console:

  • The global DNS is not resolving from a GDC VM.

Networking:

  • Updated allow-all-ingress and allow-all-egress PNP Translation.
  • Allow egress traffic from user workloads to system workloads automatically.
  • The global DNS server is not reachable.

Object storage:

  • Downloading from an S3 bucket fails.

Hotfix 11

The 1.14.3-gdch.9425-11 hotfix is available.
This hotfix fixes the following issues:

Endpoint detection response:

  • Nessus manager has duplicate agents and managers.

  • There are gaps in EDR coverage on the perimeter, user, and service clusters.

Identity and access management:

  • The service identity server fails to authenticate using zonal service account keys.

Service mesh:

  • The dataplane-ingress-gateway pods are missing the networking.private.gdc.goog/infra-access: enabled label.

Virtual machines:

  • There is a backwards compatibility issue for subnets.