Google Distributed Cloud air-gapped release notes

February 16, 2024 [GDC 1.12.0]

  • Google Distributed Cloud air-gapped 1.12.0 is now available.
    See the product overview to learn about the features of Google Distributed Cloud air-gapped.
  • Google Distributed Cloud air-gapped 1.12.0 supports two operating systems:
    • Ubuntu 20231205
    • Rocky Linux 20231208

Updated Canonical Ubuntu OS image version to 20231208 to apply the latest security patches and important updates. To take advantage of the bug and security vulnerability fixes, you must upgrade all nodes with each release. The following security vulnerabilities are fixed:

The following container image security vulnerabilities are fixed:

Updated the gcr.io/distroless/base base image to digest sha256:eda29d6da91216123f1c1841b7c7a7abe1eece155cfe8d20d2a29e5daf9ad497 to apply the latest security patches and important updates.

Add-on Manager:

Build and package:

  • The Golang version is upgraded to 1.20.

  • Google Distributed Cloud air-gapped 1.12.0 adds additional Software Bills Of Materials (SBOMs) to output and updates logic to ensure such SBOMs are published in the future.

  • Google Distributed Cloud air-gapped 1.12.0 adds gdch_notice_license_files tar files to upload manifests.

Inventory management:

  • Google Distributed Cloud air-gapped 1.12.0 adds validations for hardware version 3.0 connection listings.
  • Google Distributed Cloud air-gapped 1.12.0 updates console server management port pattern to allow LAN1A and LAN2A.
  • Google Distributed Cloud air-gapped 1.12.0 adds a message to mitigate the confusion of PA850 active and passive modes.
  • Google Distributed Cloud air-gapped 1.12.0 supports a breakout cassette in validation.
  • Google Distributed Cloud air-gapped 1.12.0 adds validation of the permanent firewall to management firewall connection.
  • Google Distributed Cloud air-gapped 1.12.0 adds an OI Classless Inter-Domain Routing (CIDR) prompt to the Customer Intake Questionnaire generator.
  • Google Distributed Cloud air-gapped 1.12.0 improves an error message on mac address missing failure to mitigate preflight check instability when validating hsm and mgmtsw connection.

Operations Center IT organization:

  • The Operations Center IT organization has the following name updates:

    • Operations Center (OC) has been renamed to Operations Suite Facility (OIF).

    • OC Core has been renamed to Operations Suite Infrastructure Core Rack (OIR).

    • Operations Center IT (OCIT) has been renamed to Operations Suite Infrastructure (OI).

    • OCIT has been renamed to OI.

    For more information, see Terminology.

  • Google Distributed Cloud air-gapped 1.12.0 updates the Userlock configuration script to permit the use of a failover server.

  • Google Distributed Cloud air-gapped 1.12.0 pre-creates additional Operations Suite Infrastructure (OI) security groups to allow fine-grained access across OI systems.

System artifact registry:

  • Google Distributed Cloud air-gapped 1.12.0 removes -f (--force) short flag from CLI assets.

Version update:

  • The Debian-based image version is updated to bookworm-v1.0.0-gke.3.

Certificate manager:

  • Introduced configuration of the key size in a web-tls certificate for organizations.

Database Service:

  • Support for point-in-time recovery (PITR) for its Oracle databases.
  • Support for Postgres advanced migration to migrate on-premises databases to databases managed by the GDC database service.

Logging:

Marketplace:

  • MongoDB Enterprise Advanced (BYOL) is now available on the Google Distributed Cloud air-gapped 1.12.0 Marketplace.
    It is a collection of products and services that drive security, efficiency, and put you in control of your MongoDB databases.

Object storage:

  • Added new image required to host upgrade files in the object storage software.
  • Added an encryption version label to bucket webhooks.
  • Added a reconciler for object credential rotation.

Operations Suite Infrastructure Core Services (OIC)

  • Google Distributed Cloud air-gapped 1.12.0 collects OIC logs in Grafana.
  • Google Distributed Cloud air-gapped 1.12.0 moves the Copy-BareMetalFiles.ps1 script from installation documentation to scripts in private-cloud/operations/dsc/.
Platform security

  • A web TLS certificate for a root admin cluster is issued by the Google Distributed Cloud air-gapped internal public key infrastructure.

Security compliance:

  • Google Distributed Cloud air-gapped 1.12.0 introduces port security required to pass a security assessment.

Ticketing system

  • Updated scheduled jobs in ServiceNow to stagger when they are run to prevent database spikes.
  • Infrastructure Operator receives an alert when a meta monitoring incident in ServiceNow is stale.

Upgrade

  • Added the Upgrade Status dashboard for Infrastructure Operators and Platform Administrators.
  • Added a command for triggering user cluster upgrade.

Vertex AI:

Virtual machine management

  • Added Windows OS support for virtual machines to create, import, and connect to a Windows VM.

Billing:

  • Fixed the issue with the onetimeusage job always failing at updating labels on the onetimeusage object, causing failure alerts.
  • Fixed the issue causing the aggregate cost for a custom resource (CR) to duplicate when the job restarts after the cost of the CR writes to the database, before the label updates to processed.

Hardware security module:

  • Fixed the issue causing the hardware security module to frequently toggle between the ServicesNotStarted and ready states.

Hybrid identity:

  • Fixed the issue with network configuration in identity pods.

Inventory management:

  • Fixed the issue with the license parser not parsing object storage files whose license JSON text spreads multiple lines.
  • Fixed the issue with hardware 3.0 CellCfg CableType Validation regular expression.
  • Fixed the issue with including bootstrapper node in hardware validation.
  • Fixed the issue with root admin cluster node having SecureBootEnable turned off after server bootstrapping.

Operations Suite Infrastructure Core Services (OIC)

  • Fixed the issue with Initialize-BareMetalHost.ps1 not detecting that a reboot is required.
  • Fixed the issue with an enterprise CA root and not issuing a req file to submit for an offline CA root.
  • Fixed the issue where the OIC VM creation process left Hyper-V time synchronization enabled.

Ticketing system

  • Fixed MariaDB Audit issue.

Upgrade

  • Fixed the issue with Identity and Access Management (IAM) alerts by adding IAM postflight upgrade checks.

Virtual machine management

  • Fixed the issue with the VM status previously showing PendingIPAllocation if the VM is not able to be scheduled. After the fix, the VM status shows ErrorUnscheduable.
  • Fixed the issue with the incorrect object storage secret being used in VM image import operations.

Backup and restore:

  • Alerts for a backup repository may fire even when the repository is healthy.

Cluster management:

  • The machine-init job fails during cluster provisioning.

Physical servers:

  • The root admin cluster update progress is stuck at node upgrade, specifically NodeBIOSFirmwareUpgradeCompleted.

Database Service:

  • Database Service workloads operate within the system cluster, which could result in database workloads sharing compute infrastructure with other database instances and various control plane systems.

Harbor as a service (HAAS):

  • Being a preview feature of Google Distributed Cloud air-gapped 1.12.0, HaaS is not expected to operate in production environments.
    The preinstall job fails by design to prevent the subcomponents from reconciling properly, which prevents users from using HaaS.
    It is expected to find HaaS subcomponents in the reconciling state, which does not impact the functionality of other components.

Firewall:

  • During customer deployment, the secret.yaml file administrator username must be admin, and instead contains TO-BE-FILLED after the first creation. The admin username must be used to initialize the first configuration onto the firewall.

Hardware security module:

  • When deleting a KMS CTMKey, the PA might encounter unexpected behaviors, including the KMS service not starting for the organization.
  • A rotatable secret for hardware security modules is in an unknown state.
  • HSM internal certificate authority rotations get stuck and don't complete.

Logging:

Monitoring:

  • Node Exporter certificates might not become ready when creating an organization.
  • Some metrics from the user clusters are not collected. This issue affects the user VM clusters but not the system cluster.
  • The metrics storage class is incorrectly defined in the configuration.
  • The mon-prober-backend-prometheus-config ConfigMap gets reset to include no probe jobs, and alert MON-A0001 is triggered.

Node platform:

  • The node upgrade fails due to an outdated lvm.conf file.

Physical servers:

  • The root admin cluster update progress is stuck at node upgrade, specifically NodeBIOSFirmwareUpgradeCompleted.
  • When installing a server manually, the server installation might get stuck.

Upgrade:

  • Node upgrade fails for NodeOSInPlaceUpgradeCompleted.
  • Switch upgrade fails to run command install add bootflash://..
  • Several pods in a system cluster might get stuck in the TaintToleration state.

Upper networking:

  • A user VM cluster gets stuck in the ContainerCreating state with the FailedCreatePodSandBox warning.

Vertex AI:

  • The MonitoringTarget shows a Not Ready status when user clusters are being created, causing pre-trained APIs to continually show an Enabling state in the user interface.

VM Backup and Restore:

  • Role-based access control (RBAC) and schema settings in the VM manager is stopping users from starting VM backup and restore processes.
  • The VM image import fails at the image translation step due to insufficient disk size and timeout on obj storage proxy response.

SIEM:

  • OCLCM Preinstall jobs repeatedly fail on feature gate check.

Performance:

  • Google Distributed Cloud air-gapped 1.12.0 deprecates the ability to execute provision key benchmarks.