[[["容易理解","easyToUnderstand","thumb-up"],["確實解決了我的問題","solvedMyProblem","thumb-up"],["其他","otherUp","thumb-up"]],[["難以理解","hardToUnderstand","thumb-down"],["資訊或程式碼範例有誤","incorrectInformationOrSampleCode","thumb-down"],["缺少我需要的資訊/範例","missingTheInformationSamplesINeed","thumb-down"],["翻譯問題","translationIssue","thumb-down"],["其他","otherDown","thumb-down"]],["上次更新時間：2025-09-04 (世界標準時間)。"],[],[],null,["# Private cloud maintenance and updates\n=====================================\n\nPrivate cloud environments are designed in the following ways to have no single\npoint of failure:\n\n- ESXi clusters are configured with vSphere high availability (HA). Clusters are sized to have at least one spare node for resiliency.\n- vSAN provides redundant primary storage, requiring at least three nodes to provide protection against a single failure. For larger clusters, you can configure vSAN to provide higher resiliency.\n- vCenter, PSC, and NSX Manager virtual machines (VMs) are configured with RAID-10 storage to protect against storage failure. The VMs are additionally protected against node and network failures by vSphere HA.\n- ESXi hosts have redundant fans and NICs.\n- TOR and spine switches are configured in HA pairs to provide resiliency.\n\nVMware Engine continuously monitors uptime, monitors availability,\nand provides availability SLAs for the following types of VMs:\n\n- ESXi hosts\n- vCenter\n- PSC\n- NSX Manager\n\nVMware Engine continuously monitors the following for failures:\n\n- Hard disks\n- Physical NIC ports\n- Servers\n- Fans\n- Power\n- Switches\n- Switch ports\n\nIf a disk or node fails, VMware Engine immediately and automatically\nadds a new node to the affected VMware cluster to restore service operability. The following processes take place on your private cloud:\n\n- **Automated monitoring and alerting**: Our monitoring system constantly tracks the health of your nodes. When an issue is detected that indicates a potential hardware failure, an alert is triggered.\n- **Human-in-the-loop for diagnosis**: While the system is designed for automated replacement, our engineers review these alerts to quickly determine the root cause. This ensures that we're addressing the correct issue and prevents unnecessary node replacements when a simpler solution (like a reboot) is recommended. For example, temporary network issues or software glitches can trigger similar alerts to hardware failures, and we want to avoid impacting your cluster with node replacement when it might not be the recommended action. An unnecessary node replacement triggers a full vSAN Resync, which is a storage I/O intensive operation.\n- **Automated node replacement for hardware failures**: If our engineers confirm a hardware failure, the automated node replacement process begins immediately. A new node is added to your cluster, and vSAN initiates the data resync on that node.\n\nThe following VMware elements in private clouds are backed up, maintained, and\nupdated:\n\n- ESXi\n- vCenter Platform Services Controller\n- vSAN\n- NSX\n\nBackup and restore\n------------------\n\nBackups include the following:\n\n- Nightly incremental backups of vCenter, PSC, and DVS rules.\n- vCenter native APIs to back up components at the application layer.\n- Automatic backup prior to update or upgrade of the VMware management software.\n\nMaintenance\n-----------\n\nThe following types of planned maintenance are included.\n\n### Backend and internal maintenance\n\nBackend and internal maintenance typically involves reconfiguring physical\nassets or installing software patches. It doesn't affect normal consumption of\nthe assets being serviced. With redundant NICs going to each physical rack,\nnormal network traffic and private cloud operations aren't affected. You might\nnotice a performance impact only if your organization expects to use the full\nredundant bandwidth during the maintenance interval.\n\n### Portal maintenance\n\nSome limited service downtime is required when the control plane or\ninfrastructure is updated. Maintenance intervals can be as frequent as once per\nmonth, and are expected to decline in frequency over time.\nVMware Engine notifies you about impending portal maintenance and\nmakes an effort to keep the maintenance interval as short as possible. During a\nportal maintenance interval, the following services continue to function without\nany impact:\n\n- VMware management plane and applications\n- vCenter access\n- All networking and storage\n\n### VMware infrastructure maintenance\n\nIt's occasionally necessary to make changes to the configuration of the VMware\ninfrastructure. These intervals can occur every one to two months, but the\nfrequency is expected to decline over time. This type of maintenance can usually\nbe done without interrupting normal private cloud consumption. During a VMware\nmaintenance interval, the following services continue to function without any\nimpact:\n\n- VMware management plane and applications\n- vCenter access\n- All networking and storage\n\nUpdates and upgrades\n--------------------\n\nVMware Engine is responsible for lifecycle management of VMware\nsoftware (ESXi, vCenter, PSC, and NSX) in private clouds.\n\nSoftware updates include the following:\n\n- **Patches:** security patches or bug fixes released by VMware\n- **Updates:** minor version change of a VMware stack component\n- **Upgrades:** major version change of a VMware stack component\n\nVMware Engine tests critical security patches as soon as they become\navailable from VMware. Google will aim to commence the rollouts of relevant\ncritical patches to private cloud environments within one week of their\navailability. The actual patching completion timeline will vary depending on\nscheduling availability and the need to time the patching to avoid any downtime\nfor customer workloads.\n\nWhen a new major version of VMware software is available,\nVMware Engine works with customers to coordinate a suitable\nmaintenance window for applying the upgrade. VMware Engine applies\nmajor version upgrades at least six months after the major version is released\nand notifies customers one month in advance of applying major version upgrades.\n\nVMware Engine also works with key industry vendors to ensure that\nthey support the latest VMware software version before rolling out a major\nversion upgrade. For information about support for specific vendors, [contact\nCloud Customer Care](/vmware-engine/docs/getting-support).\n\n### Certificate update responsibility\n\nCertificate updates are a Google-owned responsibility. If you get a certificate\nupdate error, no action is required and the certificate is renewed before\nexpiration. However, if LDAPS is configured in your private cloud, you are\nsolely responsible for the specific certificate associated with that error.\n\n### Preparation\n\nGoogle recommends taking the following preparations before starting an update or\nupgrade:\n\n- **Check storage capacity:** Ensure your vSphere cluster's storage space utilization is lower than 80% to maintain the [SLA](/vmware-engine/sla). If the utilization is higher than 80%, upgrades might take longer than normal or fail completely. If your storage utilization is higher than 70%, [add a node](/vmware-engine/docs/private-clouds/howto-manage-private-cloud#add-nodes) to expand the cluster and avoid any potential downtime during upgrades.\n- **Change vSAN storage policies with FTT of 0:** Change VMs configured with a vSAN storage policy for Failures to Tolerate (FTT) of 0 to a vSAN storage policy with FTT of 1 to maintain the SLA.\n- **Remove VM CD mounts:** Remove any CDs mounted on your workload VMs that are not compatible with vMotion.\n- **Complete VMware tool installations:** Complete any installations or upgrades of VMware tools before the scheduled upgrade begins.\n- **Remove SCSI bus sharing on VMs:** Remove SCSI bus sharing on VMs if you don't want the VMs to be powered off.\n- **Remove inaccessible VMs and datastores:** Remove unused and inaccessible VMs from the vCenter inventory. Remove any inaccessible external datastores.\n- **Disable Distributed Resource Scheduler (DRS) rules:** DRS rules that pin a VM to a host prevent a node from entering maintenance mode. You can disable the DRS rules before the upgrade and enable them after the upgrade is complete.\n- **Update VMware add-ons and third-party solutions:** Verify that VMware add-ons and third party solutions deployed on your private cloud vCenter are compatible with the post-upgrade versions mentioned previously. Examples of tools include those for backup, monitoring, disaster recovery orchestration, and other similar functions. Check with the solution vendor and update ahead of time if necessary to ensure compatibility after the upgrade.\n\nConfigurations that might affect maintenance processes\n------------------------------------------------------\n\nVMware Engine leverages VMware's Maintenance Mode to carry out\nupgrades, updates, and node maintenance. This helps ensure continued operation\nof your Private Cloud workloads. However, the following configurations might\nrequire additional steps before a node can enter Maintenance Mode:\n\n- **DRS rules:** MUST rules that force VMs to stay on a specific node.\n- **SCSI bus sharing:** VMs configured to share SCSI buses.\n- **CD-ROM mounts:** VMs with CD-ROMs attached, especially if those CD-ROMs cannot be moved to another node using vMotion.\n- **Serial port connections:** VMs using serial port connections that prevent them from being moved to another node using vMotion.\n- **Raw device mappings (RDM):** VMs directly accessing physical storage devices.\n\n| **Important:** Proactively managing these configurations can help streamline future VMware Engine maintenance procedures.\n\n### If action is necessary\n\nIf any of these configurations exist on a node, Cloud Customer Care notifies you\nat least 24 hours before taking the remediation steps required to maintain the\navailability of your Private Cloud. In some cases, steps such as powering off a\nVM and moving it with vMotion and then powering it on, or removal of CD-ROMs,\nmight briefly disrupt your workload.\n\nWhat's next\n-----------\n\n- Learn about [VMware Engine security](/vmware-engine/docs/concepts-security)"]]
