Open-source and enterprise-ready: IBM Spectrum Symphony connectors for Google Cloud

At Google Cloud, we are committed to helping customers deploy their high performance computing (HPC) grid workloads to our platform. Today, we are thrilled to announce the general availability of open-source IBM Spectrum Symphony HostFactory connectors for Google Compute Engine and Google Kubernetes Engine (GKE).

This integration between Google Cloud and IBM Spectrum Symphony gives you access to the benefits of Google Cloud for your grid workloads by supporting common architectures and requirements, namely:

• Extending your on-premises cluster to Google Cloud and automatically adding compute capacity to reduce execution time of your jobs, or

• Deploying an entire cluster in Google Cloud and automatically provisioning and decommissioning compute resources based on your workloads

These connectors are provided in the form of IBM Spectrum Symphony HostFactory custom cloud providers. They are open source and can be easily deployed either via Cluster Toolkit or manually.

Partner-built and tested for enterprise scale

To deliver robust, production-ready connectors, we collaborated with key partners who have deep expertise in financial services and HPC. Accenture built the Compute Engine and GKE connectors and Aneo performed rigorous user acceptance testing to ensure they met the stringent demands of our enterprise customers.

“Accenture is proud to have collaborated with Google Cloud to help develop the IBM Spectrum Symphony connectors. Our expertise in both financial services and cloud solutions allows us to enable customers to seamlessly migrate their critical HPC workloads to Google Cloud's high-performance infrastructure." - Keith Jackson, Managing Director - Financial Services, Accenture

“At Aneo, we subjected the IBM Spectrum Symphony connectors to rigorous, large-scale testing to ensure they meet the demanding performance and scalability requirements of enterprise HPC. We validated the connector's ability to efficiently manage up to 5,000 server nodes, confirming its readiness for production workloads." - William Simon Horn, Cloud HPC Engineer, and Wilfried Kirschenmann, CTO, Aneo

Google Cloud rapidly scales to meet extreme HPC demands, provisioning over 100,000 vCPUs across 5,000 compute pods in under 8 minutes with the new IBM Spectrum Symphony connector for GKE. IBM has tested and supports Spectrum Symphony up to 5,000 compute nodes, so we set this as our target for scale testing the new GCP connector.

We achieved this performance by leveraging innovative GKE features like image preloading and custom compute classes, enabling customers in demanding sectors like FSI to accelerate mission-critical workloads while optimizing for cost and hybrid cloud flexibility. 

Powerful features to run your way

The connectors are built to provide the flexibility and control you need to manage complex HPC environments. They are available as open-source software in a Google-owned repository. Key features include:

• Support for Compute Engine and GKE: Separate IBM Spectrum Symphony Host Factory cloud providers for Compute Engine and GKE allow you to scale your cluster across both virtual machines and containerized environments.

• Flexible consumption models: Support for Spot VMs, on-demand VMs, or a mix of both lets you optimize cost and performance.

• Template-based provisioning: Use configurable resource templates that align with your workloads requirements.

• Comprehensive instance support: Full integration with managed instance group (MIG) APIs, GPUs, Local SSD, and Confidential Computing VMs.

• Event-driven management: Pub/Sub integration allows for event-driven resource management for Compute Engine instances.

• Kubernetes-native: The GKE connector uses a custom Kubernetes operator with Custom Resource Definitions (CRDs) to manage the entire lifecycle of Symphony compute pods. Leverage GKE’s scaling capabilities and custom hardware like GPUs and TPUs through transparent compatibility with GKE custom computeClasses (CCC) and Node Pool Autoscaler.

• High-scalability: The connectors are built for high-performance with asynchronous operations to handle large-scale deployments.

• Resiliency: Automatic detection and handling of Spot VM preemptions helps ensure workload reliability.

• Logging and monitoring: Integrated with Google Cloud's operations suite for observability and reporting.

• Enterprise support: The connectors are supported as a first-party solution by Google Cloud, with an established escalation path to our development partner, Accenture.

Getting started

You can begin using the IBM Spectrum Symphony connectors for Google Cloud today.

• Find the connectors in the Google Cloud repository.

• Explore the technical documentation, including the reference architecture, to get started.

• Contact Google Cloud or your Google Cloud account team to learn more about how to migrate your HPC workloads.

To help ensure your success, we will continue to invest in the solutions you need to accelerate your research and business goals. We look forward to seeing what you can achieve with the scale and power of Google Cloud.