This document describes Symphony Connectors for Google Cloud, which provide integration between IBM Spectrum Symphony and Google Cloud. This integration lets you use Google Cloud resources as flexible compute capacity for your Symphony clusters. Symphony then manages the cluster by using host factory. Host factory is a component of IBM Spectrum Symphony that extends your cluster with compute hosts from cloud providers like Google Cloud to meet resource demand in your Symphony clusters.
Use Symphony Connectors for Google Cloud to supplement your existing cloud infrastructure with flexible Google Cloud resources, particularly for high performance computing (HPC) workloads. Symphony Connectors for Google Cloud supports both Compute Engine and Google Kubernetes Engine services.
When to use Symphony Connectors for Google Cloud
Symphony Connectors for Google Cloud is designed to bridge the gap between Symphony's host factory service and Google Cloud. Use Symphony Connectors for Google Cloud in the following scenarios:
Cloud bursting: When your on-premises cluster resources are insufficient, dynamically scale your Symphony workloads into Google Cloud to access additional compute capacity. Symphony Connectors for Google Cloud lets you automatically provision resources as needed, integrated with existing Symphony job scheduling.
Hybrid cloud deployments: If you run Symphony workloads across both on-premises and Google Cloud infrastructure, you can use this solution to unify resource management through the Symphony host factory framework. This approach creates a consistent experience for job submission and monitoring, and flexible resource allocation.
Cost optimization: Use Google Cloud's flexible instance types and pricing models, such as Spot VMs (preemptible VMs), to run cost-sensitive workloads.
Development and testing: To provision temporary compute resources for development or testing workflows, use Symphony Connectors for Google Cloud rapidly set up and tear down environments. Costs are optimized through automatic resource cleanup.
Unified management: Use the host factory to create a consistent interface to manage both GKE pods and the Compute Engine instances that are used by Symphony.
Seamless enterprise integration: Use this solution to extend your existing Symphony deployments into Google Cloud without requiring significant architectural changes.
Learn how Symphony Connectors for Google Cloud works
Symphony Connectors for Google Cloud provide the necessary components to make Google Cloud a provider for host factory.
The following diagram shows the architecture of Symphony Connectors for Google Cloud:
As shown in the preceding diagram, this architecture contains three main groups: an IBM Spectrum Symphony environment, a Google Cloud host factory provider layer, and Google Cloud services. The Symphony environment contains the host factory and provider scripts. This architecture uses the Google Cloud host factory provider CLI tools layer to translate requests, allowing the Symphony scripts to control Compute Engine resources and a Kubernetes operator running in GKE.
The following sections explain the main components of Symphony Connectors for Google Cloud.
IBM Spectrum Symphony host factory
An IBM Spectrum Symphony service that manages the lifecycle of compute hosts from various providers based on workload demand. This Symphony component is the software that Google Cloud integrates with.
Provider scripts
The Symphony host factory service uses a set of shell scripts to integrate with a cloud provider. These scripts perform actions like requesting and returning resources.
By default, Symphony doesn't include a Host factory provider for Google Cloud.
The Google Cloud host factory provider enables the Symphony host factory framework to communicate with Google Cloud.
The following table shows the scripts that Symphony Connectors for Google Cloud uses:
| Name | Description | IBM documentation |
|---|---|---|
getAvailableTemplates.sh |
Retrieves the available
resource templates defined in the provider configuration. Host factory
calls getAvailableTemplates.sh to know what types of
resources it can request. |
getAvailableTemplates |
requestMachines.sh |
Provisions new compute resources from Google Cloud. It receives a JSON payload specifying the template and quantity. | requestMachines |
requestReturnMachines.sh |
Host factory deprovisions compute resources used by the Symphony cluster. It receives a JSON payload that identifies the machines to be returned. | requestReturnMachines |
getRequestStatus.sh |
Host factory polls the status of ongoing provisioning requests
initiated by requestMachines.sh. It receives a JSON payload
that contains request IDs. |
getRequestStatus |
getReturnRequests.sh |
Host factory polls the
status of ongoing deprovisioning requests initiated by
requestReturnMachines.sh. It receives a JSON payload that
contains request IDs. | getReturnRequests |
Google Cloud host factory provider
The host factory provider is the interface created by Google that integrates with IBM Spectrum Symphony's host factory framework. This Google Cloud plugin includes CLI tools that Symphony calls to request and manage resources in Google Cloud. CLI tools for both Compute Engine and GKE workloads are available. Symphony Connectors for Google Cloud refers to these CLI tools as providers:
- For Compute Engine workloads, you use the Compute Engine provider.
- For GKE workloads, you use the GKE provider. For GKE workloads, you need an additional operator. This Kubernetes operator runs within your GKE cluster to manage the lifecycle of the IBM Spectrum Symphony compute pods requested using the GKE provider.
After you install these CLI tools, you can also manually call the Compute Engine and GKE provider CLI tools using Python. The CLI tools are Python modules, making them executable using the Python interpreter. For more information, see the Running from Python documentation on GitHub.
Compute Engine provider
The Compute Engine provider (hf-gce) is a CLI tool designed to
integrate Symphony's host factory with
Compute Engine. After you install the Compute Engine provider, you
must configure it in
Symphony host factory. Once this configuration is complete, host factory can
automatically invokes scripts that use the hf-gce CLI.
The Compute Engine provider interacts directly with the Google Compute Engine API to manage the lifecycle of VM instances based on Symphony's demands and the configurations defined in the provider instance and templates. It handles the creation, monitoring, and termination of Compute Engine VMs, allowing them to join the Symphony cluster as compute nodes.
GKE provider
The GKE provider (hf-gke) is a CLI tool that connects
Symphony's host factory service with GKE. Host factory uses
scripts that call the hf-gke CLI.
The GKE provider interacts with the
GKE cluster's API to create, query, and delete Custom Resource
Definitions (CRDs), specifically GCPSymphonyResource and
MachineReturnRequest.
The Kubernetes operator running within the GKE cluster handles the actual Pod lifecycle management based on these custom resources. The GKE provider acts as the bridge between Symphony's resource requests and the Kubernetes custom resource-based orchestration.
Google Cloud services
You can connect to Compute Engine and Google Kubernetes Engine. There are no additional components for Compute Engine connections. For Google Kubernetes Engine connections, there is a Kubernetes operator.
Kubernetes operator
You only need this additional operator for GKE workloads that you run using the GKE provider. This operator runs within your GKE cluster to manage the lifecycle of Symphony compute Pods requested using the GKE provider. The GKE provider has the following features:
- Asynchronous operations: supports high-performance asynchronous processing, suitable for large-scale deployments.
- Preemption handling: includes automatic detection and management of Spot VM preemptions.
- Custom resource definitions: uses
GCPSymphonyResourceandMachineReturnRequestcustom resources to manage resources. - Resource cleanup: automates cleanup of completed resources, with configurable retention periods.
- Health monitoring: monitors the integration of the GKE provider with built-in health checks and status reporting.
Limitations
Symphony Connectors for Google Cloud has the following limitations:
- Google Cloud does not create, manage, schedule, or execute your IBM Spectrum Symphony workloads. You are responsible for all workload management within the Symphony software.
What's next
Visit the GitHub documentation to install and configure the Google Cloud host factory provider: