Product overview

This page gives you an overview of Mainframe Connector, its capabilities, and key benefits.

IBM mainframes are used by organizations to perform critical computing tasks. In recent years, many companies that rely on mainframes have been working towards migrating to the cloud. Mainframe Connector lets you move your mainframe data to Google Cloud so that you can offload CPU-intensive report workloads to Google Cloud.

Key benefits of Mainframe Connector

The following are the key benefits of using Mainframe Connector to move mainframe data to Google Cloud:

  • Simplified data transfer: Simplifies the movement of mainframe data to Google Cloud storage services like Cloud Storage and BigQuery.
  • Batch job integration: Lets you submit BigQuery jobs using mainframe batch jobs defined in job control language (JCL). As queries are read in from datasets or files, analysts can use scheduled jobs with minimal knowledge and understanding of mainframe environments.
  • Easy monitoring: Mainframe operations personnel don't have to monitor a different environment as jobs are submitted with familiar schedules using JCL.
  • Reduced MIPS: Mainframe Connector uses a Java virtual machine (JVM) for most processing to minimize mainframe processor workload during data transfer, reducing million instructions per second (MIPS) thereby lowering costs. Mainframe Connector offloads most processor-intensive work to auxiliary processors. If auxiliary processors are stressed, you can also configure Mainframe Connector to perform transcoding and conversion using Compute Engine. For more information about Mainframe Connector configurations, see Mainframe Connector configurations.
  • Streaming transformation: Transcode queued sequential access method (QSAM) mainframe datasets associated with COBOL copybooks in extended binary coded decimal interchange code (EBCDIC), or files in ASCII UTF-8 to the ORC format which is compatible with Google Cloud services like BigQuery. By default, Mainframe Connector transcodes datasets from the US EBCDIC: Cp037 character set to ORC. However, Mainframe Connector also supports transcoding datasets from the following regional EBCDIC character sets:

    • French: Cp297
    • German: Cp1141
    • Spanish: Cp1145

    A custom character set can be implemented if an appropriate one is not included in the IBM JVM.

How Mainframe Connector works

Mainframe Connector lets you move data located on your mainframe in and out of Cloud Storage, and submit BigQuery jobs from mainframe-based batch jobs defined in JCL. Mainframe Connector lets you transcode mainframe datasets directly to Optimized Row Columnar (ORC) format.

Transcoding is the process of converting information from one form of coded representation to another, in this case to ORC. ORC is an open source column-oriented data format that is widely used in the Apache Hadoop ecosystem, and is supported by BigQuery.

Mainframe Connector provides a subset of the Google Cloud SDK command-line utilities enabling you to transfer data and interact with Google Cloud services. The shell interpreter and JVM-based implementations of gsutil and bq command-line utilities make it possible for you to manage a complete extract, load, transform (ELT) pipeline entirely from IBM z/OS while retaining your existing job scheduler.

One of the main challenges in transferring your mainframe data to and from the cloud is that it is a multi-step process that would normally include executing the following steps:

  1. Copy data to a file server.
  2. Copy data from the file server to another location for processing.
  3. Use a data processing stack to convert the data to a modern format.
  4. Write back the processed data to yet another location.
  5. Load the processed data to a database or a data warehouse where data can be queried or used.

The following figure shows the multi-step process that is normally used to transfer data from a mainframe to Google Cloud.

Multiple steps to move mainframe data to Google Cloud
Multiple steps to move mainframe data to Google Cloud

Mainframe Connector lets you perform all these steps with a single command using Cloud Storage as an intermediate storage location. This reduces the time taken for the mainframe data to be processed and made available in a database or data warehouse, as shown in the following figure.

Steps eliminated by Mainframe Connector
Steps eliminated by Mainframe Connector

What's next