Replication of SAP data into BigQuery by using the BigQuery Toolkit for SAP

For replication of SAP data into BigQuery, you use the data load class /GOOG/CL_BQTR_DATA_LOAD within your SAP applications such as custom ABAP programs, user exits, Business Add-Ins (BAdIs), and enhancements. Your SAP application needs to pass the data to be replicated as an input to the class.

Call the data replication method

You instantiate the BigQuery data load class /GOOG/CL_BQTR_DATA_LOAD by passing the mass transfer key and the name of the dictionary object configured in the BigQuery Data Transfer configuration.

To send the data to BigQuery, you call the REPLICATE_DATA method of the class /GOOG/CL_BQTR_DATA_LOAD.

DATA(lo_bq_data_load) =
  NEW /GOOG/CL_BQTR_DATA_LOAD( IV_MASS_TR_KEY = 'MASS_TRANSFER_KEY'
                               IV_DATA_SOURCE = 'DATA_SOURCE'
                               IV_CDC_FRAMEWORK = 'CDC_FRAMEWORK'
                               IV_FLDNM_MANDT = 'MANDT_FIELDNAME'
                               IV_MANDT_VALUE = 'MANDT_VALUE'
 ).
"Write Logic to populate ITAB_DATA, WA_HEADER

"Replicate Data to BigQuery
lo_bq_data_load->replicate_data(
  EXPORTING
    it_content       = ITAB_DATA
    IS_HEADER_INFO   = WA_HEADER
  IMPORTING
    ev_error_code    = data(lv_error_code)
    et_return        = data(lt_return) ).

Replace the following:

  • MASS_TRANSFER_KEY: The mass transfer key configured in the BigQuery Data Transfer configuration.
  • DATA_SOURCE: The name of the SAP dictionary object such as table, dictionary view, or CDS view.

  • CDC_FRAMEWORK: Optional. The Change Data Capture (CDC) framework used by the calling program. The supported values are ODQ and SLT.

    The BigQuery Toolkit for SAP has built-in logic to determine the operation flag (insert, update, and delete) based on the CDC framework. For more information, see Extra fields for record changes and count queries.

    When CDC_FRAMEWORK is passed as ODQ, the BigQuery Toolkit for SAP gets the value of column ODQ_CHANGEMODE in payload passed in the input:

    • If the value = C, the operation_flag is set to I.
    • If the value = U, the operation_flag is set to U.
    • If the value = D, the operation_flag is set to D.
    • For all other values, the operation_flag is set to L.

    When CDC_FRAMEWORK is passed as SLT, the BigQuery Toolkit for SAP gets the value of column 'OPERATION' from WA_HEADER:

    • If the value = I, the operation_flag is set to I.
    • If the value = U, the operation_flag is set to U.
    • If the value = D, the operation_flag is set to D.
    • For all other values, the operation_flag is set to L.

    If CDC_FRAMEWORK is not passed or passed as any other value, then the operation_flag is set to L.

  • MANDT_FIELDNAME: Optional. The name of the SAP client field that needs to be appended to the BigQuery table definition. This can be used for SAP dictionary objects such as CDS entities, which don't have the client field in their definition, but you want the field to be added to BigQuery.

  • MANDT_VALUE: Optional. The value of the SAP client field that needs to be populated in BigQuery. This can be only used when IV_FLDNM_MANDT is also populated.

  • ITAB_DATA: An internal table of type DATA_SOURCE. You populate ITAB_DATA with the data that needs to be replicated to the BigQuery table.

  • WA_HEADER: A flat structure, which contains header information about the replication. This is used to determine the operation_flag when CDC_FRAMEWORK is SLT.

When you call this method, the BigQuery Data Transfer module does the following actions:

  • Sync BigQuery table definition with SAP dictionary object definition based on the table and field settings maintained in BigQuery Data Transfer configuration.
  • Map data populated in ITAB_DATA from SAP format to BigQuery format based on field level settings maintained in the BigQuery Data Transfer configuration. If field level settings are not maintained, then the default data mapping rules are used.
  • Call the BigQuery API InsertAllTabledata method to load data to the target BigQuery table.

If all steps are successful, then the exporting parameter ev_error_code returns the value 0. If there was an error in any step, ev_error_code returns a non-zero value and the corresponding error message is populated in et_return.

As an application developer, you can handle errors, including saving them for later analysis and retrying the loading of records that failed replication.

Get support

Google Cloud offers support for issues and questions related to the installation, configuration, operation, and maintenance of the BigQuery Toolkit for SAP. However, support is limited to the toolkit itself.

Google Cloud doesn't support other environment components like network infrastructure, databases, operating systems, or third-party software. For issues related to any environment components other than the BigQuery Toolkit for SAP, contact the appropriate vendor or support provider.

For functionalities delivered by SAP, such as Operational Data Provisioning (ODP) and SAP Landscape Transformation (SLT), contact SAP support for assistance.

To troubleshoot problems with the BigQuery Toolkit for SAP, see Troubleshooting the BigQuery Toolkit for SAP.

If you need help resolving problems with the ABAP SDK for Google Cloud, then do the following: