Release notes

This page documents production updates to Google Cloud Cortex Framework. Check this page for announcements about new or updated features, bug fixes, known issues, and deprecated functionality.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.

May 14, 2025

6.x

Release 6.3.1

  • A dedicated Cortex-specific Cloud Build Service Account will now be automatically created during the deployment process for both the standard Quickstart demo and the Cortex for Meridian Quickstart demo.
  • When executing gcloud builds submit command for Cortex Framework deployments, the _BUILD_ACCOUNT parameter is now required. If you are updating an existing deployment, ensure you provide the appropriate build account in the following format: _BUILD_ACCOUNT='projects/SOURCE_PROJECT/serviceAccounts/SERVICE_ACCOUNT@SOURCE_PROJECT.iam.gserviceaccount.com'.
  • Resolved inconsistencies between the standard and Cortex for Meridian Quickstart demos related to Cloud Build Service Account behavior.
  • The Quickstart demo now automatically enables the Data Catalog API during its execution.
  • Oracle builds may time out when using a private worker pool created with default parameters.

April 22, 2025

6.x

Release 6.3

  • Cortex for Meridian.
    • Cortex Framework for Meridian, Google's open-source Marketing Mix Modeling (MMM) tool (v1.0.5), delivers ready data models and automation of Meridian Model execution using Google Cloud tools like Enterprise Colab and Cloud Workflows.
    • This integration empowers users to make data-driven marketing decisions by providing accurate campaign performance measurement and budget optimization.
    • Cortex for Meridian simplifies the pre-modeling process by gathering and transforming data from core Cortex Framework data sources, including:
  • Task Dependent DAGs:
    • Provided out-of-the-box, recommended DAGs and task dependencies for SAP ECC/S4 reporting.
    • Enabled the creation and deployment of customized task-dependent reporting settings for all data sources.
  • CATGAP has been deprecated.
  • SAP Machine Learning (ML) models have been deprecated.

March 04, 2025

6.x

Release 6.2

  • Minor refactoring for Data Mesh configuration specs code, which are now moved to the /src/common/py_libs directory.

The following standalone accelerators and samples are removed from Cortex Framework:

  • Demand Sensing
  • AppLayer
  • CATGAP
  • SAP ML Model samples

December 19, 2024

6.x

Release 6.1

  • SAP Annotations: All SAP Reporting views and fields are now fully annotated with functional descriptions and business context. Deploy Data Mesh to take advantage of this feature.
  • SAP Hierarchy Reader: As announced in the previous release notes, the hier_reader code has been fully deprecated. Relevant SAMPLE scripts have been updated to use the new hierarchy reader DAG output tables.
  • SAP Fiscal and Currency functions: As announced in the previous release notes, these functions have been removed. Please use the relevant tables (currency_conversion, currency_decimal, and fiscal_date_dim) instead.
  • Google Trends DAG: The Google Trends API calls issued by this DAG may intermittently fail. If this happens, try rerunning the DAG.
  • 1-click deployer: The 1-click deployer for Oracle EBS currently requires manual naming. Autoname mode is not yet supported.

October 21, 2024

6.x

Release 6.0.1

  • Quickstart demo updates for permissions with legacy accounts.
  • CATGAP Python library vulnerability updates.

September 30, 2024

6.x

Release 6.0

GitHub Submodules

Submodules are officially removed. Workloads have been migrated as follows:

Previous Submodule New Directory
cortex-dag-generator cortex-data-foundation/tree/main/src/SAP/SAP_CDC
cortex-ml-models cortex-data-foundation/tree/main/src/SAP/SAP_ML_MODELS
cortex-reporting cortex-data-foundation/tree/main/src/SAP/SAP_REPORTING
cortex-salesforce cortex-data-foundation/tree/main/src/SFDC
cortex-marketing cortex-data-foundation/tree/main/src/marketing

Note: As announced in release 5.4, workload specific repositories will continue to receive code updates for at least one more release.

New Data Sources

  • Marketing: YouTube (with DV360). Get comprehensive overview of your campaign spend, performance and audience targeting with actionable insights for paid media campaigns on YouTube purchased in DV360 platform. Access the Looker Block for YouTube (with DV360) with sample dashboards for faster data analysis and exploration.

  • Marketing: Google Analytics 4 (GA4). Understand website engagement and purchases using page views, time spent, call to action ratios, average purchases, lifetime value over time, traffic volumes, and success rates for defined conversion.

  • Oracle EBS: Measure and understand Order to Cash operational metrics around sales performance, order status, order fulfillment, invoicing, and receivables with new BigQuery integration templates and data models. Access the Looker Block for Oracle EBS with sample dashboards for further analytics.

SAP

  • SAP Raw to CDC DAG: Rewrote logic and became less costly with about 25% improvement in performance.
  • Fiscal dimension: All views now use the new Fiscal Dimension instead of function. ECC/S4 specific Fiscal Functions are now removed as announced in v3.1. Fiscal functions will be removed in the next version.
  • Future deprecation: Legacy Currency_Conversion and Currency_Decimal functions will be deprecated in the next version. Join the views currency_conversion and currency_decimal instead.
  • Finance: New views for FSV, Cost, and Profit center hierarchy that supersedes the old hier_reader local k9, which will be removed in the next version. For more information, see Cortex Framework: integration with SAP.
  • Improved SalesStatus_Items and S/4 Billing logic.

  • Inventory: Moved the following views and functions from src/SAP/SAP_REPORTING/local_k9/inventory_snapshots to src/SAP/SAP_REPORTING/{SAP_FLAVOR}; and renamed them to make the structure more logical and succinct. Also added notes and instructions on this module in the Cortex Framework: integration with SAP . There isn't any content or logic change.

    • From 00_stock_characteristics_config to StockCharacteristicsConfig.
    • From 00_slow_moving_threshold to SlowMovingThreshold.
    • From 0_{weekly|monthly}_inventory_aggregation.sql to Aggregate{Weekly|Monthly}Inventory.
    • From stock_weekly_snapshots_inventory_aggregation_update to UpdateMonthlyInventoryAggregation.
    • From 01_stock_{weekly|monthly}_snapshots to Update{Weekly|Monthly}StockSnapshots.

New documentation page

Known issues and limitations

  • Data Mesh deployments where only column descriptions are deployed require Data Catalog API to be enabled. This will be patched in a future release.
  • Local K9 for SAP produces a temporary folder (tmp*) in the target bucket for DAGs. Please, remove it manually. There is no impact in the execution.

May 15, 2024

5.x

Release 5.4.1

  • Realigned table schema directory location for Google Ads, CM360 and TikTok, by moving it from src/marketing/src/SOURCE/src/table_schema to src/marketing/src/SOURCE/config/table_schema matching other marketing sources.
  • Reporting table refresh operation is now atomic, which ensures data is available in reporting during table refresh.

Google Ads

  • Renamed Keywords view to KeywordStats.
  • Added new reporting views to support campaign level metrics by country (CampaignStats, CampaignStatsByUserCountry, GeoTargetConstant), and new experimental views (CampaignDailyAgg, CampaignDailyAggByUserCountry).
  • Enabled auto discovery for all accessible customer IDs. Therefore login_customer_id no longer needs to be explicitly specified in the Secret Manager configuration.
  • Improved readability for raw view queries.

Meta

  • Updated source raw request specification format to YAML. Required fields can now be specified in a readable format.
  • Maximum load days and per request delay can be configured in src/marketing/src/Meta/src/raw/pipelines/config.ini.
  • Added table annotations for reporting views.
  • Fixed a bug for source raw load date range, causing data to be backfilled multiple times in edge cases.
  • Included performance optimizations of the hierarchy flattener code for SAP Finance Financial Statement Version, Profit Center, and Cost Center hierarchies.

K9

  • Converted scores to INTEGER type from STRING based on source definition. Sustainability views are now GA.
  • Holiday Calendar updated to correctly load data for most recent years.
  • Added support for polymorphic fields to Salesforce Sales Cloud (SFDC).
  • SFDC encoding text is now configurable in src/SFDC/src/raw_dag_generator/dependencies/config.ini. Default value is UTF-8.

Repository READMEs

  • Moved SAP and Salesforce Sales Cloud (SFDC) content from main README to their respective README files.
  • Added Google BigQuery Connector for SAP configuration considerations to the README_SAP.

January 19, 2024

5.x

Release 5.3.1

  • Fixed race condition and numbering sorting that leads to Cloud Build sequence bypass (SAP_REPORTING/common/materializer/generate_build_files.py).
  • Experimental preview of hierarchy flattener and base view for P&L Report. These artifacts are subject to change to include further optimizations, but provide a preview of an upcoming feature.

January 04, 2024

5.x

Release 5.3

  • New experimental functionality to deploy Data Mesh definitions leveraging Data Catalog, Dataplex and BigQuery. Current functionality includes definitions for lakes, zones, access policies, and other exciting features. By default, configuration only creates descriptions in BigQuery and doesn't require any additional configuration.
  • Experimental Balance Sheet report with financial statement version flattening is available. Due to potential performance issues, the underlying data processing pipeline is under development for optimization and would be improved in a future release based on user input. Review CDC settings, as ACDOCA, BSEG and BKPF have modified clustering and partition settings.
  • New persisted and optimized fiscal time dimension.
  • Google Ads ingestion pipeline upgrade to Google Ads API version 15.
  • Minor modifications to marketing ingestion pipelines for LiveRamp: streamlined imports for Python libraries.
  • Directory external_dags for SAP was converted into local K9.
  • Modification on AccountsPayable for S4 to match invoice based on reference.

Known issues and limitations

  • Local K9 for SAP produces a temporary folder (tmp) in the target bucket for DAGS. Remove it manually. There is no impact in the execution.
  • Nested fields aren't supported yet by Data Mesh annotations. They would be incorporated in a follow-up release.
  • It's expected that data in the test harness for finance doesn't zero out in the balance sheet because it is incomplete.

November 20, 2023

5.x

Release 5.2.1

  • Performance improvements for SAP Inventory. Updated InventoryByPlant and InventoryKeyMetrics for ECC and S4 to reduce memory processing requirements. Updated partitions and clusters for StockMonthlySnapshots and StockWeeklySnapshot.
  • Increasing Cloud Build timeout for SAP reporting deployment and Data Foundation deployer.

September 28, 2023

5.x

Release 5.2

  • LiveRamp for Marketing: New integration templates and data models to enable identity resolution by integrating with LiveRamp.
  • SAP Reusable DAGs are now deployed via the K9 deployment mechanism so they now behave like other cross-workload DAGs. This applies to all DAGs in the external_dag directory. These DAGs no longer search for SQL files from bq_data_replication directory. If you have modified the SQLs for them, follow these steps to migrate:
    1. Back up any SQL file(s) you have modified from gs://<your_airflow_bucket>/data/bq_data_replication.
    2. If any of the relevant DAG(s) are running, pause them in Airflow.
    3. Deploy Cortex Data Foundations v5.2.
    4. Copy your modified SQL file into the deployment directory, which by default is gs://<your_deployment_bucket>/dags/sap/reporting/<dag_name>.
    5. Copy /dags directory from the deployment bucket into your Airflow bucket. Then, wait for Airflow to refresh, and unpause the DAGs in Airflow.
    6. Remove migrated files from gs://<your_airflow_bucket>/data/bq_data_replication.
  • Metadata for SAP and TikTok: New experimental annotations feature for SAP artifacts. The deployer now creates views with descriptions for views and their fields.
  • Support for new regions: New supported regions incorporated into the test harness.
  • Fix for set hierarchy reader in SAP, generating a node with no associated main data for the header.

February 17, 2023

4.x

Release 4.1

  • More options for Salesforce integration with provided scripts or external tools: End-to-end integration from Salesforce APIs (provided out-of-the-box), structure mapping with no change data capture processing and optional CDC and mapping generation scripts.
  • Option for sequential deployment (TURBO=false) for SFDC incorporated.
  • Salesforce integration now updates the RAW tables with changes, merging changes on landing. This removes the need for additional CDC processing. Deltas are captured using SystemModstamp provided by Salesforce APIs.
  • IsArchived flag is removed from CDC processing for Salesforce.
  • Errors originating from gsutil steps in cloudbuild.sfdc.yaml not finding files to copy are now caught and surfaced gracefully.
  • Removing some substitution defaults (for example, LOCATION) from cloudbuild.yaml file so all configurations are either passed from the command line or read from config/config.json. These parameters would be removed from substitution defaults in future releases.
  • Detecting version for Airflow in DAG templates to use updated libraries for Airflow v2 in SAP and Salesforce. This removes some deprecation warnings, but it might need additional libraries installed in your Airflow instance.
  • Fix for test harness data not loading in an intended location when the location isn't passed as a substitution.
  • Checking existence of DAG-generated files before attempting to copy with gsutil to avoid errors.
  • Structure of RAW landed tables has changed to not require additional DAG processing. Review the documentation about mapping and use of the new extraction process before upgrading to avoid disruption.

January 16, 2023

4.x

Release 4.0

  • New Salesforce data source. The module includes optional integration and CDC scripts and reporting views. For more information, see the Entity Relationship Diagram for a list of tables and views and the Looker repository for sample dashboards.
  • New configuration file (config/config.json) for deployment parameters. We maintain backward compatibility with the existing gcloud builds submit command parameters. Enhanced parameters like SAP UNION datasets and those for Salesforce, can be configured in config.json. This file format replaces the .env format for SAP_REPORTING in the next releases.
  • New preview models for SAP Finance: AccountingDocumentsReceivable, AccountsPayable, AccountsPayableTurnover, CashDiscountUtilization, CurrencyConversion (exposing table processed by DAG), VendorPerformance, and VendorConfig (dependant on TVARVC values).
  • New split views for SAP Order-To-Cash: SalesOrderPricing and PricingConditions. For more information, see the updated Looker reports to make use of the split models.

Known issues and limitations

  • Salesforce integration DAGs have been tested in Airflow 1.0 and Airflow 2.0. They might require library updates or use of backwards compatible libraries to be tested and confirmed in the next release.
  • Finance views for SAP are good candidates for partial or total materialization. Check BigQuery's execution details to identify opportunities to create materialization processes and further optimizations that fit your data best.

September 07, 2022

3.x

Release 3.0

  • New submodule structure. All SAP-related submodules have been moved under the directory src/SAP/.
  • Submodules now execute independently. SAP's ML Models and SAP Reporting submodules now have their own YAML file for independent execution.
  • New experimental UNION logic available if your SAP ECC and S/4 systems don't share the same MANDT. All views are now moved to /ecc or /s4 respectively. This alters the templates of all of the views and requires two sets of datasets (for example, one raw landing for ECC, one for S/4). These datasets can be passed to the deployment in the file sap_config.env.
  • SAP Reporting can now be deployed using a configuration file (sap_config.env) instead of parameters from the gcloud builds submit --substitutions command. Parameters received from the substitutions override configurations in the file to maintain backwards compatibility. This affects all the cloudbuild.yaml files.
  • New simplified view for Sales Orders (_SalesOrders_V2_) with less underlying joins for use cases that don't require them.
  • New experimental views for a collaboration with C3.ai. These views are considered experimental as we increase the functionality around and on top of them in collaboration with C3.ai. Experimental views are flagged with an ## EXPERIMENTAL comment.
  • Turbo mode for SAP Reporting: The file sap_config.env contains a variable called TURBO. If set to true, the cloudbuild.yaml file is generated dynamically from the dependencies file. This reduces the deployment time.
  • Candidates for enhancement points within the code, that is portions where customers should consider incorporating changes or applying extensions, are flagged with a comment ## CORTEX-CUSTOMER.
  • Templatized currency and language for views that require the values to be hardcoded for better performance.
  • New non-changing views created as tables by default. Sources that are unlikely to ever change (for example, material document types, valuation areas) are now created as tables, instead of views, by default.
  • Added the following new views: WeatherDaily, SalesOrders_V2, DeliveryBlockingReasonsMD, BillingBlockingReasonsMD, StorageLocationsMD, PurchaseDocumentTypesMD, MovementTypesMD, ReasonForMovementTypesMD, SpecialStocksMD, PurchasingOrganizationsMD, BatchesMD, MaterialPlantsMD, ProductionOrders, PurchasingGroupsMD, MaterialsValuation, MaterialsMovement, MaterialTypesMD, BillOfMaterialsMD, DivisionsMD, and ValuationAreasMD.
  • Added new DAG for currency conversion and currency decimal shifting.
  • When test data is set to true, the dynamically generated MERGE operation now copies the test data from the raw landing dataset into the CDC dataset. Test data in CDC dataset isn't copied as it is from the raw landing. Test data is still meant to provide a base for specific demo scenarios and isn't guaranteed to be a reliable source with any quality or referential integrity.
  • Started marking releases with tags. This release is v3.0
  • Views for ECC are now matched in S4.
  • New guide with recommendations for upgrades.

August 15, 2022

2.x

Release 2.2

  • New external datasets available: Holidays, Weather, Trends and product hierarchy flattener to feed the Trends search terms.
  • Region-specific deployments with test data are now possible. For more information about supported regions and limitations of regional datasets in BigQuery, see BigQuery locations.
  • Addressing records with the same key and operation flag sent more than once in different chunks into raw landing.
  • Escaping of table and column names in DAG and real-time view generation to avoid issues with reserved words (Issue #7).
  • Minor tweak to CustomersMD: Coalesce address between customer main data and latest record in ADRC.
  • Minor correction to OrderToCash: Use CDC dataset as source for VBRP.

Known issues

  • Analytics hub supports regions in the EU and US . For more information, see Analytics hub introduction. If deploying the Weather DAG in a specific location where the source linked datasets aren't available, you might need to resort to another source for weather data or create a scheduled query to feed a copy in the same location, and use a transfer service to copy the records into a table in the desired location.

May 01, 2022

2.x

Release 2.1.1

  • Fixed AddressesMD date format to 9999-12-31.
  • Merged original 800 and 200 demo client tables into test harness for ECC (now it has both 800 and 100). This client was mostly used for demos.
  • Commenting out deployment of Product Recommender ML model as it requires specific reservations and a sample dataset that is only offered in the US region.
  • Addressing Issue #4.
  • Ignoring .INCLUDE when generating DAGs and Runtime views.
  • Fixed RUNTIME view logic generation affecting keys with deletion.
  • New Looker blocks available replacing the older blocks.

March 01, 2022

2.x

Release 2.1

Features and improvements

  • Templatized deployment and SQL Flavour for views, setting.yaml file and test harness to deploy either ECC or S4:
    1. New split source of data with MANDT = 050 for ECC and MANDT = 100 for S/4. These populates automatically based on the _SQL_FLAVOUR parameter in the build (default to ECC).
    2. Original dataset from ECC and S/4 mixed still present in the source bucket (MANDT = 800).
    3. The following views would be adapted to S/4 with source tables in a next minor release: FixedAssets, SDStatus_Items, GLDocumentsHdr, BSID and BSAD implementations, InvoiceDocuments_flow,DeliveriesStatus_PerSalesOrg, SalesFulfillment_perOrder, SalesFulfillment, and UoMUsage_SAMPLE.
  • Additional views for ECC and S/4 with Accounts Receivables and On-Time/In-Full analysis: Billing, OrderToCas, and AccountingDocumentsReceivable.
  • Additional helper functions for currency conversion.
  • Additional function for Due Date calculation for discounts.
  • New helper function for Fiscal period.
  • Renamed CDC parameters to match Reporting and ML Models.
  • SAP Reporting now has a SQL Flavour switch to correctly deploy S4 or ECC based on the _SQL_FLAVOUR parameter. Possible values are ECC or S4.
  • Enhanced reporting and calculation logics in: Deliveries, PurchaseDocuments and SalesOrders.
  • Reduced CDC deployment copy step by 80% implementing multithreaded copy.
  • Optimized overall build time when Cloud Build has the container cached, improving overall deployment time by 50%.

Bug Fixes

  • Fix for Runtime views not generated from settings.yaml.
  • Deployment dependencies causing errors on view creation.

Known issues

  • Hierarchy flattener not configured to find sets in new test data.
  • .INCLUDE in some tables in some versions of S/4 is known to be marked as key, which results in invalid SQLs in DAG generation. Recommendation as of now is to clear the keyflag field in the offending .INCLUDE entry in the replicated DD03L table in BigQuery when generating the DAG.

New

  • Cortex Application Layer: Google Marketplace offering showcasing how to consume the Cortex Data Foundation in your own cloud-native applications.
  • Looker Blocks dashboards for Orders Fulfillment, Order Snapshot, Order Details, Sales Performance, Billing, and Pricing.