Configure external datasets

This page describes an optional step to configure external datasets for the Cortex Framework Data Foundation deployment. Some advanced use cases might require external datasets to complement an enterprise system of record. In addition to external exchanges consumed from Analytics Hub, some datasets might need custom or tailored methods to ingest data and join them with the reporting models.

Configure the Directed Acyclic Graphs (DAGs) for the supported external datasets following these steps:

  1. Holiday Calendar: This DAG retrieves the special dates from PyPi Holidays.

    1. Adjust the list of countries, the list of years, as well as other DAG parameters to retrieve holidays in holiday_calendar.ini.

  2. Trends: This DAG retrieves Interest Over Time for a specific set of terms from Google Search trends. The terms can be configured in trends.ini.

    1. After an initial run, adjust the start_date to 'today 7-d' in trends.ini.
    2. Get familiarized with the results coming from the different terms to tune parameters.
    3. We recommend partitioning large lists to multiple copies of this DAG running at different times.
    4. For more information about the underlying library being used, see Pytrends.

  3. Weather: By default, this DAG uses the publicly available test dataset BigQuery-public-data.geo_openstreetmap.planet_layers. The query also relies on an NOAA dataset only available through Analytics Hub: noaa_global_forecast_system.

    This dataset needs to be created in the same region as the other datasets prior to executing deployment. If the datasets aren't available in your region, you can continue with the following instructions to transfer the data into the chosen region:

    1. Go to BigQuery Analytics Hub.
    2. Click Search Listings.
    3. Search for NOAA Global Forecast System.
    4. Click Add dataset to project.
    5. When prompted, keep noaa_global_forecast_system as the name of the dataset. If needed, adjust the name of the dataset and table in the FROM clauses in weather_daily.sql.
    6. Repeat the listing search for Dataset OpenStreetMap Public Dataset.
    7. Adjust the FROM clauses containing: BigQuery-public-data.geo_openstreetmap.planet_layers in postcode.sql.

  4. Sustainability and ESG insights: Cortex Framework combines SAP supplier performance data with advanced ESG insights to compare delivery performance, sustainability and risks more holistically across global operations. For more information, see the Dun & Bradstreet data source.

General considerations

  • Analytics Hub is only supported in EU and US locations, and some datasets, such as NOAA Global Forecast, are only offered in a single multi location.

    If you are targeting a location different from the one available for the required dataset, we recommended to create a scheduled query to copy the new records from the Analytics Hub linked dataset followed by a transfer service to copy those new records into a dataset located in the same location or region as the rest of your deployment. You then need to adjust the SQL files.

  • Before copying these DAGs to Cloud Composer, add the required python modules as dependencies:

    Required modules:
pytrends~=4.9.2
holidays