透過集合功能整理內容
你可以依據偏好儲存及分類內容。
升級建議
本頁說明如何從自訂的 Cortex Framework Data Foundation 升級至新版本。每次發布新版本時,Cortex 團隊都會盡量減少中斷,同時為 Cortex Framework 新增功能。新版更新會優先考量回溯相容性。不過,本指南可協助您盡量減少可能發生的問題。
Cortex Framework 資料基礎
提供一組預先定義的內容和範本,可加快從複製到 BigQuery 的資料中發掘價值。
機構可根據自身需求,調整這些範本、模組、SQL、Python 指令碼、管道和其他提供的內容。
核心元件
Cortex Framework 資料基礎內容的設計原則是開放性。機構可以選擇最適合自己的工具,處理提供的 BigQuery 資料模型。基礎架構唯一緊密依附的平台是 BigQuery。所有其他工具都可以視需要互換:
- 資料整合:只要整合工具能與 BigQuery 互連,且可複製原始資料表和結構,就能加以運用。舉例來說,原始資料表應與在 SAP 中建立的結構定義相同 (名稱、欄位和資料類型都相同)。此外,整合工具應能提供基本轉換服務,例如更新目標資料類型以符合 BigQuery 相容性,以及新增時間戳記或作業標記等額外欄位,以醒目顯示新記錄和變更的記錄。
- 資料處理:變更資料擷取 (CDC) 處理指令碼可與 Cloud Composer (或 Apache Airflow) 搭配使用,但並非必要。反之,SQL 陳述式會盡可能與 Airflow 專屬檔案分開產生,因此客戶可視需要使用其他工具中的個別 SQL 檔案。
- 資料視覺化:系統提供 Looker 資訊主頁範本,內含視覺化內容和最低邏輯,但核心邏輯仍會保留在 BigQuery 的資料基礎中,方便您使用偏好的報表工具建立視覺化內容。
主要優點
Cortex Framework 資料基礎架構可因應各種業務需求。這項服務的元件具有彈性,可讓機構根據特定需求調整平台,並享有下列優點:
- 開放性:除了 BigQuery 之外,還能與各種資料整合、處理和視覺化工具完美整合。
- 自訂:機構可以修改及擴充預建元件 (例如 SQL 檢視畫面),以符合資料模型和業務邏輯。
- 效能最佳化:可根據個別工作負載和資料量調整分割、資料品質檢查和叢集等技術。
- 回溯相容性:Cortex 致力於在日後發布的版本中維持回溯相容性,盡量減少對現有實作項目的影響。如要瞭解版本異動,請參閱「版本資訊」。
- 社群貢獻:鼓勵使用者分享知識及協作。
更新程序
以下各節將說明開發人員如何透過其中一種方式,讓程式碼與 Cortex Framework Data Foundation 存放區保持同步,同時保留自訂項目。在 CI/CD 管道中使用預先交付的部署指令碼。不過,機構可以採用其他工具和方法,例如 Dataform,或不同 Git 主機提供的自動化工具 (例如 GitHub 動作),以符合自身偏好。
設定存放區
本節概述設定存放區的方法。建議您先充分瞭解 Git,再按照這些步驟操作。
分叉核心存放區:
建立 Cortex Framework Data
Foundation 存放區的分叉。分叉會讓該存放區持續接收 Google Cloud 存放區的更新,並為公司主要建立獨立存放區。
建立公司存放區:為公司存放區建立新的 Git 主機 (例如 Cloud Source)。在新主機上,建立與已分支存放區同名的存放區。
初始化公司存放區:將從分支存放區複製的程式碼貼到新建立的公司存放區。使用下列指令,將原始的分叉存放區新增為上游遠端存放區,並確認已新增遠端存放區。這會在公司存放區和原始存放區之間建立連線。
git remote add google <<remote URL>>
git remote -v
git push --all google
驗證存放區設定:確認公司存放區包含複製的程式碼和記錄。您應該會看到兩個遠端,分別是 origin 和您在使用指令後新增的遠端:
git remote -v:
您現在擁有存放區 (即「公司存放區」),開發人員可以在其中提交變更。開發人員現在可以複製新存放區中的分支並在其中工作。
將變更內容與新版 Cortex 合併
本節說明如何合併公司存放區的變更,以及 Google Cloud 存放區的變更。
更新分支:按一下「Sync fork」,即可使用 Google Cloud 存放區的變更內容更新存放區的分支。舉例來說,以下變更會套用至公司存放區。此外, Google Cloud 在最新版本中對 Data Foundation 存放區進行了其他變更。
- 在 SQL 中建立並納入新檢視表的使用
- 修改現有檢視區塊
- 完全以自己的邏輯取代指令碼
下列指令序列會將分支存放區新增為上游遠端存放區,從 GitHub 提取更新版本,並將其主要分支簽出為 GitHub-main。接著,這個範例會從「來源」 Google Cloud 中公司的存放區簽出主要分支版本,並建立名為 merging_br
的分支版本以供合併。
git remote add github <<github fork>>
git fetch github main
git checkout -b github-main github/main
git checkout main
git checkout -b merging_br
您可以透過多種方式建構這項流程,合併程序也可能在 GitHub 的分支版本中進行,並以重新設定基準取代合併,合併分支版本也可能以合併要求的形式傳送。這些程序差異取決於目前的機構政策、變更深度和便利性。
完成這項設定後,您就能比較傳入的變更與本機變更。建議您在所選的圖形 IDE 中使用工具,查看變更並選擇要合併的內容。例如 Visual Studio。
建議使用醒目的註解標記自訂項目,方便進行差異比較。
開始合併程序:使用建立的分支 (在本例中,分支稱為 merging_br
) 匯集所有變更並捨棄檔案。準備就緒後,您可以將這個分支併回公司存放區的主要分支或其他分支,以建立合併要求。從公司存放區主要分支 (git checkout merging_br
) 中簽出的合併分支,合併來自遠端分支的變更。
## git branch -a
## The command shows github-main which was created from the GitHub fork
## You are in merging_br
git merge github-main
## If you don't want a list of the commits coming from GitHub in your history, use `--squash`
這項指令會產生衝突清單。使用圖形化 IDE 比較功能瞭解變更,並選擇「目前」、「即將推出」和「兩者」。這時,程式碼中與自訂項目相關的註解就派上用場了。
您可以選擇完全捨棄變更、刪除不想合併的檔案,以及忽略已自訂檢視區或指令碼的變更。
合併變更:決定要套用哪些變更後,請檢查摘要並使用下列指令提交變更:
git status
## If something doesn't look right, you can use git rm or git restore accordingly
git add --all #Or . or individual files
git commit -m "Your commit message"
如果您對任何步驟感到不安全,請參閱 Git 基本復原操作。
測試及部署:目前您只會合併到「暫時」分支。
建議您此時從 cloudbuild\*.yaml
指令碼執行測試部署,確保一切運作正常。自動化測試有助於簡化這項程序。確認這個合併分支沒問題後,您可以簽出主要目標分支,然後將 merging_br
分支合併到該分支。
除非另有註明,否則本頁面中的內容是採用創用 CC 姓名標示 4.0 授權,程式碼範例則為阿帕契 2.0 授權。詳情請參閱《Google Developers 網站政策》。Java 是 Oracle 和/或其關聯企業的註冊商標。
上次更新時間:2025-09-04 (世界標準時間)。
[[["容易理解","easyToUnderstand","thumb-up"],["確實解決了我的問題","solvedMyProblem","thumb-up"],["其他","otherUp","thumb-up"]],[["難以理解","hardToUnderstand","thumb-down"],["資訊或程式碼範例有誤","incorrectInformationOrSampleCode","thumb-down"],["缺少我需要的資訊/範例","missingTheInformationSamplesINeed","thumb-down"],["翻譯問題","translationIssue","thumb-down"],["其他","otherDown","thumb-down"]],["上次更新時間:2025-09-04 (世界標準時間)。"],[[["\u003cp\u003eThis guide provides instructions for upgrading to new versions of the Cortex Framework Data Foundation while maintaining customizations made to fit organizational needs.\u003c/p\u003e\n"],["\u003cp\u003eCortex Framework Data Foundation is designed with openness and customization in mind, allowing organizations to utilize a variety of data integration, processing, and visualization tools alongside BigQuery.\u003c/p\u003e\n"],["\u003cp\u003eThe core components of Cortex Framework Data Foundation offer flexibility, enabling organizations to interchange tools, customize SQL views, and adjust performance optimization techniques to align with their specific needs.\u003c/p\u003e\n"],["\u003cp\u003eThe upgrade process involves forking the core repository, setting up a company repository, merging changes from the new Cortex release, and strategically addressing conflicts to retain customizations, and can be done with multiple variations.\u003c/p\u003e\n"],["\u003cp\u003eThe recommended process includes thorough testing and deployment of merged changes to ensure compatibility and proper functionality within the company's customized environment, before merging it into the main branch.\u003c/p\u003e\n"]]],[],null,["# Upgrade recommendations\n=======================\n\nThis page describes recommendations to upgrade to new versions from customized\n[Cortex Framework Data Foundation](https://github.com/GoogleCloudPlatform/cortex-data-foundation).\nOn every release, the Cortex team commits to minimize disruptions while it add\nnew features to the Cortex Framework. New updates prioritize backward\ncompatibility. However, this guide helps you minimize the possible issues.\n\n[Cortex Framework Data Foundation](https://github.com/GoogleCloudPlatform/cortex-data-foundation)\nprovides a set of predefined content and templates to accelerate value from\ndata replicated into [BigQuery](https://cloud.google.com/bigquery).\nOrganizations adapt these templates, modules, SQL, Python scripts, pipelines\nand other content provided to fit their needs.\n\nCore components\n---------------\n\nCortex Framework Data Foundation content is designed with a principle of openness in mind.\nOrganizations can use the tools that work best for them when working with\nthe BigQuery data models provided. The only platform on which\nthe foundation has a tight dependency on is BigQuery. All\nother tools can be interchanged as required:\n\n- **Data Integration:** Any integration tool that has interconnectivity with BigQuery can be leveraged provided it can replicate raw tables and structures. For example, raw tables should resemble the same schema as they were created in SAP (same names, fields, and data types). In addition, the integration tool should be able to provide basic transformation services such as updating target data types for BigQuery compatibility as well as adding additional fields like timestamp or operations flag for highlighting new and changed records.\n- **Data Processing:** The Change Data Capture (CDC) processing scripts provide work with [Cloud Composer](https://cloud.google.com/composer) (or Apache Airflow) are optional. Conversely, the SQL statements are produced separately from the Airflow-specific files where possible, so that customers can make use of the separate SQL files in another tool as needed.\n- **Data Visualization:** While [Looker](https://cloud.google.com/looker) dashboarding templates are provided and contain visualizations and minimum logic, the core logic remains available in the data foundation within BigQuery by design to create visualizations with their reporting tool of choice.\n\nKey benefits\n------------\n\nCortex Framework Data Foundation is designed to be adaptable to\nvarious business needs. Its components are built with flexibility,\nallowing organizations to tailor the platform to their specific\nrequirements and getting the following benefits:\n\n- **Openness**: Integrates seamlessly with various data integration, processing, and visualization tools beyond BigQuery.\n- **Customization:** Organizations can modify and expand pre built components like SQL views to match their data models and business logic.\n- **Performance Optimization:** Techniques like partitioning, data quality checks, and clustering can be adjusted based on individual workloads and data volumes.\n- **Backward Compatibility:** Cortex strives to maintain backward compatibility in future releases, minimizing disruption to existing implementations. For information about version changes, see the [Release Notes](/cortex/docs/release-notes).\n- **Community Contribution:** Encourages knowledge sharing and collaboration among users.\n\nUpdate process\n--------------\n\nThe following sections share the instructions for one way in which developers\ncan keep their code up-to-date with the Cortex Framework Data Foundation repository while\nretaining their customizations. Use of the pre-delivered deployment scripts in\nCI/CD pipelines. However, organizations can employ alternative tools and\nmethodologies to suit their preferences, such as [Dataform](/dataform/docs),\nor automation tools provided by the different Git hosts, such as GitHub actions.\n\n### Set up your repository\n\nThis section outlines one approach to setting up your repository. Before following\nthese steps, a solid understanding of Git is recommended.\n\n1. **[Fork](https://github.com/GoogleCloudPlatform/cortex-data-foundation/fork) core repository** :\n Create a fork of the Cortex Framework Data\n Foundation repository. The fork keeps\n that repository receiving updates from the Google Cloud repository, and a\n separate repository for the *Company's main*.\n\n2. **Create Company Repository**: Establish a new Git host for your\n company's repository (for example, Cloud Source). Create a repository with the same\n names as your forked repository on the new host.\n\n3. **Initialize Company Repository** : Copy the code from your forked Repository\n into the newly created company repository. Add the original forked repository as an\n [upstream remote repository](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/configuring-a-remote-repository-for-a-fork) with the following command,\n and verify the remote has been added. This establishes a connection between\n your company repository and the original repository.\n\n git remote add google \u003c\u003cremote URL\u003e\u003e\n git remote -v\n git push --all google\n\n4. **Verify Repository Setup**: Ensure your company repository contains the\n cloned code and history. You should see the two remotes,\n origin and the one you added after using the command:\n\n git remote -v:\n\n You now have the repository, the *Company's repository*, where\n developers can submit their changes. Developers can now clone and work in\n branches in the new repository.\n\n### Merge your changes with a new Cortex release\n\nThis section describes the process of merging changes from the *Company's repository*\nand changes coming from the Google Cloud repository.\n\n1. **Update forks** : Click **Sync fork** to update your forks for your\n repository with the changes from the Google Cloud repository. For example,\n the following changes to the *Company's repository* are done. And there has\n been some other changes in the Data Foundation repository by Google Cloud in a\n new release.\n\n - Created and incorporated the use of a new view in SQL\n - Modified existing views\n - Replaced a script entirely with our own logic\n\n The following commands sequence adds the fork repository as\n an upstream remote repository to pull the updated release from as *GitHub*\n and checks out its main branch as *GitHub-main.* Then, this example checks\n out the main branch from the *Company's repository* in Google Cloud Source\n and creates a branch for merging called `merging_br`. \n\n git remote add github \u003c\u003cgithub fork\u003e\u003e\n git fetch github main\n git checkout -b github-main github/main\n git checkout main\n git checkout -b merging_br\n\n There are multiple ways to build this flow. The merging process could also\n happen in the fork in GitHub, be replaced by a rebase instead of a merge,\n and the merging branch could also be sent as a merge request. These variations\n of the process depend on current organizational policies, depth of changes\n and convenience.\n\n With this setup in place, you can compare the incoming changes to your local\n changes. It's recommended to use a tool in a graphic IDE of choice to see the\n changes and choose what gets merged. For example, Visual Studio.\n\n It's recommended flagging customizations using comments that stand out\n visually, to make the diff process easier.\n2. **Start the merge process** : Use the created branch (in this example, is\n the branch called `merging_br`) to converge all changes\n and discard files. When ready, you can merge this branch back into the main or\n another branch for your Company's repository to create a merge request. From\n that merging branch that was checked out from your Company's repository's main\n (`git checkout merging_br`), merge the incoming changes from the remote fork.\n\n ## git branch -a\n ## The command shows github-main which was created from the GitHub fork\n ## You are in merging_br\n\n git merge github-main\n\n ## If you don't want a list of the commits coming from GitHub in your history, use `--squash`\n\n This command generates a list of conflicts. Use the graphical IDE comparison\n to understand the changes and choose between *current* , *incoming* and *both*.\n This is where having a comment in the code around customizations becomes handy.\n Choose to discard changes altogether, delete files that you don't want to\n merge at all and ignore changes to views or scripts that you have already customized.\n3. **Merge changes**: After you have decided on the changes to apply, check the\n summary and commit them with the command:\n\n git status\n ## If something doesn't look right, you can use git rm or git restore accordingly\n git add --all #Or . or individual files\n git commit -m \"Your commit message\"\n\n If you feel insecure about any step, see [Git basic undoing things](https://git-scm.com/book/en/v2/Git-Basics-Undoing-Things).\n4. **Test and deploy** : So far you are only merging into a \"temporary\" branch.\n It's recommended running a test deployment from the `cloudbuild\\*.yaml` scripts\n at this point to make sure everything is executing as expected. Automated\n testing can help streamline this process. Once this merging branch looks good,\n you can checkout your main target branch and merge the `merging_br` branch into it."]]