Cortex Framework: integration with LiveRamp
This page describes the required configurations to bring data from LiveRamp as a data source of the marketing workload of Cortex Framework Data Foundation.
LiveRamp is a data collaboration platform that helps businesses connect, control, and activate their data to improve customer experiences and drive better business outcomes. Cortex Framework offers the tools and platform to analyze it, combine it with other data sources, and use AI to gain deeper insights and optimize your marketing strategy.
The following diagram describes how LiveRamp data source is available through the marketing workload of Cortex Framework Data Foundation:
Configuration file
The config.json
file configures the settings required to connect to data sources for transferring
data from various workloads. This file contains the following parameters for LiveRamp:
"marketing": {
"deployLiveRamp": true,
"LiveRamp": {
"datasets": {
"cdc": ""
}
}
}
The following table describes the value for each LiveRamp parameter:
Parameter | Meaning | Default Value | Description |
marketing.LiveRamp
|
Deploy LiveRamp | true
|
Execute the deployment for LiveRamp data source. |
marketing.LiveRamp.datasets.cdc
|
CDC dataset for LiveRamp | CDC dataset for LiveRamp. |
API connection
Cortex Framework Data Foundation enables Identity Resolution by integrating with LiveRamp.Cortex Frameworks can perform RampID lookup, which allow businesses to identify known audiences or customer segments from their CRM systems.
RampID is an identifier generated by LiveRamp's API using Personally Identifiable Information (PII) like email, phone number, and name. This RampID allows businesses to identify and consolidate records across multiple systems, such as audiences of different campaigns. By integrating with LiveRamp, businesses can improve their ability to identify and target their audiences, leading to more effective and personalized marketing campaigns.
LiveRamp Identity Resolution Retrieval API lets businesses to programmatically resolve PII data to individuals. Cortex Framework uses LiveRamp's Lookup Endpoint by sending hashed PII data over the API call.
Input and output tables
After deploying Cortex Framework,
the system creates the following two BigQuery tables in
relevant dataset provided in config.json
.
rampid_lookup_input
table
The rampid_lookup_input
table is the input for the RampID Lookup process.
Column | Data Type | Description | Example | Primary Key |
id | STRING | Unique ID of this record. | "123" | Yes |
segment_name | STRING | Name of audience/CRM/Customer segment. | "High Value" | No |
source_system_name | STRING | Source system where the record came from. | "Salesforce" | No |
name | STRING | Customer Name | "John Doe" | No |
STRING | Customer Email | "example@example.com" | No | |
phone_number | STRING | Customer Phone | "1234567890" | No |
postal_code | STRING | Customer postal code | "12345" | No |
is_processed | BOOL |
Indicates if a record is already processed.
For new records, populate this with FALSE. System updates this to TRUE once it's processed. |
FALSE | No |
load_timestamp | TIMESTAMP | Timestamp when the record was inserted in the system. This is purely for audit purpose. | "2020-01-01 00:00:00 UTC" | No |
processed_timestamp | TIMESTAMP | Timestamp when system performed API Lookup for this record. This is always populated by system. | "2020-01-01 00:00:00 UTC" | No |
The rampid_lookup_input
table needs to be populated on a periodic basis
(based on your business needs) with your customer's PII details like.
Cortex Framework does not provide an automated way to do so, but it
provides a sample script ddls/samples/populate_rampid_lookup_input.sql
that shows how you could populate this table using data in your
Salesforce system already deployed with Cortex Framework. You can use
this file as a guide in case your data comes from another system.
Ensure you don't have duplicate entries in the rampid_lookup_input
table
(for example, the same individual is present multiple times with same PII information,
even if their ID
might be different). Cortex Framework Lookup DAG would
fail if a segment contains many duplicate entries. This is enforced by
LiveRamp APIs.
The rampid_lookup
table
The rampid_lookup
table is an output table containing RampIDs for each
segment in the input record. LiveRamp by design does not allow to map RampID
to an individual record.
Column | Data Type | Description |
segment_name | STRING | Segment name from input table. |
ramp_id | STRING | LiveRamp RampID |
record stamp | TIMESTAMP | Timestamp when this RampID lookup was performed. |
LiveRamp RampIDs can change over a period of time,
for the same individual. This means, you need to perform a fresh lookup for
already processed data from time to time. Cortex Framework provides a
sample script ddls/samples/clean_up_segment_matching.sql
that shows how you can do this at a segment level. This way, you can reset
a whole segment, and the system would perform a lookup for that segment and
give you up to date RampIDs.
The rampid_lookup
(output table) might contain slightly less number of
records compared to the input table. This is by design, as
Cortex Framework tries to debug the input records using PII details
to ensure LiveRamp API lookup does not fail.
Account authentication
- Contact LiveRamp to obtain authentication credentials. This should include Client ID and Client Secret.
Create a secret using Secret Manager with name
cortex-framework-liveramp
, and use the following as value. See instructions in Secret-Manager documentation.{ 'client_id':'CLIENT_ID', 'client_secret':'CLIENT_SECRET', 'grant_type':'client_credentials' }
Replace the following:
- 'CLIENT_ID' with the client ID obtained from step 1.
- 'CLIENT_SECRET' with the client secret obtained from step 1.
Cloud Composer connections
Create the following connections in Cloud Composer. See more details in the Manage Airflow connections documentation.
Connection Name | Purpose |
liveramp_cdc_bq
|
For LiveRamp API > CDC dataset transfer |
Configuration
The config.ini
file controls some behavior of the Cloud Composer DAG,
as well as how LiveRamp APIs are consumed. Configure the LiveRamp/src/pipelines/config.ini
file
based on your needs. While these parameters are already described in the file,
pay attention to liveramp_api_base_url
. This parameter by default points
to LiveRamp's production API URL, for testing purposes, you might need to
point it to the staging version, depending on your setup.
What's next?
- For more information about other data sources and workloads, see Data sources and workloads.
- For more information about the steps for deployment in production environments, see Cortex Framework Data Foundation deployment prerequisites.