A Replication job is a CDAP application, and can be managed with CDAP REST APIs.
Create a Replication job
Use the CDAP application deployment API to create a Replication job.
PUT -H "Authorization: Bearer ${AUTH_TOKEN}" "${CDAP_ENDPOINT}/v3/namespaces/namespace-id/apps/replicator-name"
| Parameter | Description |
|---|---|
namespace-id |
The namespace in which to create the Replication job.
If your pipeline belongs to a Basic edition instance, the namespace ID
is always default. If your pipeline belongs to an
Enterprise edition instance, you can
create a namespace. |
replicator-name |
Replicator name |
Example Config
The following (partial) config of the app is sent in the API request body:
{
"name": "my-sample-pipeline",
"artifact": {
"name": "delta-app",
"version": "0.6.2",
"scope": "SYSTEM"
},
"config": {
"connections": [
{
"from": "Oracle (by Datastream)",
"to": "BigQuery"
}
],
"stages": [
{
"name": "Oracle (by Datastream)",
"plugin": {
"name": "OracleDatastream",
"type": "cdcSource",
"artifact": {
"name": "datastream-delta-plugins",
"version": "0.4.2",
"scope": "SYSTEM"
},
"properties": {
"usingExistingStream": "false",
"region": "us-central1",
"connectivityMethod": "ip-allowlisting",
"port": "1521",
"sid": "ORCL",
"replicateExistingData": "true",
"project": "auto-detect",
"dsServiceAccountKey": "auto-detect",
"gcsServiceAccountKey": "auto-detect",
"host": "",
"user": "",
"password": ""
}
}
},
{
"name": "BigQuery",
"plugin": {
"name": "bigquery",
"type": "cdcTarget",
"artifact": {
"name": "bigquery-delta-plugins",
"version": "0.6.3",
"scope": "SYSTEM"
},
"properties": {
"project": "auto-detect",
"serviceAccountKey": "auto-detect",
"stagingBucketLocation": "us",
"loadInterval": "90",
"stagingTablePrefix": "_staging_",
"requireManualDrops": "false",
"softDeletes": "false",
"datasetName": "mysampledataset"
}
}
}
],
"tables": [
{
"database": "ORCL",
"table": "MYTABLE1",
"schema": "HR"
},
{
"database": "ORCL",
"table": "MYTABLE2",
"schema": "HR"
}
],
"parallelism": {
"numInstances": 1
},
"tableTransformations": []
}
}
See the CDAP application deployment API for more information.
List Replication jobs
Use the CDAP application list API to list Replication jobs.
GET -H "Authorization: Bearer ${AUTH_TOKEN}" "${CDAP_ENDPOINT}/v3/namespaces/namespace-id/apps"
| Parameter | Description |
|---|---|
namespace-id |
The namespace of the Replication jobs to list.
If your pipeline belongs to a Basic edition instance, the namespace ID
is always default. |
See the CDAP application list API for more information.
View Replication job details
Use the CDAP application detail API to view the details of a Replication job.
GET -H "Authorization: Bearer ${AUTH_TOKEN}" "${CDAP_ENDPOINT}/v3/namespaces/namespace-id/apps/replicator-name"
| Parameter | Description |
|---|---|
namespace-id |
The namespace of the Replication job to view.
If your pipeline belongs to a Basic edition instance, the namespace ID
is always default. |
replicator-name |
Replicator name |
See the CDAP application detail API for more information.
Start a Replication job
Use the CDAP program start API to start a Replication job.
POST -H "Authorization: Bearer ${AUTH_TOKEN}" "${CDAP_ENDPOINT}/v3/namespaces/namespace-id/apps/replicator-name/workers/DeltaWorker/start"
| Parameter | Description |
|---|---|
namespace-id |
The namespace of the Replication job to start.
If your pipeline belongs to a Basic edition instance, the namespace ID
is always default. |
replicator-name |
Replicator name |
See the CDAP program start API for more information.
Stop a Replication job
Use the CDAP program stop API to stop a Replication job.
POST -H "Authorization: Bearer ${AUTH_TOKEN}" "${CDAP_ENDPOINT}/v3/namespaces/namespace-id/apps/replicator-name/workers/DeltaWorker/stop"
| Parameter | Description |
|---|---|
namespace-id |
The namespace of the Replication job to stop.
If your pipeline belongs to a Basic edition instance, the namespace ID
is always default. |
replicator-name |
Replicator name |
See the CDAP program stop API for more information.