A replication job is a CDAP application, and can be managed with CDAP REST APIs.
Create a replication job
Use the CDAP application deployment API to create a replication job.
PUT -H "Authorization: Bearer ${AUTH_TOKEN}" "${CDAP_ENDPOINT}/v3/namespaces/namespace-id/apps/replicator-name"
Parameter | Description |
---|---|
namespace-id |
The namespace in which to create the replication job.
If your pipeline belongs to a Basic edition instance, the namespace ID
is always default . If your pipeline belongs to an
Enterprise edition instance, you can
create a namespace. |
replicator-name |
Replicator name |
Example Config
The following (partial) config of the app is sent in the API request body:
{ "name": "my-sample-pipeline", "artifact": { "name": "delta-app", "version": "0.6.2", "scope": "SYSTEM" }, "config": { "connections": [ { "from": "Oracle (by Datastream)", "to": "BigQuery" } ], "stages": [ { "name": "Oracle (by Datastream)", "plugin": { "name": "OracleDatastream", "type": "cdcSource", "artifact": { "name": "datastream-delta-plugins", "version": "0.4.2", "scope": "SYSTEM" }, "properties": { "usingExistingStream": "false", "region": "us-central1", "connectivityMethod": "ip-allowlisting", "port": "1521", "sid": "ORCL", "replicateExistingData": "true", "project": "auto-detect", "dsServiceAccountKey": "auto-detect", "gcsServiceAccountKey": "auto-detect", "host": "", "user": " ", "password": " " } } }, { "name": "BigQuery", "plugin": { "name": "bigquery", "type": "cdcTarget", "artifact": { "name": "bigquery-delta-plugins", "version": "0.6.3", "scope": "SYSTEM" }, "properties": { "project": "auto-detect", "serviceAccountKey": "auto-detect", "stagingBucketLocation": "us", "loadInterval": "90", "stagingTablePrefix": "_staging_", "requireManualDrops": "false", "softDeletes": "false", "datasetName": "mysampledataset" } } } ], "tables": [ { "database": "ORCL", "table": "MYTABLE1", "schema": "HR" }, { "database": "ORCL", "table": "MYTABLE2", "schema": "HR" } ], "parallelism": { "numInstances": 1 }, "tableTransformations": [] } }
Contents of the replication job config depend on the Cloud Data Fusion instance and plugin versions. To get started, export an existing replication job or draft to use as a template for creating the request body.
See the CDAP application deployment API for more information.
List replication jobs
Use the CDAP application list API to list replication jobs.
GET -H "Authorization: Bearer ${AUTH_TOKEN}" "${CDAP_ENDPOINT}/v3/namespaces/namespace-id/apps"
Parameter | Description |
---|---|
namespace-id |
The namespace of the replication jobs to list.
If your pipeline belongs to a Basic edition instance, the namespace ID
is always default . |
See the CDAP application list API for more information.
View replication job details
Use the CDAP application detail API to view the details of a replication job.
GET -H "Authorization: Bearer ${AUTH_TOKEN}" "${CDAP_ENDPOINT}/v3/namespaces/namespace-id/apps/replicator-name"
Parameter | Description |
---|---|
namespace-id |
The namespace of the replication job to view.
If your pipeline belongs to a Basic edition instance, the namespace ID
is always default . |
replicator-name |
Replicator name |
See the CDAP application detail API for more information.
Start a replication job
Use the CDAP program start API to start a replication job.
POST -H "Authorization: Bearer ${AUTH_TOKEN}" "${CDAP_ENDPOINT}/v3/namespaces/namespace-id/apps/replicator-name/workers/DeltaWorker/start"
Parameter | Description |
---|---|
namespace-id |
The namespace of the replication job to start.
If your pipeline belongs to a Basic edition instance, the namespace ID
is always default . |
replicator-name |
Replicator name |
See the CDAP program start API for more information.
Stop a replication job
Use the CDAP program stop API to stop a replication job.
POST -H "Authorization: Bearer ${AUTH_TOKEN}" "${CDAP_ENDPOINT}/v3/namespaces/namespace-id/apps/replicator-name/workers/DeltaWorker/stop"
Parameter | Description |
---|---|
namespace-id |
The namespace of the replication job to stop.
If your pipeline belongs to a Basic edition instance, the namespace ID
is always default . |
replicator-name |
Replicator name |
See the CDAP program stop API for more information.
What's next
- Learn more about Replication in Cloud Data Fusion.