REST Resource: projects.locations.workflows

Resource: MigrationWorkflow

A migration workflow which specifies what needs to be done for an EDW migration.

JSON representation
{
  "name": string,
  "displayName": string,
  "tasks": {
    string: {
      object (MigrationTask)
    },
    ...
  },
  "state": enum (State),
  "createTime": string,
  "lastUpdateTime": string
}
Fields
name

string

Output only. Immutable. Identifier. The unique identifier for the migration workflow. The ID is server-generated.

Example: projects/123/locations/us/workflows/345

displayName

string

The display name of the workflow. This can be set to give a workflow a descriptive name. There is no guarantee or enforcement of uniqueness.

tasks

map (key: string, value: object (MigrationTask))

The tasks in a workflow in a named map. The name (i.e. key) has no meaning and is merely a convenient way to address a specific task in a workflow.

state

enum (State)

Output only. That status of the workflow.

createTime

string (Timestamp format)

Time when the workflow was created.

lastUpdateTime

string (Timestamp format)

Time when the workflow was last updated.

MigrationTask

A single task for a migration which has details about the configuration of the task.

JSON representation
{
  "id": string,
  "type": string,
  "details": {
    "@type": string,
    field1: ...,
    ...
  },
  "state": enum (State),
  "processingError": {
    object (ErrorInfo)
  },
  "createTime": string,
  "lastUpdateTime": string,
  "orchestrationResult": {
    object (MigrationTaskOrchestrationResult)
  },
  "resourceErrorDetails": [
    {
      object (ResourceErrorDetail)
    }
  ],
  "resourceErrorCount": integer,
  "metrics": [
    {
      object (TimeSeries)
    }
  ],

  // Union field task_details can be only one of the following:
  "assessmentTaskDetails": {
    object (AssessmentTaskDetails)
  },
  "translationTaskDetails": {
    object (TranslationTaskDetails)
  },
  "translationConfigDetails": {
    object (TranslationConfigDetails)
  },
  "translationDetails": {
    object (TranslationDetails)
  }
  // End of list of possible types for union field task_details.
}
Fields
id

string

Output only. Immutable. The unique identifier for the migration task. The ID is server-generated.

type

string

The type of the task. This must be one of the supported task types: Translation_Teradata2BQ, Translation_Redshift2BQ, Translation_Bteq2BQ, Translation_Oracle2BQ, Translation_HiveQL2BQ, Translation_SparkSQL2BQ, Translation_Snowflake2BQ, Translation_Netezza2BQ, Translation_AzureSynapse2BQ, Translation_Vertica2BQ, Translation_SQLServer2BQ, Translation_Presto2BQ, Translation_MySQL2BQ, Translation_Postgresql2BQ.

details

object

DEPRECATED! Use one of the task_details below. The details of the task. The type URL must be one of the supported task details messages and correspond to the Task's type.

state

enum (State)

Output only. The current state of the task.

processingError

object (ErrorInfo)

Output only. An explanation that may be populated when the task is in FAILED state.

createTime

string (Timestamp format)

Time when the task was created.

lastUpdateTime

string (Timestamp format)

Time when the task was last updated.

orchestrationResult

object (MigrationTaskOrchestrationResult)

Output only. Additional information about the orchestration.

resourceErrorDetails[]

object (ResourceErrorDetail)

Output only. Provides details to errors and issues encountered while processing the task. Presence of error details does not mean that the task failed.

resourceErrorCount

integer

The number or resources with errors. Note: This is not the total number of errors as each resource can have more than one error. This is used to indicate truncation by having a resourceErrorCount that is higher than the size of resourceErrorDetails.

metrics[]

object (TimeSeries)

The metrics for the task.

Union field task_details. The details of the task. task_details can be only one of the following:
assessmentTaskDetails

object (AssessmentTaskDetails)

Task configuration for Assessment.

translationTaskDetails

object (TranslationTaskDetails)

Task configuration for Batch SQL Translation.

translationConfigDetails

object (TranslationConfigDetails)

Task configuration for CW Batch/Offline SQL Translation.

translationDetails

object (TranslationDetails)

Task details for unified SQL Translation.

AssessmentTaskDetails

Assessment task config.

JSON representation
{
  "inputPath": string,
  "outputDataset": string,
  "querylogsPath": string,
  "dataSource": string
}
Fields
inputPath

string

Required. The Cloud Storage path for assessment input files.

outputDataset

string

Required. The BigQuery dataset for output.

querylogsPath

string

Optional. An optional Cloud Storage path to write the query logs (which is then used as an input path on the translation task)

dataSource

string

Required. The data source or data warehouse type (eg: TERADATA/REDSHIFT) from which the input data is extracted.

TranslationTaskDetails

The translation task config to capture necessary settings for a translation task and subtask.

JSON representation
{
  "inputPath": string,
  "outputPath": string,
  "filePaths": [
    {
      object (TranslationFileMapping)
    }
  ],
  "schemaPath": string,
  "fileEncoding": enum (FileEncoding),
  "identifierSettings": {
    object (IdentifierSettings)
  },
  "specialTokenMap": {
    string: enum (TokenType),
    ...
  },
  "filter": {
    object (Filter)
  },
  "translationExceptionTable": string,

  // Union field language_options can be only one of the following:
  "teradataOptions": {
    object (TeradataOptions)
  },
  "bteqOptions": {
    object (BteqOptions)
  }
  // End of list of possible types for union field language_options.
}
Fields
inputPath

string

The Cloud Storage path for translation input files.

outputPath

string

The Cloud Storage path for translation output files.

filePaths[]

object (TranslationFileMapping)

Cloud Storage files to be processed for translation.

schemaPath

string

The Cloud Storage path to DDL files as table schema to assist semantic translation.

fileEncoding

enum (FileEncoding)

The file encoding type.

identifierSettings

object (IdentifierSettings)

The settings for SQL identifiers.

specialTokenMap

map (key: string, value: enum (TokenType))

The map capturing special tokens to be replaced during translation. The key is special token in string. The value is the token data type. This is used to translate SQL query template which contains special token as place holder. The special token makes a query invalid to parse. This map will be applied to annotate those special token with types to let parser understand how to parse them into proper structure with type information.

filter

object (Filter)

The filter applied to translation details.

translationExceptionTable

string

Specifies the exact name of the bigquery table ("dataset.table") to be used for surfacing raw translation errors. If the table does not exist, we will create it. If it already exists and the schema is the same, we will re-use. If the table exists and the schema is different, we will throw an error.

Union field language_options. The language specific settings for the translation task. language_options can be only one of the following:
teradataOptions

object (TeradataOptions)

The Teradata SQL specific settings for the translation task.

bteqOptions

object (BteqOptions)

The BTEQ specific settings for the translation task.

TeradataOptions

This type has no fields.

Teradata SQL specific translation task related settings.

BteqOptions

BTEQ translation task related settings.

JSON representation
{
  "projectDataset": {
    object (DatasetReference)
  },
  "defaultPathUri": string,
  "fileReplacementMap": {
    string: string,
    ...
  }
}
Fields
projectDataset

object (DatasetReference)

Specifies the project and dataset in BigQuery that will be used for external table creation during the translation.

defaultPathUri

string

The Cloud Storage location to be used as the default path for files that are not otherwise specified in the file replacement map.

fileReplacementMap

map (key: string, value: string)

Maps the local paths that are used in BTEQ scripts (the keys) to the paths in Cloud Storage that should be used in their stead in the translation (the value).

DatasetReference

Reference to a BigQuery dataset.

JSON representation
{
  "datasetId": string,
  "projectId": string,
  "datasetIdAlternative": [
    string
  ],
  "projectIdAlternative": [
    string
  ]
}
Fields
datasetId

string

A unique ID for this dataset, without the project name. The ID must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_). The maximum length is 1,024 characters.

TranslationFileMapping

Mapping between an input and output file to be translated in a subtask.

JSON representation
{
  "inputPath": string,
  "outputPath": string
}
Fields
inputPath

string

The Cloud Storage path for a file to translation in a subtask.

outputPath

string

The Cloud Storage path to write back the corresponding input file to.

FileEncoding

The file encoding types.

Enums
FILE_ENCODING_UNSPECIFIED File encoding setting is not specified.
UTF_8 File encoding is UTF_8.
ISO_8859_1 File encoding is ISO_8859_1.
US_ASCII File encoding is US_ASCII.
UTF_16 File encoding is UTF_16.
UTF_16LE File encoding is UTF_16LE.
UTF_16BE File encoding is UTF_16BE.

IdentifierSettings

Settings related to SQL identifiers.

JSON representation
{
  "outputIdentifierCase": enum (IdentifierCase),
  "identifierRewriteMode": enum (IdentifierRewriteMode)
}
Fields
outputIdentifierCase

enum (IdentifierCase)

The setting to control output queries' identifier case.

identifierRewriteMode

enum (IdentifierRewriteMode)

Specifies the rewrite mode for SQL identifiers.

IdentifierCase

The identifier case type.

Enums
IDENTIFIER_CASE_UNSPECIFIED The identifier case is not specified.
ORIGINAL Identifiers' cases will be kept as the original cases.
UPPER Identifiers will be in upper cases.
LOWER Identifiers will be in lower cases.

IdentifierRewriteMode

The SQL identifier rewrite mode.

Enums
IDENTIFIER_REWRITE_MODE_UNSPECIFIED SQL Identifier rewrite mode is unspecified.
NONE SQL identifiers won't be rewrite.
REWRITE_ALL All SQL identifiers will be rewrite.

TokenType

The special token data type.

Enums
TOKEN_TYPE_UNSPECIFIED Token type is not specified.
STRING Token type as string.
INT64 Token type as integer.
NUMERIC Token type as numeric.
BOOL Token type as boolean.
FLOAT64 Token type as float.
DATE Token type as date.
TIMESTAMP Token type as timestamp.

Filter

The filter applied to fields of translation details.

JSON representation
{
  "inputFileExclusionPrefixes": [
    string
  ]
}
Fields
inputFileExclusionPrefixes[]

string

The list of prefixes used to exclude processing for input files.

TranslationConfigDetails

The translation config to capture necessary settings for a translation task and subtask.

JSON representation
{
  "sourceDialect": {
    object (Dialect)
  },
  "targetDialect": {
    object (Dialect)
  },
  "sourceEnv": {
    object (SourceEnv)
  },
  "sourceTargetLocationMapping": [
    {
      object (SourceTargetLocationMapping)
    }
  ],
  "requestSource": string,

  // Union field source_location can be only one of the following:
  "gcsSourcePath": string
  // End of list of possible types for union field source_location.

  // Union field target_location can be only one of the following:
  "gcsTargetPath": string
  // End of list of possible types for union field target_location.

}
Fields
sourceDialect

object (Dialect)

The dialect of the input files.

targetDialect

object (Dialect)

The target dialect for the engine to translate the input to.

sourceEnv

object (SourceEnv)

The default source environment values for the translation.

sourceTargetLocationMapping[]

object (SourceTargetLocationMapping)

The mapping from source location paths to target location paths.

requestSource

string

The indicator to show translation request initiator.

Union field source_location. The chosen path where the source for input files will be found. source_location can be only one of the following:
gcsSourcePath

string

The Cloud Storage path for a directory of files to translate in a task.

Union field target_location. The chosen path where the destination for output files will be found. target_location can be only one of the following:
gcsTargetPath

string

The Cloud Storage path to write back the corresponding input files to.

Dialect

The possible dialect options for translation.

JSON representation
{

  // Union field dialect_value can be only one of the following:
  "bigqueryDialect": {
    object (BigQueryDialect)
  },
  "hiveqlDialect": {
    object (HiveQLDialect)
  },
  "redshiftDialect": {
    object (RedshiftDialect)
  },
  "teradataDialect": {
    object (TeradataDialect)
  },
  "oracleDialect": {
    object (OracleDialect)
  },
  "sparksqlDialect": {
    object (SparkSQLDialect)
  },
  "snowflakeDialect": {
    object (SnowflakeDialect)
  },
  "netezzaDialect": {
    object (NetezzaDialect)
  },
  "azureSynapseDialect": {
    object (AzureSynapseDialect)
  },
  "verticaDialect": {
    object (VerticaDialect)
  },
  "sqlServerDialect": {
    object (SQLServerDialect)
  },
  "postgresqlDialect": {
    object (PostgresqlDialect)
  },
  "prestoDialect": {
    object (PrestoDialect)
  },
  "mysqlDialect": {
    object (MySQLDialect)
  },
  // End of list of possible types for union field dialect_value.
}
Fields
Union field dialect_value. The possible dialect options that this message represents. dialect_value can be only one of the following:
bigqueryDialect

object (BigQueryDialect)

The BigQuery dialect

hiveqlDialect

object (HiveQLDialect)

The HiveQL dialect

redshiftDialect

object (RedshiftDialect)

The Redshift dialect

teradataDialect

object (TeradataDialect)

The Teradata dialect

oracleDialect

object (OracleDialect)

The Oracle dialect

sparksqlDialect

object (SparkSQLDialect)

The SparkSQL dialect

snowflakeDialect

object (SnowflakeDialect)

The Snowflake dialect

netezzaDialect

object (NetezzaDialect)

The Netezza dialect

azureSynapseDialect

object (AzureSynapseDialect)

The Azure Synapse dialect

verticaDialect

object (VerticaDialect)

The Vertica dialect

sqlServerDialect

object (SQLServerDialect)

The SQL Server dialect

postgresqlDialect

object (PostgresqlDialect)

The Postgresql dialect

prestoDialect

object (PrestoDialect)

The Presto dialect

mysqlDialect

object (MySQLDialect)

The MySQL dialect

BigQueryDialect

This type has no fields.

The dialect definition for BigQuery.

HiveQLDialect

This type has no fields.

The dialect definition for HiveQL.

RedshiftDialect

This type has no fields.

The dialect definition for Redshift.

TeradataDialect

The dialect definition for Teradata.

JSON representation
{
  "mode": enum (Mode)
}
Fields
mode

enum (Mode)

Which Teradata sub-dialect mode the user specifies.

Mode

The sub-dialect options for Teradata.

Enums
MODE_UNSPECIFIED Unspecified mode.
SQL Teradata SQL mode.
BTEQ BTEQ mode (which includes SQL).

OracleDialect

This type has no fields.

The dialect definition for Oracle.

SparkSQLDialect

This type has no fields.

The dialect definition for SparkSQL.

SnowflakeDialect

This type has no fields.

The dialect definition for Snowflake.

NetezzaDialect

This type has no fields.

The dialect definition for Netezza.

AzureSynapseDialect

This type has no fields.

The dialect definition for Azure Synapse.

VerticaDialect

This type has no fields.

The dialect definition for Vertica.

SQLServerDialect

This type has no fields.

The dialect definition for SQL Server.

PostgresqlDialect

This type has no fields.

The dialect definition for Postgresql.

PrestoDialect

This type has no fields.

The dialect definition for Presto.

MySQLDialect

This type has no fields.

The dialect definition for MySQL.

SourceTargetLocationMapping

Represents one mapping from a source location path to an optional target location path.

JSON representation
{
  "sourceLocation": {
    object (SourceLocation)
  },
  "targetLocation": {
    object (TargetLocation)
  }
}
Fields
sourceLocation

object (SourceLocation)

The path to the location of the source data.

targetLocation

object (TargetLocation)

The path to the location of the target data.

SourceLocation

Represents one path to the location that holds source data.

JSON representation
{

  // Union field location can be only one of the following:
  "gcsPath": string
  // End of list of possible types for union field location.
}
Fields
Union field location. The location of the source data. location can be only one of the following:
gcsPath

string

The Cloud Storage path for a directory of files.

TargetLocation

// Represents one path to the location that holds target data.

JSON representation
{

  // Union field location can be only one of the following:
  "gcsPath": string
  // End of list of possible types for union field location.
}
Fields
Union field location. The location of the target data. location can be only one of the following:
gcsPath

string

The Cloud Storage path for a directory of files.

TranslationDetails

The translation details to capture the necessary settings for a translation job.

JSON representation
{
  "sourceTargetMapping": [
    {
      object (SourceTargetMapping)
    }
  ],
  "targetBaseUri": string,
  "sourceEnvironment": {
    object (SourceEnvironment)
  },
  "targetReturnLiterals": [
    string
  ],
  "targetTypes": [
    string
  ]
}
Fields
sourceTargetMapping[]

object (SourceTargetMapping)

The mapping from source to target SQL.

targetBaseUri

string

The base URI for all writes to persistent storage.

sourceEnvironment

object (SourceEnvironment)

The default source environment values for the translation.

targetReturnLiterals[]

string

The list of literal targets that will be directly returned to the response. Each entry consists of the constructed path, EXCLUDING the base path. Not providing a targetBaseUri will prevent writing to persistent storage.

targetTypes[]

string

The types of output to generate, e.g. sql, sqlx, lineage, analysis, etc If not specified, a default set of targets will be generated. Some additional target types may be slower to generate. See the documentation for the set of available target types.

SourceTargetMapping

Represents one mapping from a source SQL to a target SQL.

JSON representation
{
  "sourceSpec": {
    object (SourceSpec)
  },
  "targetSpec": {
    object (TargetSpec)
  }
}
Fields
sourceSpec

object (SourceSpec)

The source SQL or the path to it.

targetSpec

object (TargetSpec)

The target SQL or the path for it.

SourceSpec

Represents one path to the location that holds source data.

JSON representation
{
  "encoding": string,

  // Union field source can be only one of the following:
  "baseUri": string,
  "literal": {
    object (Literal)
  }
  // End of list of possible types for union field source.
}
Fields
encoding

string

Optional. The optional field to specify the encoding of the sql bytes.

Union field source. The specific source SQL. source can be only one of the following:
baseUri

string

The base URI for all files to be read in as sources for translation.

literal

object (Literal)

Source literal.

Literal

Literal data.

JSON representation
{
  "relativePath": string,

  // Union field literal_data can be only one of the following:
  "literalString": string,
  "literalBytes": string
  // End of list of possible types for union field literal_data.
}
Fields
relativePath

string

Required. The identifier of the literal entry.

Union field literal_data. The literal SQL contents. literal_data can be only one of the following:
literalString

string

Literal string data.

literalBytes

string (bytes format)

Literal byte data.

TargetSpec

Represents one path to the location that holds target data.

JSON representation
{
  "relativePath": string
}
Fields
relativePath

string

The relative path for the target data. Given source file baseUri/input/sql, the output would be targetBaseUri/sql/relativePath/input.sql.

SourceEnvironment

Represents the default source environment values for the translation.

JSON representation
{
  "defaultDatabase": string,
  "schemaSearchPath": [
    string
  ],
  "metadataStoreDataset": string
}
Fields
defaultDatabase

string

The default database name to fully qualify SQL objects when their database name is missing.

schemaSearchPath[]

string

The schema search path. When SQL objects are missing schema name, translation engine will search through this list to find the value.

metadataStoreDataset

string

Optional. Expects a validQ BigQuery dataset ID that exists, e.g., project-123.metadata_store_123. If specified, translation will search and read the required schema information from a metadata store in this dataset. If metadata store doesn't exist, translation will parse the metadata file and upload the schema info to a temp table in the dataset to speed up future translation jobs.

State

Possible states of a migration task.

Enums
STATE_UNSPECIFIED The state is unspecified.
PENDING The task is waiting for orchestration.
ORCHESTRATING The task is assigned to an orchestrator.
RUNNING The task is running, i.e. its subtasks are ready for execution.
PAUSED Tha task is paused. Assigned subtasks can continue, but no new subtasks will be scheduled.
SUCCEEDED The task finished successfully.
FAILED The task finished unsuccessfully.

MigrationTaskOrchestrationResult

Additional information from the orchestrator when it is done with the task orchestration.

JSON representation
{

  // Union field details can be only one of the following:
  "assessmentDetails": {
    object (AssessmentOrchestrationResultDetails)
  },
  "translationTaskResult": {
    object (TranslationTaskResult)
  }
  // End of list of possible types for union field details.
}
Fields
Union field details. Details specific to the task type. details can be only one of the following:
assessmentDetails

object (AssessmentOrchestrationResultDetails)

Details specific to assessment task types.

translationTaskResult

object (TranslationTaskResult)

Details specific to translation task types.

AssessmentOrchestrationResultDetails

Details for an assessment task orchestration result.

JSON representation
{
  "outputTablesSchemaVersion": string,
  "reportUri": string
}
Fields
outputTablesSchemaVersion

string

Optional. The version used for the output table schemas.

reportUri

string

Optional. The URI of the Data Studio report.

TranslationTaskResult

Translation specific result details from the migration task.

JSON representation
{
  "translatedLiterals": [
    {
      object (Literal)
    }
  ],
  "reportLogMessages": [
    {
      object (GcsReportLogMessage)
    }
  ]
}
Fields
translatedLiterals[]

object (Literal)

The list of the translated literals.

reportLogMessages[]

object (GcsReportLogMessage)

The records from the aggregate CSV report for a migration workflow.

GcsReportLogMessage

A record in the aggregate CSV report for a migration workflow

JSON representation
{
  "severity": string,
  "category": string,
  "filePath": string,
  "filename": string,
  "sourceScriptLine": integer,
  "sourceScriptColumn": integer,
  "message": string,
  "scriptContext": string,
  "action": string,
  "effect": string,
  "objectName": string
}
Fields
severity

string

Severity of the translation record.

category

string

Category of the error/warning. Example: SyntaxError

filePath

string

The file path in which the error occurred

filename

string

The file name in which the error occurred

sourceScriptLine

integer

Specifies the row from the source text where the error occurred (0 based, -1 for messages without line location). Example: 2

sourceScriptColumn

integer

Specifies the column from the source texts where the error occurred. (0 based, -1 for messages without column location) example: 6

message

string

Detailed message of the record.

scriptContext

string

The script context (obfuscated) in which the error occurred

action

string

Category of the error/warning. Example: SyntaxError

effect

string

Category of the error/warning. Example: SyntaxError

objectName

string

Name of the affected object in the log message.

State

Possible migration workflow states.

Enums
STATE_UNSPECIFIED Workflow state is unspecified.
DRAFT Workflow is in draft status, i.e. tasks are not yet eligible for execution.
RUNNING Workflow is running (i.e. tasks are eligible for execution).
PAUSED Workflow is paused. Tasks currently in progress may continue, but no further tasks will be scheduled.
COMPLETED Workflow is complete. There should not be any task in a non-terminal state, but if they are (e.g. forced termination), they will not be scheduled.

Methods

create

Creates a migration workflow.

delete

Deletes a migration workflow by name.

get

Gets a previously created migration workflow.

list

Lists previously created migration workflow.

start

Starts a previously created migration workflow.