REST Resource: projects.locations.workflows

Resource: MigrationWorkflow

A migration workflow which specifies what needs to be done for an EDW migration.

JSON representation 
{
  "name": string,
  "displayName": string,
  "tasks": {
    string: {
      object (MigrationTask)
    },
    ...
  },
  "state": enum (State),
  "createTime": string,
  "lastUpdateTime": string
}
Fields
name

string

Output only. Immutable. Identifier. The unique identifier for the migration workflow. The ID is server-generated.

Example: projects/123/locations/us/workflows/345
displayName

string

The display name of the workflow. This can be set to give a workflow a descriptive name. There is no guarantee or enforcement of uniqueness.
tasks

map (key: string, value: object (MigrationTask))

The tasks in a workflow in a named map. The name (i.e. key) has no meaning and is merely a convenient way to address a specific task in a workflow.
state

enum (State)

Output only. That status of the workflow.
createTime

string (Timestamp format)

Time when the workflow was created.
lastUpdateTime

string (Timestamp format)

Time when the workflow was last updated.

MigrationTask

A single task for a migration which has details about the configuration of the task.

JSON representation 
{
  "id": string,
  "type": string,
  "state": enum (State),
  "processingError": {
    object (ErrorInfo)
  },
  "createTime": string,
  "lastUpdateTime": string,
  "resourceErrorDetails": [
    {
      object (ResourceErrorDetail)
    }
  ],
  "resourceErrorCount": integer,
  "metrics": [
    {
      object (TimeSeries)
    }
  ],
  "taskResult": {
    object (MigrationTaskResult)
  },
  "totalProcessingErrorCount": integer,
  "totalResourceErrorCount": integer,

  // Union field task_details can be only one of the following:
  "translationConfigDetails": {
    object (TranslationConfigDetails)
  },
  "translationDetails": {
    object (TranslationDetails)
  }
  // End of list of possible types for union field task_details.
}
Fields
id

string

Output only. Immutable. The unique identifier for the migration task. The ID is server-generated.
type

string

The type of the task. This must be one of the supported task types: Translation_Teradata2BQ, Translation_Redshift2BQ, Translation_Bteq2BQ, Translation_Oracle2BQ, Translation_HiveQL2BQ, Translation_SparkSQL2BQ, Translation_Snowflake2BQ, Translation_Netezza2BQ, Translation_AzureSynapse2BQ, Translation_Vertica2BQ, Translation_SQLServer2BQ, Translation_Presto2BQ, Translation_MySQL2BQ, Translation_Postgresql2BQ, Translation_SQLite2BQ, Translation_Greenplum2BQ.
state

enum (State)

Output only. The current state of the task.
processingError

object (ErrorInfo)

Output only. An explanation that may be populated when the task is in FAILED state.
createTime

string (Timestamp format)

Time when the task was created.
lastUpdateTime

string (Timestamp format)

Time when the task was last updated.
resourceErrorDetails[]

object (ResourceErrorDetail)

Output only. Provides details to errors and issues encountered while processing the task. Presence of error details does not mean that the task failed.
resourceErrorCount

integer

The number or resources with errors. Note: This is not the total number of errors as each resource can have more than one error. This is used to indicate truncation by having a resourceErrorCount that is higher than the size of resourceErrorDetails.
metrics[]

object (TimeSeries)

The metrics for the task.
taskResult

object (MigrationTaskResult)

Output only. The result of the task.
totalProcessingErrorCount

integer

Count of all the processing errors in this task and its subtasks.
totalResourceErrorCount

integer

Count of all the resource errors in this task and its subtasks.
Union field task_details. The details of the task. task_details can be only one of the following:
translationConfigDetails

object (TranslationConfigDetails)

Task configuration for CW Batch/Offline SQL Translation.
translationDetails

object (TranslationDetails)

Task details for unified SQL Translation.

TranslationConfigDetails

The translation config to capture necessary settings for a translation task and subtask.

JSON representation 
{
  "sourceDialect": {
    object (Dialect)
  },
  "targetDialect": {
    object (Dialect)
  },
  "sourceEnv": {
    object (SourceEnv)
  },
  "requestSource": string,
  "targetTypes": [
    string
  ],

  // Union field source_location can be only one of the following:
  "gcsSourcePath": string
  // End of list of possible types for union field source_location.

  // Union field target_location can be only one of the following:
  "gcsTargetPath": string
  // End of list of possible types for union field target_location.

  // Union field output_name_mapping can be only one of the following:
  "nameMappingList": {
    object (ObjectNameMappingList)
  }
  // End of list of possible types for union field output_name_mapping.
}
Fields
sourceDialect

object (Dialect)

The dialect of the input files.
targetDialect

object (Dialect)

The target dialect for the engine to translate the input to.
sourceEnv

object (SourceEnv)

The default source environment values for the translation.
requestSource

string

The indicator to show translation request initiator.
targetTypes[]

string

The types of output to generate, e.g. sql, metadata etc. If not specified, a default set of targets will be generated. Some additional target types may be slower to generate. See the documentation for the set of available target types.
Union field source_location. The chosen path where the source for input files will be found. source_location can be only one of the following:
gcsSourcePath

string

The Cloud Storage path for a directory of files to translate in a task.
Union field target_location. The chosen path where the destination for output files will be found. target_location can be only one of the following:
gcsTargetPath

string

The Cloud Storage path to write back the corresponding input files to.
Union field output_name_mapping. The mapping of full SQL object names from their current state to the desired output. output_name_mapping can be only one of the following:
nameMappingList

object (ObjectNameMappingList)

The mapping of objects to their desired output names in list form.

ObjectNameMappingList

Represents a map of name mappings using a list of key:value proto messages of existing name to desired output name.

JSON representation 
{
  "nameMap": [
    {
      object (ObjectNameMapping)
    }
  ]
}
Fields
nameMap[]

object (ObjectNameMapping)

The elements of the object name map.

ObjectNameMapping

Represents a key-value pair of NameMappingKey to NameMappingValue to represent the mapping of SQL names from the input value to desired output.

JSON representation 
{
  "source": {
    object (NameMappingKey)
  },
  "target": {
    object (NameMappingValue)
  }
}
Fields
source

object (NameMappingKey)

The name of the object in source that is being mapped.
target

object (NameMappingValue)

The desired target name of the object that is being mapped.

NameMappingKey

The potential components of a full name mapping that will be mapped during translation in the source data warehouse.

JSON representation 
{
  "type": enum (Type),
  "database": string,
  "schema": string,
  "relation": string,
  "attribute": string
}
Fields
type

enum (Type)

The type of object that is being mapped.
database

string

The database name (BigQuery project ID equivalent in the source data warehouse).
schema

string

The schema name (BigQuery dataset equivalent in the source data warehouse).
relation

string

The relation name (BigQuery table or view equivalent in the source data warehouse).
attribute

string

The attribute name (BigQuery column equivalent in the source data warehouse).

Type

The type of the object that is being mapped.

Enums
TYPE_UNSPECIFIED Unspecified name mapping type.
DATABASE The object being mapped is a database.
SCHEMA The object being mapped is a schema.
RELATION The object being mapped is a relation.
ATTRIBUTE The object being mapped is an attribute.
RELATION_ALIAS The object being mapped is a relation alias.
ATTRIBUTE_ALIAS The object being mapped is a an attribute alias.
FUNCTION The object being mapped is a function.

NameMappingValue

The potential components of a full name mapping that will be mapped during translation in the target data warehouse.

JSON representation 
{
  "database": string,
  "schema": string,
  "relation": string,
  "attribute": string
}
Fields
database

string

The database name (BigQuery project ID equivalent in the target data warehouse).
schema

string

The schema name (BigQuery dataset equivalent in the target data warehouse).
relation

string

The relation name (BigQuery table or view equivalent in the target data warehouse).
attribute

string

The attribute name (BigQuery column equivalent in the target data warehouse).

Dialect

The possible dialect options for translation.

JSON representation 
{

  // Union field dialect_value can be only one of the following:
  "bigqueryDialect": {
    object (BigQueryDialect)
  },
  "hiveqlDialect": {
    object (HiveQLDialect)
  },
  "redshiftDialect": {
    object (RedshiftDialect)
  },
  "teradataDialect": {
    object (TeradataDialect)
  },
  "oracleDialect": {
    object (OracleDialect)
  },
  "sparksqlDialect": {
    object (SparkSQLDialect)
  },
  "snowflakeDialect": {
    object (SnowflakeDialect)
  },
  "netezzaDialect": {
    object (NetezzaDialect)
  },
  "azureSynapseDialect": {
    object (AzureSynapseDialect)
  },
  "verticaDialect": {
    object (VerticaDialect)
  },
  "sqlServerDialect": {
    object (SQLServerDialect)
  },
  "postgresqlDialect": {
    object (PostgresqlDialect)
  },
  "prestoDialect": {
    object (PrestoDialect)
  },
  "mysqlDialect": {
    object (MySQLDialect)
  },
  "db2Dialect": {
    object (DB2Dialect)
  },
  "sqliteDialect": {
    object (SQLiteDialect)
  },
  "greenplumDialect": {
    object (GreenplumDialect)
  }
  // End of list of possible types for union field dialect_value.
}
Fields
Union field dialect_value. The possible dialect options that this message represents. dialect_value can be only one of the following:
bigqueryDialect

object (BigQueryDialect)

The BigQuery dialect
hiveqlDialect

object (HiveQLDialect)

The HiveQL dialect
redshiftDialect

object (RedshiftDialect)

The Redshift dialect
teradataDialect

object (TeradataDialect)

The Teradata dialect
oracleDialect

object (OracleDialect)

The Oracle dialect
sparksqlDialect

object (SparkSQLDialect)

The SparkSQL dialect
snowflakeDialect

object (SnowflakeDialect)

The Snowflake dialect
netezzaDialect

object (NetezzaDialect)

The Netezza dialect
azureSynapseDialect

object (AzureSynapseDialect)

The Azure Synapse dialect
verticaDialect

object (VerticaDialect)

The Vertica dialect
sqlServerDialect

object (SQLServerDialect)

The SQL Server dialect
postgresqlDialect

object (PostgresqlDialect)

The Postgresql dialect
prestoDialect

object (PrestoDialect)

The Presto dialect
mysqlDialect

object (MySQLDialect)

The MySQL dialect
db2Dialect

object (DB2Dialect)

DB2 dialect
sqliteDialect

object (SQLiteDialect)

SQLite dialect
greenplumDialect

object (GreenplumDialect)

Greenplum dialect

BigQueryDialect

This type has no fields.

The dialect definition for BigQuery.

HiveQLDialect

This type has no fields.

The dialect definition for HiveQL.

RedshiftDialect

This type has no fields.

The dialect definition for Redshift.

TeradataDialect

The dialect definition for Teradata.

JSON representation 
{
  "mode": enum (Mode)
}
Fields
mode

enum (Mode)

Which Teradata sub-dialect mode the user specifies.

Mode

The sub-dialect options for Teradata.

Enums
MODE_UNSPECIFIED Unspecified mode.
SQL Teradata SQL mode.
BTEQ BTEQ mode (which includes SQL).

OracleDialect

This type has no fields.

The dialect definition for Oracle.

SparkSQLDialect

This type has no fields.

The dialect definition for SparkSQL.

SnowflakeDialect

This type has no fields.

The dialect definition for Snowflake.

NetezzaDialect

This type has no fields.

The dialect definition for Netezza.

AzureSynapseDialect

This type has no fields.

The dialect definition for Azure Synapse.

VerticaDialect

This type has no fields.

The dialect definition for Vertica.

SQLServerDialect

This type has no fields.

The dialect definition for SQL Server.

PostgresqlDialect

This type has no fields.

The dialect definition for Postgresql.

PrestoDialect

This type has no fields.

The dialect definition for Presto.

MySQLDialect

This type has no fields.

The dialect definition for MySQL.

DB2Dialect

This type has no fields.

The dialect definition for DB2.

SQLiteDialect

This type has no fields.

The dialect definition for SQLite.

GreenplumDialect

This type has no fields.

The dialect definition for Greenplum.

SourceEnv

Represents the default source environment values for the translation.

JSON representation 
{
  "defaultDatabase": string,
  "schemaSearchPath": [
    string
  ],
  "metadataStoreDataset": string
}
Fields
defaultDatabase

string

The default database name to fully qualify SQL objects when their database name is missing.
schemaSearchPath[]

string

The schema search path. When SQL objects are missing schema name, translation engine will search through this list to find the value.
metadataStoreDataset

string

Optional. Expects a valid BigQuery dataset ID that exists, e.g., project-123.metadata_store_123. If specified, translation will search and read the required schema information from a metadata store in this dataset. If metadata store doesn't exist, translation will parse the metadata file and upload the schema info to a temp table in the dataset to speed up future translation jobs.

TranslationDetails

The translation details to capture the necessary settings for a translation job.

JSON representation 
{
  "sourceTargetMapping": [
    {
      object (SourceTargetMapping)
    }
  ],
  "targetBaseUri": string,
  "sourceEnvironment": {
    object (SourceEnvironment)
  },
  "targetReturnLiterals": [
    string
  ],
  "targetTypes": [
    string
  ]
}
Fields
sourceTargetMapping[]

object (SourceTargetMapping)

The mapping from source to target SQL.
targetBaseUri

string

The base URI for all writes to persistent storage.
sourceEnvironment

object (SourceEnvironment)

The default source environment values for the translation.
targetReturnLiterals[]

string

The list of literal targets that will be directly returned to the response. Each entry consists of the constructed path, EXCLUDING the base path. Not providing a targetBaseUri will prevent writing to persistent storage.
targetTypes[]

string

The types of output to generate, e.g. sql, metadata, lineage_from_sql_scripts, etc. If not specified, a default set of targets will be generated. Some additional target types may be slower to generate. See the documentation for the set of available target types.

SourceTargetMapping

Represents one mapping from a source SQL to a target SQL.

JSON representation 
{
  "sourceSpec": {
    object (SourceSpec)
  },
  "targetSpec": {
    object (TargetSpec)
  }
}
Fields
sourceSpec

object (SourceSpec)

The source SQL or the path to it.
targetSpec

object (TargetSpec)

The target SQL or the path for it.

SourceSpec

Represents one path to the location that holds source data.

JSON representation 
{
  "encoding": string,

  // Union field source can be only one of the following:
  "baseUri": string,
  "literal": {
    object (Literal)
  }
  // End of list of possible types for union field source.
}
Fields
encoding

string

Optional. The optional field to specify the encoding of the sql bytes.
Union field source. The specific source SQL. source can be only one of the following:
baseUri

string

The base URI for all files to be read in as sources for translation.
literal

object (Literal)

Source literal.

Literal

Literal data.

JSON representation 
{
  "relativePath": string,

  // Union field literal_data can be only one of the following:
  "literalString": string,
  "literalBytes": string
  // End of list of possible types for union field literal_data.
}
Fields
relativePath

string

Required. The identifier of the literal entry.
Union field literal_data. The literal SQL contents. literal_data can be only one of the following:
literalString

string

Literal string data.
literalBytes

string (bytes format)

Literal byte data.

TargetSpec

Represents one path to the location that holds target data.

JSON representation 
{
  "relativePath": string
}
Fields
relativePath

string

The relative path for the target data. Given source file baseUri/input/sql, the output would be targetBaseUri/sql/relativePath/input.sql.

SourceEnvironment

Represents the default source environment values for the translation.

JSON representation 
{
  "defaultDatabase": string,
  "schemaSearchPath": [
    string
  ],
  "metadataStoreDataset": string
}
Fields
defaultDatabase

string

The default database name to fully qualify SQL objects when their database name is missing.
schemaSearchPath[]

string

The schema search path. When SQL objects are missing schema name, translation engine will search through this list to find the value.
metadataStoreDataset

string

Optional. Expects a validQ BigQuery dataset ID that exists, e.g., project-123.metadata_store_123. If specified, translation will search and read the required schema information from a metadata store in this dataset. If metadata store doesn't exist, translation will parse the metadata file and upload the schema info to a temp table in the dataset to speed up future translation jobs.

State

Possible states of a migration task.

Enums
STATE_UNSPECIFIED The state is unspecified.
PENDING The task is waiting for orchestration.
ORCHESTRATING The task is assigned to an orchestrator.
RUNNING The task is running, i.e. its subtasks are ready for execution.
PAUSED Tha task is paused. Assigned subtasks can continue, but no new subtasks will be scheduled.
SUCCEEDED The task finished successfully.
FAILED The task finished unsuccessfully.

MigrationTaskResult

The migration task result.

JSON representation 
{

  // Union field details can be only one of the following:
  "translationTaskResult": {
    object (TranslationTaskResult)
  }
  // End of list of possible types for union field details.
}
Fields
Union field details. Details specific to the task type. details can be only one of the following:
translationTaskResult

object (TranslationTaskResult)

Details specific to translation task types.

TranslationTaskResult

Translation specific result details from the migration task.

JSON representation 
{
  "translatedLiterals": [
    {
      object (Literal)
    }
  ],
  "reportLogMessages": [
    {
      object (GcsReportLogMessage)
    }
  ]
}
Fields
translatedLiterals[]

object (Literal)

The list of the translated literals.
reportLogMessages[]

object (GcsReportLogMessage)

The records from the aggregate CSV report for a migration workflow.

GcsReportLogMessage

A record in the aggregate CSV report for a migration workflow

JSON representation 
{
  "severity": string,
  "category": string,
  "filePath": string,
  "filename": string,
  "sourceScriptLine": integer,
  "sourceScriptColumn": integer,
  "message": string,
  "scriptContext": string,
  "action": string,
  "effect": string,
  "objectName": string
}
Fields
severity

string

Severity of the translation record.
category

string

Category of the error/warning. Example: SyntaxError
filePath

string

The file path in which the error occurred
filename

string

The file name in which the error occurred
sourceScriptLine

integer

Specifies the row from the source text where the error occurred (0 based, -1 for messages without line location). Example: 2
sourceScriptColumn

integer

Specifies the column from the source texts where the error occurred. (0 based, -1 for messages without column location) example: 6
message

string

Detailed message of the record.
scriptContext

string

The script context (obfuscated) in which the error occurred
action

string

Category of the error/warning. Example: SyntaxError
effect

string

Effect of the error/warning. Example: COMPATIBILITY
objectName

string

Name of the affected object in the log message.

State

Possible migration workflow states.

Enums
STATE_UNSPECIFIED Workflow state is unspecified.
DRAFT Workflow is in draft status, i.e. tasks are not yet eligible for execution.
RUNNING Workflow is running (i.e. tasks are eligible for execution).
PAUSED Workflow is paused. Tasks currently in progress may continue, but no further tasks will be scheduled.
COMPLETED Workflow is complete. There should not be any task in a non-terminal state, but if they are (e.g. forced termination), they will not be scheduled.

Methods

create

 Creates a migration workflow.

delete

 Deletes a migration workflow by name.

get

 Gets a previously created migration workflow.

list

 Lists previously created migration workflow.

start

 Starts a previously created migration workflow.