Unified types

Overview

Unified types are the data types that appear in the Avro or JSON events. They are a Datastream-specific, unified representation of a data type across multiple data sources, such as an Oracle database or MySQL database.

The unified types are the superset of all type representations across all supported source types, which represent the original source type in a generic but lossless way.

The following tables list:

  • The unified types associated with Datastream
  • The mappings between the data types for an Oracle database or a MySQL database, and the Datastream unified types

Datastream unified types

Type name Info Avro definition JSON definition
BOOLEAN boolean boolean boolean
BYTES A sequence of unsigned bytes bytes string
DECIMAL (p,s) An arbitrary-precision signed decimal number A decimal logical type number
DOUBLE 64-bit floating point numbers double number
FLOAT 32-bit floating point numbers float number
INTEGER A 32-bit integer int number
LONG A 64-bit integer long number
NUMBER A numeric data type

A custom logical type

{
  "type": "string", 
  "logicalType": "number"
}
string
RECORD A fully structured data type (for example, a user-defined data type) Record object
STRING An unlimited string length string string
TIME_INTERVAL How many microseconds elapsed between two events

A custom logical type

{
  "type": "long", 
  "logicalType": "time-interval-micros"
}
long
TIMESTAMP How many milliseconds or microseconds elapsed since the epoch, regardless of timezone A timestamp logical type string [ISO-8601]
TIMESTAMP WITH TIME ZONE How many milliseconds or microseconds elapsed since the epoch with a specific timezone offset

A custom type

{
  "type": "record",
  "name": "timestampTz",
  "fields": [
    {"name": "timestamp",
     "type": "long"
     "logicalType": "timestamp-micros"},
    {"name": "offset", 
     "type": "long"
     "logicalType": "time-millis"}
  ]
}
    
string [ISO-8601]
UNION A varying data type Union array
UNSUPPORTED An unsupported data type

A custom logical type

{
  "type": "null", 
  "logicalType": "unsupported"
}
null
VARCHAR A string with a maximum length of n characters

A custom logical type

{
  "type": "string", 
  "logicalType": "varchar"
  "length": N
}
string

Map Oracle data types to Datastream unified types

Oracle data type Datastream unified type
ANYDATA Unsupported
BFILE STRING
BINARY DOUBLE DOUBLE
BINARY FLOAT FLOAT
BLOB Unsupported
CHAR VARCHAR
CLOB Unsupported
DATE TIMESTAMP
DOUBLE PRECISION DOUBLE
FLOAT(p) DOUBLE
INTERVAL DAY TO SECOND Unsupported
INTERVAL YEAR TO MONTH Unsupported
LONG/LONG RAW Unsupported
NCHAR STRING
NCLOB Unsupported
NUMBER NUMBER
NUMBER(p,s<=0)

If p<=18, then LONG. If p>18 or p=*, then NUMBER.

NUMBER(p,s>0)

If p=*, then DECIMAL(38,s), else DECIMAL(p,s).

NVARCHAR2 STRING
RAW BYTES
ROWID STRING
SMALLINT INTEGER
TIMESTAMP TIMESTAMP
TIMESTAMP WITH TIME ZONE TIMESTAMP WITH TIME ZONE
UDT (user-defined type) Unsupported
UROWID Unsupported
VARCHAR VARCHAR
VARCHAR2 VARCHAR
XMLTYPE Unsupported

Map MySQL data types to Datastream unified types

MySQL data type Datastream unified type
BIGINT(size) LONG
BINARY(size) STRING (hex encoded)
BIT(size) LONG
BLOB(size) STRING (hex encoded)
BOOL INTEGER
CHAR(size) STRING
DATE TIMESTAMP
DATETIME(fsp) TIMESTAMP
DECIMAL(size, d) DECIMAL(size, d)
DOUBLE(size, d) DOUBLE
ENUM(val1, val2, val3, ...) STRING
FLOAT(p) FLOAT
FLOAT(size, d) FLOAT
INTEGER(size) INTEGER
LONGBLOB STRING (hex encoded)
LONGTEXT STRING (hex encoded)
MEDIUMBLOB STRING (hex encoded)
MEDIUMINT(size) INTEGER
MEDIUMTEXT STRING
SET(val1, val2, val3, ...) STRING
SMALLINT(size) INTEGER
TEXT(size) STRING
TIME(fsp) TIME_INTERVAL
TIMESTAMP(fsp) TIMESTAMP
TINYBLOB STRING (hex encoded)
TINYINT(size) INTEGER
TINYTEXT STRING
VARBINARY(size) STRING (hex encoded)
VARCHAR STRING
YEAR INTEGER