Transformations

Manufacturing Data Engine (MDE) provides a straightforward way to do post-processing of messages during ingestion and generate new derived tags and records from the input messages. These post-processors are called Transformations. They can be associated to a given Type and generate new messages that are then re-ingested in MDE. Transformations are built-in Dataflow pipelines and can be configured using certain specific Transformation Parameters.

MDE 1.3 and later versions provide two default transformations:

  • Group Event Change
  • Windowing

Transformations are activated at the Type level. This implies that all Tags that belong to that Type will be transformed. A Type can be associated to none, one or more than one Transformations. The association between the Type and a specific Transformation can be set up in the UI and using the Configuration API. A Type can be associated to a Transformation at any given time and that association can also be removed whenever is no longer needed.

A transformation might need Transformation Parameters to define its specific behavior. The Transformation Parameters are specific to each Transformation type and can be different from one Transformation to another.

The specific Transformation Parameter Values for a given Transformation are set at Type level too. Different Types can use different parameter values and use the same Transformation differently.

Transformations receive all messages mapped to the Type as input where the transformation is active. Each message is processed by the Transformation processor in Dataflow.

The output of any Transformation is a raw JSON message that is ingested in the initial landing topic. MDE provides a unique Configuration Package with each Transformation. This Configuration Package needs to be applied to MDE if you are planning to use the Transformation. The Configuration Package creates a set of a Message Class, Parser, Type and Metadata Bucket collection. These configuration entities allow MDE to identify and process the Transformation output messages into a specific Type. Transformation-specific Types store the information emitted by the Transformation. They are also associated to specific Transformation Metadata Buckets storing reference information about the original transformed Tag or Record.

The two Types provided to process the raw messages generated by each Transformation are the following:

  • group-event-change-records: destination Type for the output messages of Group Event Change Transformation.
  • window-transformation-records: destination Type for the output messages of Windowing Transformation.

Group Event Change Transformation

This Transformation monitors any field in a Tag or a group of Tags and generates an event every time the monitored values changes in any of those Tags. This Transformation can be associated to a Type of any Archetype. The Transformation appears in the UI as eventChange.

This Transformation emits a message that is parsed to a specific CONTINUOUS_DATA_SERIES Type named group-event-change-records provided in its associated Configuration Package. The Type contains the start time and end time of the period between value changes as well as the previous value, the current value and the name of the Tag that triggered the change.

You can define a group of Tags of the same Type to be monitored together. You can chose a Key criteria to define how to group Tags. All Tags where the Key has the same value are monitored together. The value of the Key can be any element of the Tag Proto, including any Data field and any element of the Metadata.

To make the Transformation work for an individual Tag the Key is set to the TagName.

Parameters

The two Transformation Parameters required to configure this Transformation are the following:

  1. Key Expression: It is displayed in the UI as keyExpression. Defines the expression that will be used to group Tags. All Tags where Key Expression has the same value are monitored together. The Transformation emits a value change event every time a change is detected in any of those Tags. The values are compared to the previous values of the same Tag, but an event is triggered when any of the Tags changes. The Key Expression is defined as an SpEL expression and can use any of the available fields in the Tag proto. For example, to only monitor a single Tag based on the Tag name, the SpEL expression would be defined as:

    #message['tagName']
    

    Alternatively, to group Tags based on a common metadata field, like Machine Name (assuming an asset-hierarchy Bucket exists with an attribute called machine-name in it), the expression can be set to:

    #message['metadata']['asset-hierarchy']['machine-name']
    
  2. Value Expression: It is displayed in the UI as valueExpression. Defines the value field that the Transformation monitors to emit a new change event. It is also defined as an SpEL expression. The expression can use any field of the Tag proto structure. The output of the expression must be of type String to function correctly.

    For example, to monitor a value field in the Data section of the Tag proto, the Value Expression can be set to:

    #message['data']['numeric'].toString()
    

Output Message

The content of the Output Message generated by the Group Event Change Transformation provides the details of the detected event. The specific attributes available in the Output Message are:

  1. TagName: The Transformation creates an event under a Tagname which corresponds to the value of the Key Expression and the suffix "_event_change". If the Key Expression is defined as an individual TagName the derived Tag will be "TagName_event_change".
  2. Event Change Continuous Timestamp Object: contains the initial (event_timestamp_start attribute) and final (event_timestamp_end attribute) time stamps of the event. It also contains the duration (duration attribute) of the event as a double.
  3. Event Change Message Data Object: Contains the name of the Tags that changed value during the event (changed_tags attribute) and the list of Tags that match the grouping criteria at the time the event is emitted (contributing_tags attribute). Both can contain more than 1 Tag per message.
  4. Tag Value Change Object: The name (tag_name attribute), old value (old_expression_value attribute), new value (new_expression_attribute attribute) and message (message_id attribute) that triggered the state change event.
  5. Changed Value Message Object (on v1.3.6 and later): Contains the message that triggered the event change.

See the specific schema in the reference section.

Windowing Transformation

This Transformation applies a Windowing function to all Tags of a Type. Windowing refers to the summarization of multiple records over a period of time in a single record. The Windowing Transformation can be associated to any Type of any Archetype. It can summarize the records of Tags from 1 minute up to 60 minutes. The result of the Transformation is a raw message containing the summary values for the value expression during the selected period of time.

This Transformation can be applied multiple times to the same Type as far as the Id of each Windowing Transformation (as defined in its parameters) is unique within the Type. You can add the Windowing Transformation using the UI but you must use the API to add multiple Windowing Transformations to a given Type.

The Transformation generates a derived Tag of a provided Type (named window-transformation-records) containing a summary of all the records received during that time window for that Tag. This Transformation operates at the Tag level: it will generate a summarized Tag for each Tag of the Type to which the transformation has been associated. The name of the Tag is composed from the original Tagname and the suffix -window-stats and the id of the Windowing Transformation.

Parameters

This transformation requires three parameters to operate:

  1. Duration of the Window: available in the UI as durationMinutes. It refers to the duration of the Window to apply in minutes. For example, a value of 1, will generate a derived time series of 1 minute resolution.
  2. ID of the Window: available in the UI as id. Assigns a unique identifier to each Windowing Transformation of the Type in order to enable more than one to be applied in parallel. The ID can be any String.
  3. Value Expression: available in the UI as valueExpression. It is the SpEL expression that is applied to the Tag proto to generate a Numeric value that will be summarized by the Windowing Transformation. For example, if you require to use the value field of the a generic numeric Tag, the expression of valueExpression can be defined as:

    #message['data']['numeric']
    

Output message

The Transformation emits one message for each time period defined by the Transformation Parameter durationMinutes. The output message contains the following attributes:

  1. TagName: Name of the Tag of the derived time series. It is generated automatically by the Windowing Transformation. It is formed using the original TagName and the suffix -window-stats and the id of the Windowing Transformation.. For example, for a TagName called Temperature and an ID 1, the derived Tag will be named Temperature-window-stats-1.
  2. TimeStamp: Contains the start and end time stamps of the Window (available as event_timestamp_start and event_timestamp_end attributes).
  3. Window Summary Stats: Contains all the summarized data for the period which includes:
    1. Mean: The mean value of all values received during the time window.
    2. Min: The minimal value of all values received during the time window.
    3. Max: The maximum value of all values received during the time window.
    4. Standard Deviation: The standard deviation of all value received during the time window.
    5. p50: The 50% confidence level of not being exceeded for the values received during the time window.
    6. p90: The 90% confidence level of not being exceeded for the values received during the time window.
    7. Sum: The summary of all values received during the time window.
    8. Count: The number of values received during the time window.
    9. Delta: The difference between the last and the first value received during the time window.

For more information, see the specific schema in the Transformation Schemas section.