Key concepts
This page describes the key concepts for understanding the components of Manufacturing Data Engine (MDE).
Manufacturing data
MDE information architecture is designed to process two types of manufacturing data:
- Manufacturing facts: Sensor readings or events acquired from the factory floor or derived from factory floor data.
- Context about facts (metadata): Origin, the way they were captured, their type, nature (the measurements they represent), assets that refer to, and provide context about them.
MDE is designed to be used as a data repository of factual and contextual manufacturing data and to capture relationships between them. The main design objective is MDE is to provide rapid data ingestion, performant data access, and enable value-added use cases.
Manufacturing data and metadata are stored in and sent to different data sinks that are optimized for specific use-cases.
To provide the flexibility in organizing and storing data, MDE defines a model that allows configurability and extensibility of logical schemas while maintaining physical data schema stability.
This diagram shows at a high level the message flow as processed by MDE:
Source message
Any message published to the input-messages Pub/Sub
topic, which is the data entry point to
MDE.
Source message class
A group of Source messages that share a common structure or semantics. For more information, see Source Message Class.
Parser
A configuration entity that transforms messages of a specific Source message class into records of a specific Type by means of a Whistle script. A parser can produce records with differing Tags, but all records must share the same Type. For more information, see Parser.
Archetype
A base for a Type that defines specific fields and behaviors. There are six archetypes grouped in three archetype families:
- Numeric (standard and clustered)
- Discrete (standard and clustered)
- Continuous (standard and clustered)
Natural key
A key uniquely identifying an entity outside of MDE.
Tag
An entity, such as a sensor, device, or machine emitting a stream of Source messages. The entity is identified by a Tag name which also serves as the natural key in tag cloud tag metadata buckets. Configuration settings can be attached to tags in the context of Type versions. For more information, see Tag.
Type
A configuration entity describing a common set of Records. A Type is associated to exactly one Archetype. A Type may be associated with one or more Metadata buckets, it may hold a Data field schema, and provide Storage specifications. For more information, see Type.
Data field
A field in a record. The object written to the field can be constrained by schema on Types of Discrete and Continuous Archetype.
Metadata bucket
A configuration entity that models a related set of contextual information about a manufacturing fact (such as a sensor reading) associated with a Source message. Metadata buckets are modeled as dimensions in the analytics domain. Buckets contain schemas that define the structure of Metadata instances contained within them. Changes to Metadata instances within a bucket are versioned. For more information, see Metadata buckets.
Metadata instance
An entry within a Metadata bucket that describes an entity with a given Natural key. For more information, see Metadata instances.
Proto metadata instance
A Metadata instance that is constructed in the parser.
Record table
A table in BigQuery tied to a Type. Each Type is associated to one Record table.
Record
A single processed record. Also a row in a Record table. All records share the same Type.
Proto record
A precursor record that is constructed and emitted by a Parser.
Embedded Metadata
A field that stores rapidly changing metadata in the Record. For more information, see Embedded Metadata.
Cloud metadata reference
A field in a Record that stores metadata instance identifiers per bucket (foreign keys).
Materialized metadata
A field in a Record that stores materialized metadata instances listed in Cloud metadata reference.