While often used together, data lineage and data provenance focus on different aspects of the data journey.

• Data lineage looks at the data's journey from a macro, historical, and strategic perspective. It focuses on the full path and transformation logic that led to the current state of a data asset. It's the whole map.
• Data provenance is more granular and specific, often focusing on the immediate source and ownership of a specific data point or record at a single point in time. It's often used to authenticate the origin of a piece of data.

In short, lineage shows the entire evolution of data over time and across systems, while provenance often focuses on the source and authenticity of a particular data element.

A good data lineage map can help you quickly answer the "who, what, when, where, and why" questions about any data asset. The essential components tracked include:

• Source: Where the data originated, such as a transactional database, a file, or an external system
• Transformation logic: The specific operations or business rules applied to the data; this might include SQL queries, Python scripts, or ETL (Extract, Transform, Load) job logic
• Path/flow: The sequence of systems, processes, and data stores the data moves through
• Time/version: When the data was processed and which version of the data or the transformation logic was used
• Destination/consumer: The final resting place of the data and who or what used it, such as a regulatory report or a machine

Data lineage can be tracked at different stages of the data development life cycle and at various levels of detail, depending on the need.

Design-time lineage captures the data flow as it's being designed and configured in development and testing environments. It's based on reading the blueprints of the data pipelines, such as the schemas, scripts, and ETL job configurations. It tells you what should happen to the data.

Run-time lineage captures the data flow as it actually happens in the production environment. It records the specific inputs and outputs of executed jobs and processes. It tells you what did happen to the data, including any unexpected behavior or errors. For data governance, run-time lineage is often considered more valuable as it reflects reality.

The level of detail captured is called granularity. Organizations choose a level of granularity based on their data governance needs and the technical complexity of their environment.

• Table-level: Tracks the flow of data between entire tables or datasets; it shows that 'Customer Table A' flowed into 'Sales Report Table B'
• Example: A system shows that the entire raw_transactions table was loaded into the daily_aggregations table
• Column-level: Tracks the flow of data from a source column to a target column, including the transformations applied; this is often necessary for compliance
• Example: It tracks that the customer_id column from the source database was renamed to user_key in the data warehouse and then used as part of a join to create the final_report
• Report-level: Tracks which reports, dashboards, or applications consume which tables and columns; this is critical for impact analysis and business user trust
• Example: A business analyst can trace a metric on the Executive Sales Dashboard back to the specific columns and tables used in its calculation
• End-to-end: Provides a complete view across all systems, from the initial source application (like a CRM) through all staging, cleaning, and transformation steps, to the final report or machine learning model
• Example: Tracking a single customer's journey from when they first signed up (captured in the web app database) all the way to their usage being summarized in the Churn Prediction Model output