Data Flows, Pipelines, and Workflows
Overview
Data Flows define the vehicles of data movement and orchestration in Empower's 3rd generation metadata-driven stack. Flows replace the legacy Load Group and Main Pipeline system, which relied heavily upon Azure Data Factory for orchestration and execution.
As we move away from Load Groups, orchestration will now be situated firmly within the Empower Celery system, which defines the scheduling of various tasks like Flows and Workflows. While ADF is still used to execute some of those tasks, we will be moving away from this into a more Spark-executor paradigm over Q1 and Q2 FY24.
However, we don't want to STOP here. Flows are a stepping stone to the ultimate ELT end-goal: granular configuration of Lakehouse tables. On a technical level, we can consider the smallest unit of work that Empower can execute to be an Atom.
- For Acquisition, this is would be "extract one object" or "ingest one object".
- For Analytics Engineering, this would be "execute one notebook," or "build one entity," or "merge the model (merge all entities at once)".
- For Publishing, this would be "publish one entity".
Imagine the following setup, where Objects 1, 2, and 3 are acquired, then used to build a model with Entities 1 and 2, before Entity 2 is published to some target. The graph below defines how this would work with DAQ, AE, and PBL flows as a serialized/batched method. While we could still utilize shared compute to make this optimal, this is not the most optimal way to achieve this workflow.
A more granular view (from the users perspective) would give users the ability to specify what Entity 1 and Entity 2 both need to complete build, by breaking them out of the internal orchestration engine that model builder is currently today.
View the graph below to see "real" times of acquisition and buildout. Whereas in Empower v3, we would be waiting for Object 3 to finish acquiring before building Entity 1, in Empower v4, we could start Entity 1's build before Object 3 is finished being acquired.
Another benefit to Empower v4's Atoms is its ability to perform a more efficient "waterfall" execution of a Workflow. Imagine if i wanted this workflow itself to stream. I could, essentially, trigger the acquisition of an object to immediately queue itself up again as soon as it completes
Lexicon
Below, a common set of technical terms and definitions for the team:
| Term | Definition |
|---|---|
| Data Flow | A vehicle for data movement. Comes in several "types" where each type defines a destination of data. |
| Acquisition Flow | An inbound data flow, from a source into the Lakehouse. |
| Analytics Engineering Flow | An intrabound data flow, from the Lakehouse, back to the Lakehouse. Data may be transformed using notebooks to build gold layer tables and/or built into a data model (also in the gold layer). |
| Publishing Flow | An outbound data flow, where tables in any layer (bronze, silver, or gold) may be pushed to a supported set of data sources. |
| Workflow | A chained execution of jobs designed to fulfill a larger user goal; for example, acquiring tables from multiple sources, engineering them into a suitable data model, then refreshing a Power BI report. |
| Atom | The smallest technical unit of work as can be defined within Empower. • Acquisition: a single extract/ingest of an object • Analytics Engineering: a single entity buildout, a single notebook execution • Publishing: a single entity publishing |
| Waterfall | Triggering a flow or atom to immediately begin execution again the moment it completes its prior execution. For v3 Workflows, immediately beginning execution a gain the moment it completes its first "phase". |