Skip to main content

Product Requirements Document

Mission​

We want to enable users to configure and trigger the main pipeline from the UI just as they can from direct StateDB manipulation. This is meant to act as an in-between feature before data flows.

Background​

The advanced settings page functionality in Empower v1.18 does not go far enough to entirely replace direct StateDB manipulation. To do this, users need to be able to not only configure models (current-state), but also setup object load groups, configure publishing flows, and trigger the main pipeline, both scheduled and on-demand.

Targeted Personas​

  • Dario Data Professional: who will be going in to set this up in the UI based on project requirements
  • Debbie Delivery Engineer: who will be doing the same ^

Current Behavior​

We have no way of doing this from the UI. The only thing that comes close is through direct StateDB manipulation and ADF scheduling.

Outcomes / Goals​

  • The goal here is to provide a self-service pipeline

User Stories​

Adding Objects to a Load Group​

  • As a data engineer, I want to add objects from the metadata catalog into a load group to be ingested during a pipeline run

Scheduling a Pipeline to run​

  • As a data professional, I want to view a load group and schedule it to run every weekday at 5pm.

Triggering a Pipeline to run now​

  • As a data analyst, I want to trigger a pipeline to run right now so that I can study the modeled data when it is complete.

Monitoring a Publishing Flow​

  • As a data engineer, I want to use the log tables to view the progress of a publishing flow and check it for failures.

Use cases​

Configuring Pipelines​

  1. Advanced Pipeline Configuration:
    1. Primary Actor: Delivery Engineer or Trained Client Engineer
    2. Description: As an expert user, I want to configure data movement without going through StateDB and ADF tooling. This means I will need to perform all configuration and triggering/scheduling of pipelines from the UI directly.
    3. Preconditions: User must be authorized and have knowledge of StateDB tables
    4. Postconditions: User’s pipeline configuration changes are saved in stateDB, and ADF pipeline is triggerable/scheduled to run. User can observe the configurations in relevant advanced settings tables.
    5. Flow:
      1. User turns on the advanced settings toggle.
      2. User is able to manipulate advanced settings tables:
        1. Modeling
        2. Publishing
      3. User is able to create new data sources from the acquisitions tab (as before).
      4. User is able to set load groups for metadata catalog entries.
      5. Outcome: User now has load groups assigned to metadata catalog entries. When the pipeline for this load group is triggered (and there is an extract/delta step pair in the load group step commands), these objects will be acquired during the pipeline run.

Running Pipelines​

  1. Pipeline Triggering:
    1. Primary Actor: Trained Client Engineer
    2. Description: Users should be able to manually trigger a pipeline load group at will. This feature will enable the user to remain out of Azure Data Factory and do any on demand triggering from the UI with a button press.
    3. Preconditions: User must be authorized and has a basic understanding of StateDB tables
    4. Postconditions: User’s pipeline trigger is saved in StateDB and begins running as soon as possible. Logs from the runs will be monitorable from the relevant StateDB log tables.
    5. Flow:
      1. User turns on the advanced settings toggle.
      2. User goes to the Load Groups tab.
      3. User can manually trigger a load group at will by pressing the RUN button for a given Load Group row.
      4. User is presented with a confirmation modal to confirm they want to run Load Group ###.
      5. User presses confirm.
      6. User can monitor the load group runs through the ObjectExtract, ObjectIngest, and ObjectModel, publish execution log tables in Advanced Settings UI.
      7. Outcome: User is enabled through increased pipeline activation and visibility. They can run pipelines on demand at any time.
  2. Pipeline Scheduling:
    1. Primary Actor: Trained Client Engineer
    2. Description: Users should be able to schedule pipeline runs from the UI. This feature will enable the user to remain out of Azure Data Factory and do any scheduling from the UI with a button press.
    3. Preconditions: User must be authorized and has a basic understanding of StateDB tables
    4. Postconditions: User’s pipeline schedule is saved in StateDB and begins running as soon as possible. Logs from the runs will be monitorable from the relevant StateDB log tables. The next scheduled run is visible to the user in the load group table.
    5. Flow:
      1. User turns on the advanced settings toggle.
      2. User goes to LoadGroups Tab.
      3. User can schedule a load group to run by pressing the Schedule Group button for a given Load Group.
      4. User is presented with a scheduling menu (just like in flows! Designs exist)
      5. User configures schedule and cadence, and presses SCHEDULE button.
      6. If this group was already scheduled, this will overwrite its schedule.
        1. The user will not get a warning notification for this action.
      7. If the group was not scheduled, this will schedule the group.
      8. User can monitor the load group runs through the ObjectExtract, ObjectIngest, and ObjectModel, and publish execution log tables in Advanced Settings UI.
      9. Outcome: User is enabled to schedule pipelines on a repeatable cadence. They can schedule pipelines far in advance or right away.

Supported Roles​

RolePermissions
Super AdminCRUD access to all Enterprises across all environments
Enterprise AdminCRUD access to all environments within an Enterprise
Enterprise WriterCRUD access to all environments within an Enterprise
Organization AdminCRUD access to all environments within an organization
Organization WriterCRUD access to all environments within an organization
Environment AdminCRUD access within the same environment
Environment WriterCRUD access within the same environment

Requirements​

Functional​

  • StateDB Table feature parity required for full configuration:
    • ObjectList
      • Ability to modify object list load group with the advanced settings toggled on -> this is a new column in the metadata catalog
    • ObjectIngest
      • Monitor progress (view only)
    • ObjectExtract
      • Monitor progress (view only)
    • ObjectModel
      • Monitor progress (view only)
    • Model - Full CRUD
    • Entity – Full CRUD
    • EntityColumn – Full CRUD
    • EntityDependency – Full CRUD
    • DB2Step – partial CRUD
    • flows.executionLog
      • monitor progress (view only)
  • Scheduling and OnDemand triggering of Main_PL
    • To trigger main_pl, ADF needs a load group as an input parameter
    • There needs to be a minimalist UI around scheduling runs.
    • There also needs to be a UI around showing when load group is configured to run. (Want to see the schedule table – this will be something that needs to be config promotable in the future)

Non-functional requirements​

  • Developer speed - minimum time to delivery
  • Monitoring progress is up to date every 30 seconds

Out of Scope​

  • Funtionality and cases out of scope for planned feature
  • Might be treated as future updates that not yet planned and refined

Dependencies & Constraints​

  1. Coordination for API and UI is critical here. We must communicate dependencies.

Assumptions​

  1. This has no deployment or SRE related dependencies
  2. There are a limited number of new API endpoints required
  3. Most of the UI work can be immediate, since many of the table manipulation endpoints for StateDB exist

Risks​

  • This wont get out by v1.20, which threatens our ability to provide full self service pipelines before the end of the calendar year

Timeline and Milestones​

Phase I​

  • Release 1.20
    • All functionalities

Questions?​