Product Requirements Document

Mission

Creating the functionality in our Publishing Flows to publish bronze, silver, and gold tables from Delta to Dynamics 365 Dataverse for seamless data integration and improved reporting capabilities.

Background

We want Empower users to be able to seamlessly move data into and out of my data platform so they can do things with their data such as:

• Build insightful reports
• Update existing reports with the latest data sets
• Generate shareable files and distribute to key stakeholders both internal and external to my company
  etc.

We want to augment the capabilities of the existing Dynamics connector in Data-Driven Upgrade by enabling it to formally push data into Dynamics AND track the state in StateDB.

Targeted Personas

• Dario Data Professional: who will be going in to set this up in the UI based on project requirements
• Debbie Delivery Engineer: who will be doing the same ^

Current Behavior

Our current data processing pipeline creates gold tables, which are then read into a notebook to push into Dataverse with a Databricks workflow. This is manually triggered within the Databricks UI during DDU OR it can be configured to run as a staging notebook during model build (not recommended but it works).

Outcomes / Goals

Automated publishing of silver and gold tables to Dynamics 365 Dataverse through the publishing flows method. This initiative requires support for two new publishing-only data source types: Dynamics CE (Dataverse) and Dynamics F&O. Note this data source (for now) is NOT the Dynamics CE via Synapse Link Data source type, which is acquisition only!

Because we are adding new publish-only data sources to our interface, we need to add a visual component to data sources to signify their publish capabilities.

This will be a singular effort with the potential for future enhancements based on user feedback and evolving requirements.

User Stories

Configuring a Publishing Flow to Dynamics

• As a data professional, I want to take gold layer tables (which are already in the proper format) and move them to Dataverse (Dynamics CE).
• As a data professional, I want to take gold layer tables (which are already in the proper format) and move them to Dynamics F&O.

Adding a new Dynamics CE/F&O Sources

• As a data professional, I want to add a Dynamics source to my list of data sources, which is publishing capable only, so I can push data out to it in a publishing flow.

Use cases

Create/Configure Publishing Flows

1. Creating a new Dynamics Publishing Flow
   1. Primary Actor: Data Professional
   2. Description: A user can create a new publishing flow and specify a Dynamics Publishing only data source to make it a "Dynamics Publishing Flow".
   3. Preconditions: User has already created a Dynamics publish-only data source
   4. Postconditions: User has completed configuration of a publishing flow for that Dynamics source and would be able to trigger or schedule it.
   5. Flow:
      1. User navigates to the publishing flows page from the navigation menu.
      2. User clicks "CREATE" to make a new publishing flow.
      3. During the configuration step, User fills in the name of the flow.
      4. When they can specify the data source, the user picks a pre-configured Dynamics data source.
      5. User clicks "SAVE" and the publishing flow is created.
      6. Outcome: User now has an newly unscheduled dynamics publishing flow.
2. Configuring a Dynamics Publishing Flow -> this will have the same kind of UX as configuring any publishing flow.

Creating a new Dynamics CE/F&O publish-only Source

1. Creating a new Dynamics CE/F&O Source
   1. Primary Actor: Data Professional
   2. Description: A user can create a new publish-only Dynamics source like they can create any other.
   3. Preconditions: User has authentication details for this source on-hand.
   4. Postconditions: User has a newly configured Dynamics source. They will not be able to extract metadata from this source.
   5. Flow:
      1. User navigates to the Data Sources page from the navigation menu.
      2. User clicks "+" to create a new source
      3. User can search "Dynamics" to quickly find the Dynamics F&O and Dynamics CE/Dataverse publish-only data sources.
      4. User can click on one of them; fields are displayed relevant to the source's connection requirements. A tag is displayed showing this source is "Publishing" at the top of the data source configuration pane.
      5. User fills out required and optional fields, User clicks "SAVE AND TEST".
      6. Outcome: This source is now saved. Note, Dynamics sources wont be tested as they are Spark connectors.

Requirements

Functional

• List of functional description of the feature

Non-functional requirements

• Simplicity and ease of use.
• Similarity to existing publishing flow creation/configuration.

Out of Scope

• Modifications to the data processing pipeline unrelated to publishing bronze, silver, and gold tables.
• Anything other than Delta -> Dynamics, this is only concerned with pushing to Dynamics once its in Delta format.

Dependencies & Constraints

• Availability of development resources for implementing the publishing functionality.
• Access to Dynamics 365 Dataverse for testing and validation. (BlueHorizon)
• Integration with existing data processing pipeline components. (Flows)

Assumptions

We assume that publishing silver and gold tables to Dynamics 365 Dataverse can be seamlessly integrated into our existing publishing flows, and that there are no other dependencies outside of what a usual publishing flow requires, like SQL Server. SFTP was a special case that needed other ADF components. We assume that Dynamics DOES NOT require additional components.

Risks

1. Delays in development or insufficient resources.
2. Unanticipated technical challenges during the implementation of publishing functionality.
3. Potential impact on existing data processing components during deployment.

Timeline and Milestones

Phase I

• Release 2.0
• Components, other deliverables

Questions?