Skip to main content

Product Requirements Document

Mission​

The purpose of the Configuration Promotion/Migration feature is to streamline and automate the process of migrating configurations for data movement from a source environment to a target environment. This will enable users to efficiently transfer configurations between different environments, reducing the risk of manual errors and saving time.

Ultimately, this will improve the overall data management process for the customer, as they will be able to test a config in a lower environment and easily move it to the next environment when ready.

Background​

In the current system, users need to manually migrate configurations for data movement between different environments, such as development, testing, and production. This manual process is time-consuming, error-prone, and can lead to inconsistencies between environments. The Configuration Promotion/Migration feature aims to address these pain points by providing an automated solution for migrating configurations, ensuring a more accurate and efficient transfer process. This will ultimately contribute to smoother data management and help the business achieve its goals of data consistency and reliability across environments.

Targeted Personas​

Who is the primary beneficiary of this initiative? Why? Are there secondary beneficiaries?

Acceptance Criteria​

  • The system successfully checks user permissions before allowing them to trigger the config promotion feature through the UI.
  • The system applies a global GUID on all identified tables for promotion across different environments as a unique identifier.
  • The system promotes Sources and their associated secrets across environments, considering both unidirectional flow and synchronization with lower environments.
  • The system promotes all entities associated with the Model Builder together, ensuring that users do not promote partial details.
  • The system handles the promotion of ObjectList and FieldList entities, taking into account human-editable and system-editable fields, and interacts with users for confirmation of changes.
  • The system executes database triggers to set flags on fields that are edited by users or through API endpoints.
  • The system provides a user-friendly UI for initiating the configuration migration process and reviewing the changes before promoting them.
  • The system sends notifications or alerts to users regarding the success or failure of the migration process.
  • The system generates detailed logs and reports for the migration process, enabling users to troubleshoot any issues.
  • The system provides an API for programmatically interacting with the Configuration Promotion/Migration feature, allowing for integration with existing processes and tools.
  • The system ensures data consistency and accuracy during the promotion process, minimizing the risk of manual errors.
  • The system has a user-friendly interface that is easy to navigate and understand for various user roles.
  • The system is scalable and able to handle many configurations and simultaneous promotions.
  • The system has a high level of security, ensuring that confidential data and configurations are protected during the promotion process.
  • Build a capability to discard a promotion if user does not want to merge the changes after previewing the changes.
  • User can undo the last promotion changes using the Revert/Rollback capability through UI to make is fail safe.

User Stories​

User Story 1: As a data engineer, I want to migrate configurations between development, testing, and production environments using a user-friendly UI, so that I can ensure data consistency and reduce manual errors.

User Story 2: As a data analyst, I want to review and validate migrated configurations in the target environment, so that I can confirm the accuracy of the data movement settings and maintain data quality.

User Story 3: As a system administrator, I want to monitor and manage the access control of users who interact with the Configuration Promotion/Migration feature, so that I can maintain the security and integrity of our data platform.

User Story 4: As an internal delivery engineer, I want to interact with the Configuration Promotion/Migration feature at the API/Data layer, so that I can programmatically migrate configurations and integrate the feature into our existing processes and tools.

Requirements​

Functional​

  • The system should check user permissions before allowing them to trigger the config promotion feature through the UI.
  • The system should apply a global GUID on all the identified tables for promotion across different environments. The GUID will be used as a unique identifier for promotion.
  • The system should utilize existing scripts and enhance them to apply the GUID for all identified entities across different StateDb environments.
  • The system should evaluate the feasibility of using the cross-database elastic query for the config promotion.
  • The system should support the promotion of Sources and their associated secrets across environments, considering both unidirectional flow and synchronization with lower environments.
  • The system should promote all entities associated with the Model Builder together, ensuring that users do not promote partial details.
  • The system should support the independent promotion of the DatabaseToStepCommand entity across different environments.
  • The system should handle the promotion of ObjectList and FieldList entities, considering both human-editable and system-editable fields, and interact with users for confirmation of changes.
  • The system should execute database triggers to set flags on fields that are edited by users or through API endpoints.
  • Configurations for the following entities are within scope:
    • DatabaseList
    • SourcePropertyValue
    • ObjectList
    • FieldList
    • EntityDependency
    • Entity
    • EntityColumn
    • EntityColumnType
    • EntityClass
    • EntityType
    • Model
    • DatabaseToStepCommand

Non-functional requirements​

  • The system should ensure data consistency and accuracy during the promotion process, minimizing the risk of manual errors.
  • The system should have a user-friendly interface that is easy to navigate and understand for various user roles.
  • The system should be scalable and able to handle a large number of configurations and simultaneous promotions.
  • The system should have a high level of security, ensuring that confidential data and configurations are protected during the promotion process.
  • The system should provide clear and comprehensive documentation for both the UI and API, enabling users to effectively utilize the feature.
  • The system should have a high level of performance and reliability, minimizing the risk of downtime or promotion failures.

Out of Scope​

  • Dropbox Data: because 'Dropbox'/Filedrop source types are treated differently on the Empower platform than other typical data sources, the promotion of such source types is out of scope for the initial release(s) of this feature. In the meantime, Dropbox source types will be promoted manually.

Dependencies & Constraints​

  • Config promotion feature relies on the successful application of global GUIDs on all identified tables for promotion across different environments.
  • The feature depends on the availability and proper functioning of the RBAC system to ensure user permission management.
  • The feature relies on the successful integration with the Azure SQL database and Key Vault for promoting Sources and their associated secrets.
  • Stress testing of the config promotion feature on large datasets to identify the overall time required for promotion and ensure its scalability.
  • Finalizing designs for promoting various entities separately or in logical groups while maintaining data integrity.
  • Implementing download capability for users to review changes before promoting them, involving the selection of appropriate compression formats and handling edge cases.
  • Ensuring data integrity and avoiding conflicts by using Snapshot Isolation or other suitable approaches during the promotion process.
  • The successful implementation of logging capabilities to provide detailed information about promotion activities, which can be used for displaying promotion history in the UI.
  • Build a capability to discard the promotion through UI by the user

Assumptions​

  • Config promotion will be integrated with the RBAC system to ensure users with proper roles (Executor on source and Writer on target) can carry out the promotion.
  • The system will be able to handle Unicode and special characters during data export from Azure SQL without any failures.
  • The config promotion feature will be extendable to incorporate additional entities in the future if more tables need to be promoted such as Flows and Publish entities.
  • Snapshot Isolation will be used to maintain state and data integrity on the target StateDb when promotion is in progress.

Risk & Mitigation​

Risk 1: Incomplete or inaccurate migration of configurations​

  • Inaccurate migration of configurations could lead to inconsistencies between environments, affecting data quality and system functionality.

Mitigation:​

  • Implement thorough testing and validation procedures to ensure accurate migration of configurations.
  • Provide users with the ability to preview and adjust configurations before initiating the migration process.

Risk 2: Insufficient scalability and performance​

  • The system may not be able to handle a large number of configurations or simultaneous migrations, leading to performance bottlenecks, slow response times, or system failures.

Mitigation:​

  • Conduct stress testing and performance analysis to identify potential bottlenecks and optimize the system for scalability.
  • Implement performance monitoring and alerting to proactively identify and address performance issues.

Risk 3: Security vulnerabilities​

  • Unauthorized access to confidential data or configurations during the migration process could lead to data breaches or compromise the integrity of the system.

Mitigation:​

  • Implement strict access control measures and integrate the feature with the RBAC system to ensure only authorized users can initiate the migration process.
  • Regularly review and update security protocols to protect against emerging threats and vulnerabilities.

Risk 4: Integration challenges with existing systems and tools​

  • Challenges may arise when integrating the Configuration Promotion/Migration feature with existing systems, tools, or processes, potentially causing delays or inconsistencies.

Mitigation:​

  • Clearly define integration requirements and guidelines to ensure consistency across the development process.
  • Collaborate closely with relevant teams to address potential integration issues early on and establish a plan for ongoing maintenance and support.

Risk 5: Unmet user expectations or usability issues​

  • The Configuration Promotion/Migration feature may not fully address user needs or may be difficult to use, leading to low adoption or user dissatisfaction.

Mitigation:​

  • Engage users in the design and development process, gathering feedback and incorporating it into the feature's design.
  • Conduct user testing and usability studies to identify and address potential usability issues before releasing the feature.

Timeline and Milestones​

Phase I​

  • Release #.##
  • Components, other deliverables

Questions?​