Skip to main content

Product Requirements Document

Mission​

We want to simplify our existing lexical parser (see docs) and remove custom parsing, by navigating to a python AST and parsing library.

Background​

We have a complex lexical parser and mapping engine documented here. In a nutshell, they support the following functions:

  1. source_id(parser_log, _id)
  2. source_global_id(parser_log, _id)
  3. source_cluster_id(parser_log, _id)
  4. source_property_value(parser_log, _id, property_name)
  5. [internal/external]_keyvault_get_secret_value(parser_log, name)
  6. [internal/external]_keyvault_save_secret(parser_log, secret_name, connection_string)
  7. concat(parser_log, *args)
  8. string_with_value(parser_log, key, value)
  9. equality_condition(parser_log, left_part, right_part, true_case, false_case=None)
  10. env(parser_log)

Targeted Personas​

  • The users of this are purely internal developers from the Data and X pod teams who are building connection strings for data sources.

Current Behavior​

Our current parser is a custom engine with complex logic. This makes it a challenge to add new functionality to it quickly and efficiently. However, this is a complex piece of logic that works effectively today. Any new engine we build must be backwards compatible.

Outcomes / Goals​

  • The end goal of this initiative is to develop a simple lexical parsing engine using the AST library to support existing functionality while:
    • handling more corner cases
    • easing our maintenance load
    • creating an extensible engine
    • being backwards compatible with the previous iteration
    • generating more meaningful error reports for users upon validation
  • Support for ODBC/JDBC String Types ("valueA=a;valueB=b")
  • Support for JSON String Types ('{"valueA": "a", "valueB": "b"}')
  • Internal handling for any necessary encoding/decoding for special characters in either format

User Stories​

Setting up an ODBC Connection String​

  • As a connector developer, I want to build traditional ODBC Connection Strings using this parser. This string will be passed to ADF, where we want to minimize calls for any additional post-processing to cut down on unnecessary ADF calls.

Setting up a JDBC Connection String​

  • As a connector developer, I want to build JDBC Connection Strings using this parser. The string will be passed through ADF to a python notebook. We want to minimize post-processing to be able to "plug-and-play" JDBC drivers without any user-modification required.

Setting up a JSON-style Connection String​

  • As a connector developer, for non ODBC/JDBC source contexts, I want to build Connection Strings in JSON format. The string will be passed through ADF to a python notebook. I want to make sure my passwords are able to handle unique characters like "!" and "$", as examples.

Use cases​

Setting Connection Strings in FDS​

  1. Setting up an ODBC Connection String
    1. Primary Actor: Connector Developer
    2. Description: Build ODBC Connection Strings using the new parser
    3. Preconditions: Connection String details are known and data source + execution context is already setup
    4. Postconditions: Functional ODBC Connection String exists in keyvault with proper handling of unique characters with no post-processing required
    5. Flow:
      1. Developer naviates to selected source's execution context.
      2. Connector developer inputs their ODBC Connection String ("valueA=a;valueB=b")
      3. The new parser processes the string and inserts into keyvault.
      4. The output is a functional ODBC Connection String with minimal post-processing required.
  2. Setting up a JDBC Connection String
    1. Primary Actor: Connector Developer
    2. Description: Build JDBC Connection Strings using the new parser
    3. Preconditions: Connection String details are known and data source + execution context is already setup
    4. Postconditions: Functional JDBC Connection String exists in keyvault with proper handling of unique characters with no post-processing required
    5. Flow:
      1. Connector developer inputs their JDBC Connection String ("valueA=a;valueB=b")
      2. The new parser processes the string and inserts into keyvault.
      3. The output is a functional JDBC Connection String with minimal post-processing required
  3. Setting up a JSON-style Connection String
    1. Primary Actor: Connector Developer
    2. Description: Build JSON-style Connection Strings using the new parser
    3. Preconditions: Connection String details are known and data source + execution context is already setup
    4. Postconditions: Functional JSON-style Connection String exists in keyvault with proper handling of unique characters
    5. Flow:
      1. Connector developer inputs their JSON-style Connection String ('{"valueA": "a", "valueB": "b"}')
      2. The new parser processes the string and inserts into keyvault.
      3. The output is a functional JSON-style Connection String with unique characters handled correctly

Requirements​

Functional​

  • List of functional description of the feature

Non-functional requirements​

  • The parser should be easy to maintain
  • The parser should provide efficient performance
  • The parser should be user-friendly for internal developers

Out of Scope​

  • Any functionality not related to parsing connection strings
  • Support for external developers or use cases beyond the targeted personas
  • Integration with any other tools or platforms not mentioned in the document

Dependencies & Constraints​

  • The new parser should be built using the Python AST and parsing library
  • It must work with the existing connection string formats

Assumptions​

  • The internal developers have sufficient knowledge to use the new parser
  • The Python AST and parsing library will provide the necessary functionality to replace the current custom parser

Risks​

Timeline and Milestones​

Phase I​

  • Release #.##
  • Components, other deliverables

Questions?​