Configuration Promotion implementation for Flows

Flow promotion

Configuration Promotion for Flows is based on the PromotableEntity and PromotableEntityGroup model's data (same as the regular Configuration Promotion process) except it has its own set of endpoints because of the additional business logic related to the Flow types which requires a different way of filtering the promotable data.

Supported Flow types

Flow promotion currently includes 3 promotion groups:

1. Analytics Engineering (AE):
   • [dbo].[flows_flow] (root table)
   • [model].[Entity]
   • [model].[EntityClass]
   • [model].[EntityColumn]
   • [model].[EntityColumnType]
   • [model].[EntityDependency]
   • [model].[EntityType]
   • [model].[Model]
2. Data Acquisition (DAQ):
   • [dbo].[flows_flow] (root table)
   • [state_config].[DatabaseList]
   • [state].[FieldList]
   • [dbo].[flows_flowsourceentity]
   • [state].[ObjectList]
3. Custom Flow:
   • [dbo].[flows_flow] (root table)
   • [dbo].[flows_flowparameter]

Regular Promotion vs Flow Promotion

Single Flow group per promotion

The core difference between the regular promotion process and promotion of the Flows is that a single Flow type (Promotion group) can be promoted at once whereas the regular promotion flow allows multiple filters for any groups and entities.

The payload attribute filters for a regular promotion create endpoint expects an array of objects:

"filters": [
  {
    "groupId": "<UUID>",  // the ID of PromotableEntityGroup
    "values": [  // an array of root entity global_id values to filter by
      "<UUID>",
      ...
    ]
  }
]

Flow promotion has a different attribute called advancedFilters that expects a single object:

"advancedFilter": {
  "groupId": "<UUID>", // the ID of the PromotableEntityGroup
  "column": "<string>",  // the name of the DB column of the root entity to filter the related entities by 
  "values": [  // the values of the `column` attribute values to filter by
    "<string>",
    ...
  ]
}

More detailed explanation of how the "Advanced Filters" work is in the Advanced filters/joins section.

Multiple PromotableEntitieGroups with a common root entity

Alongside Flow promotion the possibility to have a common root entity among multiple promotion groups has been provided. For example, the [dbo].[flows_flow] database table can be set as a root entity for DAQ, AE and Custom Flow groups which wasn't possible before.

The implementation is rather a walk-around rather than an actual feature. In order to assign the same root entity to multiple promotion groups - multiple records of the same entity are created (having different ID) and each of the created entities would be assigned to the required promotion group.

Example:

[
  {
    "model": "config_promo.promotableentity",
    "pk": "9b279276-029e-4890-bb53-3b1214c56841",
    "fields": {
      "table_name": "flows_flow",
      "schema_name": "dbo",
      "promotion_group": "86c8bc85-2b3f-4e31-88a9-2739a9577776",  // Data Acquisition Group
      "include_in_promotion": true,
      "model_name": "flows.Flow",
      "is_root_entity": true,
      "parents": []
    }
	},
  {
    "model": "config_promo.promotableentity",
    "pk": "ad428c92-ed14-4fff-9d8e-cfb03f7481ae",
    "fields": {
      "table_name": "flows_flow",
      "schema_name": "dbo",
      "promotion_group": "d31e94ab-ce76-460c-ad88-caf9df7d1f56",  // Analytics Engineering Group
      "include_in_promotion": true,
      "model_name": "flows.Flow",
      "is_root_entity": true,
      "parents": [
        "b2c717f4-b910-4056-95fd-17da392b1a3e"
      ]
    }
  }
]

As per the example code snippet the entities are almost identical and refer the same database table and Django model except they're assigned to different promotion groups which essentially allows promotion of different Flow types.

Advanced filters/joins

The data filtering in the regular promotion process is based on the strict relation of dependent entities to the root entity. To avoid altering the tables with additional relations an "Advanced filters/joins" functionality has been implemented.

During each configuration promotion process an entity dependency tree-like structure is calculated for each group. It goes from bottom-to-top (from the root entity to the very last child entity). That tree-like structure is essential as it allows to promote data in the correct order not braking the constraints.

Having a group of entities with chain dependencies (A depends on B and C depends on B; where C is a root entity) in order to filter out lower level entities by the root entity the SQL "JOIN" operations are used. The actual "JOIN" expression is being built based on the attributes of the root PromotableEntity such as table_name, schema_name, parents, is_root_entity.

New properties

In order to have more control over the promoted entities and groups the new properties have been added:

1. New PromotableEntity columns:
   1. is_root_entity: bool: mark entity as a root entity of promotable group; used to the creation of advanced filters and joins.
   2. model_name: string: string representation of the Django model name; used to have an easier way of retrieving the Django model class by the name for the further operations over it.
   3. include_in_promotion: bool: mark an entity as the one not to include in promotion; currently set to true for [state_config].[DatabaseList] table as it has it's own set of endpoints for promotion different from the regular promotion logic.
2. New PromotableGroup columns:
   1. active: bool: mark group as inactive in order to disallow promoting it (without deleting one).
   2. is_legacy: bool: mark group as "legacy" (i.e. implemented before Flow promotion).

Shortcomings of the current implementation