Implementation Reference

Config promotion

Promotion state

Promotion state is derived from the state of each table in the promotion group. Each table can be in one of the following states:

• CREATED - the promotion was started, but celery workers have not started processing the table yet.
• EXPORT_FAILED - en error occurred while exporting the table.
• TARGET_INIT_FAILED - initialization of the target environment failed.
• PREVIEW_FAILED - an error occurred while generating the config promotion preview.
• MERGE_FAILED - an error occurred while merging the data into the target environment.
• ROLLBACK_FAILED - an error occurred while rolling back the target environment.
• EXPORT_RUNNING - the table is being exported into blob storage.
• TARGET_INIT_RUNNING - the target environment is being initialized.
• PREVIEW_RUNNING - the config promotion preview is being generated.
• MERGE_RUNNING - the data is being merged into the target environment.
• ROLLBACK_RUNNING - the target environment is being rolled back.
• EXPORT_COMPLETE - the table was successfully exported into blob storage, but merge has not started yet.
• TARGET_INIT_COMPLETE = the target environment was successfully initialized
• PREVIEW_COMPLETE - the config promotion preview was successfully generated.
• ROLLBACK_COMPLETE - the target environment was successfully rolled back.
• MERGE_COMPLETE - the data was successfully merged into the target environment.
• EXPORT_SKIPPED - export was skipped because the table is empty.
• PREVIEW_DISCARDED - users have discarded the config promotion preview.

If promotion tables have different states, the promotion state is determined by order in the list above, so if we have a promotion consisting of 3 tables,
one in MERGE_COMPLETE, one in MERGE_RUNNING and one in EXPORT_FAILED, the promotion state will be EXPORT_FAILED.

In a "happy path" scenario, we're looking into following state changes:

CREATED -> EXPORT_RUNNING -> EXPORT_COMPLETE -> TARGET_INIT_RUNNING -> TARGET_INIT_COMPLETE -> MERGE_RUNNING -> MERGE_COMPLETE.

Or when creating a config promotion preview:

CREATED -> EXPORT_RUNNING -> EXPORT_COMPLETE -> TARGET_INIT_RUNNING -> TARGET_INIT_COMPLETE -> PREVIEW_RUNNING -> PREVIEW_COMPLETE.

Public API

This section describes the endpoints that are used by the UI to create and manage Config promotion. Config promotion has several other endpoints that are used by the backend to manage the promotion state, perform internal operations but they are never called by the UI and should not be called directly by the users/QA.

Promotion list

GET /api/v1/promotions/ - returns a paginated list of promotions.

{
  "count": 1,
  "next": null,
  "previous": null,
  "results": [
    {
      "id": "39376531-cfd7-4449-85a8-36bc296aea04",  // unique id of the promotion
      "createdBy": "user@hitachisolutions.com",  // user who created the promotion
      "duration": 1.2983508329780307,  // total duration of the promotion operations on the worker
      "exportDuration": 0.8854285289708059, // total duration of the export tasks for all tables
      "uploadDuration": 0.13583119100076146,  // total duration of the upload tasks for all tables
      "targetInitDuration": 0.2770911130064633, // total duration of the target init tasks for all tables
      "mergeDuration": 0, // total duration of the merge tasks for all tables
      "status": "PREVIEW_FAILED", // current status of the promotion
      "exportedRows": 4236, // total number of rows exported for all tables
      "sourceEnvironment": "9519211a-9f8c-4dac-8e11-9210751f2ae8",  // id of the source environment
      "targetEnvironment": "57c9d42d-cac3-4bbb-ab9e-bb3a18cde833", // id of the target environment
      "backup": true,  // whether the target environment was backed up before the promotion
      "createdAt": "2023-09-29T17:22:16.870734",
      "updatedAt": "2023-09-29T17:22:16.870749"
    }
  ]
}

Create promotion

POST /api/v1/promotions/ - create a new promotion.

{
  "environmentId": "3fa85f64-5717-4562-b3fc-2c963f66afa6", // global id of the target environment
  "commit": false,  // whether the promotion should be committed or discarded
  "filters": [  // optional filters
    {
      "groupId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",  // global id of a promotion group
      "values": [ // optional list of global ids of the top-level records that will be migrated
        "3fa85f64-5717-4562-b3fc-2c963f66afa6"
      ]
    }
  ],
  "backup": true // whether the target environment should be backed up before the promotion
}

Promote sources (DatabaseList)

POST /api/v1/promotions/sources/ - create a promotion for a DatabaseList:

Request:

{
  "environmentId": "3fa85f64-5717-4562-b3fc-2c963f66afa6", // global id of the target environment
  "globalIds": [  // optional list of global ids of the top-level records that will be migrated
    "3fa85f64-5717-4562-b3fc-2c963f66afa6"
  ]
}

Promotion details

GET /api/v1/promotions/{id}/ - same as promotion list, but returns a single non-paginated promotion.

Commit config promotion preview

POST /api/v1/promotions/{id}/commit/ - if promotion was created with commit set to false, this endpoint can be used to commit the promotion. This will change the state of the promotion to MERGE_COMPLETE if successful. Can only be executed if promotion is in PREVIEW_COMPLETE state.

Example request:

{
  "force": true // force the commit even if target environment data was modified since the preview was generated, by default this is set to false.
}

Discard config promotion preview

PATCH /api/v1/promotions/{id}/discard/ - discard the promotion preview. This will change the state of the promotion to PREVIEW_DISCARDED. Can only be executed if promotion is in PREVIEW_COMPLETE state.

ZIP Downloads

GET /api/v1/promotions/{id}/downloads/ - download links for export and preview ZIP files. May return null values if zip files are not created yet.

Rollback promotion

POST /api/v1/promotions/{id}/rollback/ - rollback a target database to a state before the promotion was merged. Can only be executed if promotion is in MERGE_COMPLETE state. The records that were added to target database after the promotion was merged will be deleted.

Promotion exports

GET /api/v1/promotions/{promotionId}/exports/ - provides download urls for export files for individual tables in the promotion. May return null values if export files are not created yet.

Promotion entities

GET /api/v1/promotions/entities/ - a paginated list of all promotable entities, each item has the following schema:

{
    "id": "2690f7fe-f4f3-47d0-b838-b31db0a6f9d6",  // global id of the entity
    "tableName": "Entity",  // table name
    "schemaName": "model",  // schema name
    "promotionGroup": "e96979e2-d7b4-4d92-864b-3fd528180ecd", // global id of the promotion group entity belongs to
    "parents": [  // list of global ids of the parent entities
        "df9b07c3-04b0-477a-8b44-4a9b06688e19",
        "8a90bde7-1260-41e7-896f-dd87cafce27d"
    ]
}

Environment list

GET /api/v1/promotions/environments/ - a list of available promotion environments, returns a non-paginated list:

[
  {
    "name": "Development",
    "globalId": "9519211a-9f8c-4dac-8e11-9210751f2ae8",
    "stageOrder": 0
  },
  {
    "name": "Continuous Deployment",
    "globalId": "57c9d42d-cac3-4bbb-ab9e-bb3a18cde833",
    "stageOrder": 1
  }
]

Promotion groups

GET /api/v1/promotions/groups/ - a list of promotion groups.

{
  "count": 2,
  "next": null,
  "previous": null,
  "results": [
    {
      "id": "e96979e2-d7b4-4d92-864b-3fd528180ecd",
      "name": "Model Group",
      "filterTable": null
    },
    {
      "id": "61eba498-ac2b-4d73-ba26-5d9321553960",
      "name": "Source Group",
      "filterTable": null
    }
  ]
}

Implementation details

High level overview of the Config promotion process

sequenceDiagram

box rgb(50, 92, 168) Source Environment
    participant Source API
    participant Source DB
    participant Source Worker
end
participant Blob Storage
box rgb(40, 158, 107) Target Environment
    participant Target API
    participant Target DB
    participant Target Worker
end
Source API ->> Source DB: Crate promotion record
Source API ->> Source Worker: Create export task for each table
Source Worker ->> Source DB: Generate export CSV
Source Worker ->> Blob Storage: Upload exports CSVs
Source Worker ->> Source DB: Update promotion state
Source Worker ->> Blob Storage: Upload exports zip
Source Worker ->> Target API: Trigger merge
Target API ->> Target Worker: Run target initialization
Target Worker ->> Target DB: Create a backup
Target Worker ->> Blob Storage: Upload backup
Target Worker ->> Target DB: Start merge
Target DB ->> Blob Storage: Retrieve source exports
Target DB ->> Target DB: Merge source exports into target
Target Worker ->> Source API: Update promotion state
Source API ->> Source DB: Update promotion
Target Worker ->> Blob Storage: Upload preview csv zip

The promotion process explained

1. User retrieves a global id of a target environment using GET /api/v1/promotions/environments/ endpoint. This endpoint is powered by frondoor service discovery capabilities. Target environment can be any environment except the current environment.

2. (Optional) Build the promotion filters for the promotion:

   1. Retrieve a promotion group list using GET /api/v1/promotions/groups/ endpoint.
   2. Specify a list of global ids of the top-level records that will be migrated. This is optional, if no promotion filters are specified, all the data from the Source environment will be migrated.

3. Make a POST /api/v1/promotions/ request to create a new promotion. This will create a new promotion record in the source environment and trigger the export process.

4. Before creating a promotion record we check if there are any other unfinished promotions for the same target environment. If there are any, we return an error. This is done to prevent multiple promotions from running at the same time.

5. We create a promotion record in source environment database with state CREATED. This promotion record contains a user, source and target environment ids, whether backup should be created.

6. If promotion filter values were provided - we also create a promotion filters record for each promotion filter values. This record contains a promotion id, promotion group id and a list of global ids of the top-level records that will be migrated.

7. We query promotable entities table to get a list of all promotion tables for given promotion groups and sort the list by the dependency tree (i.e. if table A depends on table B, table B will be first in the list).

8. API spawns a group of export_entity tasks on a Source environment celery worker for each entity:

   1. It sets the state of promotion table to EXPORT_RUNNING.
   2. PrepareDataForExport stored procedure is executed to generate a dump of the table data and include row hashes.
   3. If the table is empty, we set the state of the table to EXPORT_SKIPPED and skip the rest of the steps.
   4. The export csv file is uploaded to blob storage into "{promotion_id}/export" directory.
   5. Worker updates the state of promotion table to EXPORT_COMPLETE and saves the various metadata (export durations, blob path).
   6. If exists, we pass the blob path of a previous promotion to the trigger_merge task (later used during merge).
   7. It passes export results to the trigger_merge and zip_blob_directory tasks.

9.zip_blob_directory is called for the "{promotion_id}/export" directory which produces a single zip file with contents of the directory.

10. Trigger merge makes a request to PUT /promotions/merge endpoint on a target environment with the results of export in a body

{
    "promotionId": "3fa85f64-5717-4562-b3fc-2c963f66afa6", // global id of the promotion
    "sourceEnvironmentId": "3fa85f64-5717-4562-b3fc-2c963f66afa6", // global id of the source environment
    "commit": false, // passed from the original request
    "backup": true, // passed from the original request
    "exports": [
        {
            "entityName": "field_list",  // promotable entity name
            "tableId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",  // global id of the promotion table
            "blobName": "foo/field_list.csv",
            "prevBlobName": "bar/field_list.csv",
        }
    ],
    "filters": {
        // filters are passed from the original request
    }
}

11. Merge on target environment spawns a group of init_target tasks for each export from the request:

    1. If backup was not requested and prevBlobName is present, this task is skipped.
    2. Celery worker on a target environment makes a call to PATCH /api/v1/promotions/{promotionId}/tables/{id}/ on source environment to update the state of the table to TARGET_INIT_RUNNING.
    3. If previous entity is not found for export, we execute PrepareDataForExport (this is the same procedure that is used for exporton source environment) to generate "init" dump of table on a target environment. This is treated as a previous promotion during merge. Init file is uploaded to "{promotion_id}/init" directory.
    4. If backup was requested the contents of the table are backed up into "{promotion_id}/backup" directory as csv.
    5. PATCH /api/v1/promotions/{promotionId}/tables/{id}/ is called to update the state of the table to TARGET_INIT_COMPLETE.

12. merge task is called with the list of export:

    1. For each promotable entity we make a request to source environment to update the state of the table to MERGE_RUNNING.
    2. MergePromotableData is executed with a path to the export file in blob storage and a path to the previous export file in blob storage or a path to the init file if previous export file is not found:
       • The procedure inserts data from the export file into a temporary table.
       • It updates the primary keys of the temporary table to match the primary keys of the target table.
       • The data from previous or init file is inserted into another temporary table.
       • MergePromotableData is executed which dynamically generates a MERGE statement and outputs a data for config promotion preview .
       • Config promotion preview data is uploaded to "{promotion_id}/preview" directory.
       • If commit is true, the transaction is committed and state of the promotion on Source environment is updated to MERGE_COMPLETE.
       • If commit is false, the transaction is rolled back and state of the promotion on Source environment is updated to PREVIEW_COMPLETE.

13. zip_blob_directory is called for the "{promotion_id}/preview" directory which produces a single zip file with contents of the directory.