Skip to main content

Setup Network Connectivity Configuration


Intro​

A Network Connectivity Configuration allows connectivity between databricks serverless SQL warehouses and private or firewalled storage accounts.

These docs go over how to setup an NCC for private networking.

private networking

firewalled networking

Pre Requisites​

  • Access to Empower-Service credentials
  • Contributor access on the customer storage accounts
  • Enable the Private Endpoint public preview feature on the customers databricks account.
    • This can be done through an Azure support request.

Steps​

  1. Run the following powershell script with the customer dev keyvault name and tenant id. Ensure Empower-Services secret is in the env variable in the script. Make sure to wait a minute for the aad token to be properly propogated before running the notebook.
$tenantId = ""
$spClientId = "9351f38a-380b-4255-9b08-7d2710871879" # Empower Service App ID
$spClientSecret = $Env:TF_VAR_sp_client_secret
$resource = "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d" # Global Azure Databricks Resource ID
$secretName = "empower-service-aad-token"
$keyvault = ""

az login --service-principal --tenant $tenantId --username $spClientId --password $spClientSecret
$aadToken = az account get-access-token --resource $resource --query "accessToken"
az keyvault secret set --name $secretname --vault-name $keyvault --value $aadToken

  1. Create a single user cluster in the workspace and start it up.

  2. Attatch the cluster to the notebook EmpowerSystem/UCMigration/Setup NCC.

  3. Add the customer details into the notebook. These details should be organization wide.

    • Add the ncc name based on customer naming convention
    • Add the Unity Catalog region
    • Add the customer storage account resource ids. Metastore storage account is not needed.
    • Add the customer workspace ids.

  4. Run the notebook.

  5. Take the subnet ids and whitelist them on the storage account through the Empower-Clients repo.

Potential Issues​

  1. Sometimes the token generated by the powershell script cannot be pulled from the keyvault. Manually add the value to the header generator in the notebook to solve this issue temporarily.