'Azure Databricks automation databricks-cli authentication issue aad token

I am trying to create data toolchain in automated way. I am using Azure, databricks-cli. https://github.com/Azure-Samples/modern-data-warehouse-dataops/blob/main/e2e_samples/parking_sensors/scripts/deploy_infrastructure.sh

I have issue to authenticate to databrics with aad token, generate PAT.

echo "Generate Databricks token"
databricks_host=https://$(echo "$arm_output" | jq -r '.properties.outputs.databricks_output.value.properties.workspaceUrl')
databricks_workspace_resource_id=$(echo "$arm_output" | jq -r '.properties.outputs.databricks_id.value')
databricks_aad_token=$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d --output json | jq -r .accessToken) # Databricks app global id

This is causing me problems.

# Use AAD token to generate PAT token
databricks_token=$(DATABRICKS_TOKEN=$databricks_aad_token \
    DATABRICKS_HOST=$databricks_host \
    bash -c "databricks tokens create --comment 'deployment'" | jq -r .token_value)

How to authenticate to databricks in order to being able to use databricks cli ??

later I am trying to create secrets but it fails as I am not authenticated.

# Create secret scope
databricks secrets create-scope --scope "$scope_name" \
    --scope-backend-type AZURE_KEYVAULT \
    --resource-id "$KEYVAULT_RESOURCE_ID" \
    --dns-name "$KEYVAULT_DNS_NAME"

Thank you Alex, unfortunately it still does not work on Azure. System:

az login - as subscription owner done

databricks -v
Version 0.16.4

export DATABRICKS_HOST='https://xxx-xxx.16.azuredatabricks.net'
export DATABRICKS_TOKEN=$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d --output json | jq -r .accessToken)

First attempt:

databricks tokens list
Error: b'<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>\n<title>Error 403 User not authorized.</title>\n</head>\n<body><h2>HTTP ERROR 403</h2>\n<p>Problem accessing /api/2.0/token/list. Reason:\n<pre>    User not authorized.</pre></p>\n</body>\n</html>\n'

Second attempt:

databricks secrets create-scope --scope "XXX"     --scope-backend-type AZURE_KEYVAULT     --resource-id "$KEYVAULT_RESOURCE_ID"     --dns-name "$KEYVAULT_DNS_NAME"
Error: b'<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>\n<title>Error 403 User not authorized.</title>\n</head>\n<body><h2>HTTP ERROR 403</h2>\n<p>Problem accessing /api/2.0/secrets/scopes/create. Reason:\n<pre>    User not authorized.</pre></p>\n</body>\n</html>\n

Solution 1:^[1]

You don't need personal access token to create a scope. Just set DATABRICKS_HOST to URL of workspace and DATABRICKS_TOKEN to value of AAD token, and then use databricks secrets create-scope - this command won't work with personal access token. Something like this:

export DATABRICKS_HOST=...
export DATABRICKS_TOKEN=$(az account get-access-token --resource \
  2ff814a6-3304-4ab8-85cb-cd0e6f879c1d --output tsv --query accessToken)
databricks secrets create-scope --scope "$scope_name" \
    --scope-backend-type AZURE_KEYVAULT \
    --resource-id "$KEYVAULT_RESOURCE_ID" \
    --dns-name "$KEYVAULT_DNS_NAME"

but please note that this AAD token should be of the real user, not service principal - that's a known limitation:

You need an Azure AD user token to create an Azure Key Vault-backed secret scope with the Databricks CLI. You cannot use an Azure Databricks personal access token or an Azure AD application token that belongs to a service principal.

P.S. If you're automating things, you can look onto Databricks Terraform Provider that can help with such stuff.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1

'Azure Databricks automation databricks-cli authentication issue aad token

Solution 1:[1]

Sources

Related Questions

Solution 1:^[1]