'How to uninstall pip library from Azure databricks notebook - without removing it from cluster library utility?

Trying to start data factory from databricks.

I am having conflict between Azure libraries installed on cluster level:

from azure.identity import ClientSecretCredential
from azure.mgmt.resource import ResourceManagementClient
from azure.mgmt.datafactory import DataFactoryManagementClient

azure_client_id = dbutils.secrets.get(scope="Azure_KeyVault", key="_Application_Id")
azure_client_secret = dbutils.secrets.get(scope="Azure_KeyVault", key="_Client_Secret")
azure_tenant_id = dbutils.secrets.get(scope="Azure_KeyVault", key="__Tenant_Id")

# example of trigger_object['topic']: /subscriptions/f8354c08-de3d-4a67-95ae-c7cbdb37fbf6/resourceGroups/WeS06DvBing15064/providers/Microsoft.Storage/storageAccounts/wes06dvraw15064

subscription_id = 'f4379743884938948398938493793749830'
credentials = ClientSecretCredential(client_id=azure_client_id, client_secret=azure_client_secret, tenant_id=azure_tenant_id)
dfmc = DataFactoryManagementClient(credentials, subscription_id, base_url="https://management.azure.com")

[f.id for f in dfmc.factories.list()]

Error message :

AttributeError: 'ClientSecretCredential' object has no attribute 'signed_session'

I think it could because we have Azure installed on this cluster using the cluster libraries utility. (Given that it works if i remove this library from cluster level).

When i'm doing this in the notebook : %pip uninstall Azure i'm getting :

Python interpreter will be restarted. Found existing installation: azure 4.0.0 Not uninstalling azure at /databricks/python3/lib/python3.7/site-packages, outside environment /local_disk0/.ephemeral_nfs/envs/pythonEnv-6eab9ca4-4cd6-4bd9-843f-8e33a185c96a Can't uninstall 'azure'. No files were found to uninstall. Python interpreter will be restarted.

I don't quite understand this last error message. I want to uninstall library in the notebook, but do not want to remove it from the cluster library utility level (it is used in many other notebooks)

azurepipazure-data-factoryazure-databricks


Solution 1:[1]

Libraries can be installed in two levels when it comes to data bricks.

  1. Workspace library
  2. Cluster library

1. Library

  1. Get into the folder containing libraries
  2. Select the library name which you need to uninstall
  3. Select the checkbox next to the library to which you need to uninstall, then confirm.
  4. After confirmation the status changes to uninstall pending restart

2. Cluster

  1. Go to the library folder
  2. Select the library
  3. Select the checkbox next to the name and select uninstall
  4. After confirmation it will be in pending state
  5. Restart the cluster

In this procedure, both the normal libraries and cluster libraries are isolated.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Dharman

Related Questions