Azure dataset .to_pandas_dataframe() error

Question

I am following an azure ml course on udemy and cannot get around the following error:

Execution failed in operation 'to_pandas_dataframe' for Dataset(id='id', name='Loan Applications Using SDK', version=1, error_code=None, exception_type=PandasImportError)

Here is the code for Submitting the Script:

from azureml.core import Workspace, Experiment, ScriptRunConfig, 
Environment


ws = Workspace.from_config(path="./config")


new_experiment = Experiment(workspace=ws,
                            name="Loan_Script")


script_config = ScriptRunConfig(source_directory=".",
                                script="180 - Script to Run.py")

script_config.framework = "python"
script_config.environment = Environment("conda_env")

new_run = new_experiment.submit(config=script_config)

Here is the Script being run:

from azureml.core import Workspace, Datastore, Dataset, 
Experiment

from azureml.core import Run

ws = Workspace.from_config(path="./config")
az_store = Datastore.get(ws, "bencouser_sdk_blob01")
az_dataset = Dataset.get_by_name(ws, name='Loan Applications Using SDK')
az_default_store = ws.get_default_datastore()


#%%----------------------------------------------------
# Get context of the run
#------------------------------------------------------


new_run = Run.get_context()


#%%----------------------------------------------------
# Stuff that will be logged
#------------------------------------------------------

df = az_dataset.to_pandas_dataframe()

total_observations = len(df)

nulldf = df.isnull().sum()

#%%----------------------------------------------------
# Complete the Experiment 
#------------------------------------------------------

new_run.log("Total Observations:", total_observations)

for columns in df.columns:
    new_run.log(columns, nulldf[columns])

new_run.complete()

I have run the .to_pandas_dataframe() part outside of an experiment and it worked without error. I have also tried the following (that was recommended in the driver log):

InnerException Could not import pandas. Ensure a compatible version is installed by running: pip install azureml-dataprep[pandas]

I have seen people come across this before but I cannot find a solution, any help is appreciated.

Answer 1

When doing an experiment a new azure environment was created without pandas installed. To install pandas (if using anaconda nav) go onto environments in the anaconda nav window, click the azure env, go to uninstalled packages and search pandas, click install. It worked once this was done.