'Pip install Cannot run program "pip" (in directory "."): error=2, No such file or directory --EMR trigged through airflow

I am trying to install requirements.txt through "Add steps"(command-runner.jar) in EMR triggered through Airflow.

and it is giving and error Error:

 Cannot run program "pip" (in directory "."): error=2, No such file or directory

My Airflow Code :

SPARK_STEPS = [
    {
        'Name': 'Copy S3 to EMR',
        "ActionOnFailure": "CANCEL_AND_WAIT",
        'HadoopJarStep': {
            "Jar": "command-runner.jar",
            "Args": ['aws' ,'s3', 'cp' ,'s3://datahub-pyspark/pitchbook/', '.', '--recursive'],
        },
    },
    
    {
        'Name': 'Upgrade Pip',
        "ActionOnFailure": "CANCEL_AND_WAIT",
        'HadoopJarStep': {
            "Jar": "command-runner.jar",
            "Args": ['pip3','install','--upgrade','pip'],
            
        },
    },

    {
        'Name': 'Execute Requirements',
        "ActionOnFailure": "CANCEL_AND_WAIT",
        'HadoopJarStep': {
            "Jar": "command-runner.jar",
            "Args": ['pip','install','-r','requirements.txt'],
            
        },
    },
    {
        'Name': 'Spark-Submit Command',
        "ActionOnFailure": "CANCEL_AND_WAIT",
        'HadoopJarStep': {
            "Jar": "command-runner.jar",
            "Args": [
                'spark-submit', '--py-files', 'config.zip,jobs.zip', 'main.py']
            ,
        },
    }

]

Name': 'Execute Requirements' is failing so I have 'Upgrade Pip' also.Please give me how to remove this error?

Thanks, Xi

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source

'Pip install Cannot run program "pip" (in directory "."): error=2, No such file or directory --EMR trigged through airflow

Sources

Related Questions