'EMR - Airflow to run scala jar file airflow.exceptions.AirflowException

I am trying to run a scala jar file from AIRFLOW using emr and the jar file is designed to read mssql-jdbc and postgresql.

From airflow, I'm able to create cluster enter image description here

My SPARK_STEPS looks like

SPARK_STEPS = [
    {
        'Name': 'Trigger_Source_Target',
        'ActionOnFailure': 'CONTINUE',
        'HadoopJarStep': {
            'Jar': 'command-runner.jar',
            'Args': ['spark-submit', 
                     '--master', 'yarn', 
                     '--jars', '/mnt/MyScalaImport.jar', 
                     '--class', 'org.classname', 
                     's3://path/SNAPSHOT.jar', 
                     'SQL_Pwd', 'PostgreSQL_PWD', 'loadtype'],
        }
    }
]

After this I have JOB_FLOW_OVERRIDES defined-

JOB_FLOW_OVERRIDES = {
    "Name": "pfdt-cluster-airflow",
    "LogUri": "s3://path/elasticmapreduce/",
    "ReleaseLabel": "emr-6.4.0",
    "Applications": [
        {"Name": "Spark"},
    ],
    "Instances": {
        "InstanceGroups": [
            {
                "Name": "Master nodes",
                "Market": "ON_DEMAND",
                "InstanceRole": "MASTER",
                "InstanceType": "m5.xlarge",
                "InstanceCount": 1,
            }
        ],
        "KeepJobFlowAliveWhenNoSteps": True,
        "TerminationProtected": False,
        'Ec2KeyName': 'pem_file_name',
        "Ec2SubnetId": "subnet-123"
    },
    'BootstrapActions': [
        {
            'Name': 'import custom Jars',
            'ScriptBootstrapAction': {
                'Path': 's3://path/subpath/copytoolsjar.sh',
                'Args': []
            }
        }
    ],
    'Configurations': [
      {
        'Classification': 'spark-defaults',
        'Properties': {
          'spark.jars': 's3://jar_path/mssql-jdbc-8.4.1.jre8.jar'
         }
      }
    ],
    "VisibleToAllUsers": True,
    "JobFlowRole": "EMR_EC2_DefaultRole",
    "ServiceRole": "EMR_DefaultRole",
    "Tags": [
        {"Key": "Environment", "Value": "Development"},
    ],
}

To copy the scala .jar file from S3 to local to airflow- I have a shell script which does the work: Path- s3://path/subpath/copytoolsjar.sh

aws s3 cp s3://path/SNAPSHOT.jar /mnt/MyScalaImport.jar

On triggering the airflow- It fails at node watch_step enter image description here

Errors what I'm getting are- stdout.gz => enter image description here stderr.gz =>

22/04/08 13:38:23 INFO CodeGenerator: Code generated in 25.5907 ms Exception in thread "main" java.sql.SQLException: No suitable driver at java.sql.DriverManager.getDriver(DriverManager.java:315) at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$driverClass$2(JDBCOptions.scala:108) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.(JDBCOptions.scala:108) at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.(JDBCOptions.scala:38)

How to resolve this issue- I have my jars at-

s3://path/subpath/mssql-jdbc-8.4.1.jre8.jar

s3://path/subpath/postgresql-42.2.24.jar

postgresqlapache-sparkairflowamazon-emrmwaa


Solution 1:[1]

To upload jar files(mssql-jdbc-8.4.1.jre8.jar, postgresql-42.2.24.jar) to airflow local-

In the bootstrap step- 'BootstrapActions': [ { 'Name': 'import custom Jars', 'ScriptBootstrapAction': { 'Path': 's3://path/subpath/copytoolsjar.sh', 'Args': [] } } ]

In copytoolsjar.sh file write the command as- aws s3 cp cp s3://path/SNAPSHOT.jar /mnt/MyScalaImport.jar && bash -c "sudo aws s3 cp s3://path/subpath/mssql-jdbc-8.4.1.jre8.jar /usr/lib/spark/jars/" && bash -c "sudo aws s3 cp s3://path/subpath/postgresql-42.2.24.jar /usr/lib/spark/jars/" Work will be done

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 JaySean

Related Questions