'Is there an option to print Hive SQL runtime log while executing through SSH
I had a recent Hive version upgrade from 1.2.1 to 2.3.0, along with a movement from HDP-EC2 to EMR.
Below is the shell script format using which i execute all the queries,
ssh -T -i key.pem [email protected] <<EOF
hive <<EO1
<Hive-Query>
EO1
EOF
In version 1.2.1, the running shell script used to stdout runtime log including the query being executed and the running mapper/reducer counts in realtime.
In EMR with hive version 2.3.0, with exactly the same script, log has nothing except query complete notification.
Expected Log
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id application_********_0373)
Map 1: -/- Reducer 2: 0/36
Map 1: 0/191 Reducer 2: 0/36
Map 1: 0(+1)/191 Reducer 2: 0/36
Map 1: 0(+22)/191 Reducer 2: 0/36
Map 1: 0(+44)/191 Reducer 2: 0/36
Map 1: 0(+62)/191 Reducer 2: 0/36
Map 1: 1(+61)/191 Reducer 2: 0/36
Map 1: 2(+61)/191 Reducer 2: 0/36
Map 1: 2(+62)/191 Reducer 2: 0/36
.
.
.
.
.
Moving data to directory hdfs://ip-***************
Current Log in EMR
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id application_1592149437885_0947)
Moving data to directory hdfs://ip-***************
Any pointers on what change would have caused the log to disappear?
Note : log is there when executing the query directly in cluster. But when executed through ssh, log is missing.
Solution 1:[1]
You can use ssh -t or ssh -tt to run your command remotely.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | fmx |