HiveServer2 is unable to start on hdp

Question

In my project, we are using hdp apache License version 2. All service are running and health is also good but hiveserver2 is not starting. When I try to start, it fail again and again.

hiverserver.log

same ERROR generating again and again.

2022-03-08T09:07:27,690 ERROR [main]: metrics2.CodahaleMetrics (:()) - Unable to instantiate using constructor(MetricRegistry, HiveConf) for reporter org.apache.hadoop.hive.common.metrics.metrics2.Metrics2Reporter from conf HIVE_CODAHALE_METRICS_REPORTER_CLASSES
java.lang.reflect.InvocationTargetException: null
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_171]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_171]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_171]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_171]
    at org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics.initCodahaleMetricsReporterClasses(CodahaleMetrics.java:429) ~[hive-common-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
    at org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics.initReporting(CodahaleMetrics.java:396) ~[hive-common-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
    at org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics.<init>(CodahaleMetrics.java:196) ~[hive-common-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_171]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_171]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_171]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_171]
    at org.apache.hadoop.hive.common.metrics.common.MetricsFactory.init(MetricsFactory.java:42) ~[hive-common-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
    at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:206) ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
    at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1072) ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
    at org.apache.hive.service.server.HiveServer2.access$1700(HiveServer2.java:135) ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
    at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1341) ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
    at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1185) ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
    at org.apache.hadoop.util.RunJar.run(RunJar.java:239) ~[hadoop-common-3.0.0.jar:?]
    at org.apache.hadoop.util.RunJar.main(RunJar.java:153) ~[hadoop-common-3.0.0.jar:?]
Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source hiveserver2 already exists!
    at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152) ~[hadoop-common-3.0.0.jar:?]
    at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125) ~[hadoop-common-3.0.0.jar:?]
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229) ~[hadoop-common-3.0.0.jar:?]
    at com.github.joshelser.dropwizard.metrics.hadoop.HadoopMetrics2Reporter.<init>(HadoopMetrics2Reporter.java:206) ~[dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:?]
    at com.github.joshelser.dropwizard.metrics.hadoop.HadoopMetrics2Reporter.<init>(HadoopMetrics2Reporter.java:62) ~[dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:?]
    at com.github.joshelser.dropwizard.metrics.hadoop.HadoopMetrics2Reporter$Builder.build(HadoopMetrics2Reporter.java:162) ~[dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:?]
    at org.apache.hadoop.hive.common.metrics.metrics2.Metrics2Reporter.<init>(Metrics2Reporter.java:45) ~[hive-common-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
    ... 23 more

TASK Logs

2022-03-09 14:13:34,273 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server xxxxcn001.pin0c06.hcloud.io:2181,xxxxmn001.pin0c06.hcloud.io:2181,xxxxdn001.pin0c06.hcloud.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-03-09 14:13:34,853 - call returned (1, '')
2022-03-09 14:13:34,854 - Will retry 29 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-03-09 14:13:44,865 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server xxxxcn001.pin0c06.hcloud.io:2181,xxxxmn001.pin0c06.hcloud.io:2181,xxxxdn001.pin0c06.hcloud.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-03-09 14:13:45,463 - call returned (1, '')
2022-03-09 14:13:45,465 - Will retry 28 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-03-09 14:13:55,474 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server xxxxcn001.pin0c06.hcloud.io:2181,xxxxmn001.pin0c06.hcloud.io:2181,xxxxdn001.pin0c06.hcloud.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-03-09 14:13:56,050 - call returned (1, '')
2022-03-09 14:13:56,050 - Will retry 27 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-03-09 14:14:06,055 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server xxxxcn001.pin0c06.hcloud.io:2181,xxxxmn001.pin0c06.hcloud.io:2181,xxxxdn001.pin0c06.hcloud.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-03-09 14:14:06,634 - call returned (1, '')
2022-03-09 14:14:06,635 - Will retry 26 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-03-09 14:14:16,646 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server xxxxcn001.pin0c06.hcloud.io:2181,xxxxmn001.pin0c06.hcloud.io:2181,xxxxdn001.pin0c06.hcloud.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-03-09 14:14:17,230 - call returned (1, '')
2022-03-09 14:14:17,231 - Will retry 25 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-03-09 14:14:27,234 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server xxxxcn001.pin0c06.hcloud.io:2181,xxxxmn001.pin0c06.hcloud.io:2181,xxxxdn001.pin0c06.hcloud.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-03-09 14:14:27,824 - call returned (1, '')
2022-03-09 14:14:27,825 - Will retry 24 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-03-09 14:14:37,826 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server xxxxcn001.pin0c06.hcloud.io:2181,xxxxmn001.pin0c06.hcloud.io:2181,xxxxdn001.pin0c06.hcloud.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-03-09 14:14:38,405 - call returned (1, '')
2022-03-09 14:14:38,405 - Will retry 23 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-03-09 14:14:48,415 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server xxxxcn001.pin0c06.hcloud.io:2181,xxxxmn001.pin0c06.hcloud.io:2181,xxxxdn001.pin0c06.hcloud.io:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-03-09 14:14:49,014 - call returned (1, '')