Mapreduce job when submitted in a queue is not running on the labelled node and instead running under default partition

Question

I have a Hortonworks(HDP 2.4.2) cluster with three data nodes (node1,node2 & node3). I wanted to check the node labelling feature in hortoworks . For that I created a node label(x) and mapped it to a data node node1. So now, there are two partitions:

1. Default partition - contains node2 & node3
2. Partition x - contains node1

Later on I created a queue named "protegrity* and mapped it to node label X.

Now, whene I am running any mapreduce job, this job is getting executed but on the "protegrity* queue of the default partition which I was not expecting. It ws supposed to get executed on queue "protegrity* on node1(labelled partition x). Please refer the attached screenshot of scheduler.

Scheduler The job executed was : hadoop jar ./hadoop-mapreduce/hadoop-mapreduce-examples.jar pi -Dmapred.job.queue.name=protegrity -Dnode_label_expression=x 2 2

The configs of capacity_seheduler.xml file is mentiond below:

<property>
  <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
  <value>0.2</value>
</property>

<property>
  <name>yarn.scheduler.capacity.maximum-applications</name>
  <value>10000</value>
</property>

<property>
  <name>yarn.scheduler.capacity.node-locality-delay</name>
  <value>40</value>
</property>

<property>
  <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
  <value>false</value>
</property>

<property>
  <name>yarn.scheduler.capacity.resource-calculator</name>
  <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.accessible-node-labels</name>
  <value>x</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.accessible-node-labels.x.capacity</name>
  <value>100</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.accessible-node-labels.x.maximum-capacity</name>
  <value>100</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.acl_administer_queue</name>
  <value>yarn</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.capacity</name>
  <value>100</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
  <value>yarn </value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
  <value>yarn</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.default.capacity</name>
  <value>50</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
  <value>100</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.default.state</name>
  <value>RUNNING</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
  <value>1</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.hawqque.acl_administer_queue</name>
  <value>yarn</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.hawqque.acl_submit_applications</name>
  <value>yarn</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.hawqque.capacity</name>
  <value>20</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.hawqque.maximum-capacity</name>
  <value>80</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.hawqque.state</name>
  <value>RUNNING</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.hawqque.user-limit-factor</name>
  <value>2</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.accessible-node-labels</name>
  <value>x</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.accessible-node-labels.x.capacity</name>
  <value>100</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.accessible-node-labels.x.maximum-capacity</name>
  <value>100</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.acl_administer_queue</name>
  <value>yarn </value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.acl_submit_applications</name>
  <value>yarn </value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.capacity</name>
  <value>30</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.maximum-capacity</name>
  <value>60</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.minimum-user-limit-percent</name>
  <value>100</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.ordering-policy</name>
  <value>fifo</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.state</name>
  <value>RUNNING</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.user-limit-factor</name>
  <value>1</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.queues</name>
  <value>default,hawqque,protegrity</value>
</property>

But when I executed an example query given in - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_yarn_resource_mgt/content/using_node_labels.html which doesn't triggers mapreduce job, the job executed on the labelled node and used the mentioned queue.

Query was : sudo su yarn hadoop jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -shell_command "sleep 100000" -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -queue protegrity -node_label_expression x

So, I am a bit confused here, node labelling work for mapreduce jobs or not !!??. If yes, then I need a little help

Answer 1

Node labelling doesn't work with MapReduce job until and unless we add a node labelling to a queue and run the MapReduce job using a queue with a node label. Per your given configuration: Queue protegrity will always use default partition with a capacity of 30 % and not inside the node which you have labeled as x because you need to make your queue protegrity attach to x as default node label. Please add the following configuration to make it work:

<property>  
<name>yarn.scheduler.capacity.root.protegrity.default-node-label-expression</name>
<value>x</value> 
</property>