'Hunter package manager hangs in Python on a processor with large number of cores

Here's some Python code from the Hunter package manager:

#!/usr/bin/env python3

import argparse
import os
import sys
import multiprocessing

parser = argparse.ArgumentParser(description='File link script')

parser.add_argument('--list', required=True)
parser.add_argument('--cellar', required=True)
parser.add_argument('--dest', required=True)

cmd_args = parser.parse_args()

src_list = []
with open(cmd_args.list, 'r') as f:
  for line in f:
    src_list.append(line.strip())

list_len = len(src_list)
proc_num = 32 # multiprocessing.cpu_count()
files_per_job = list_len // proc_num

def job(job_index):
  try:
    begin_ind = files_per_job * job_index
    end_ind = files_per_job * (job_index + 1)
    last_job = (job_index == proc_num - 1)
    if last_job:
      end_ind = list_len
    for i in range(begin_ind, end_ind):
      filename = src_list[i]
      link_from = os.path.join(cmd_args.cellar, filename)
      link_to = os.path.join(cmd_args.dest, filename)
      os.link(link_from, link_to)
    return 0
  except Exception as exc:
    print('Exception caught: {}'.format(exc))
    return 1

def run_link():
  pool = multiprocessing.Pool(processes=proc_num)
  result = pool.map(job, range(proc_num))
  pool.close()
  pool.join()

  if 1 in result:
    sys.exit('Some job failed')

if __name__ == '__main__':
  run_link()

This script hangs in Windows on a processor with 128 logical cores if the following line:

proc_num = 32 # multiprocessing.cpu_count()

is changed into its original

proc_num = multiprocessing.cpu_count()

So for 32 processes, it doesn't hang, but for 128 processes it hangs. What is the reason?

I have a guess that perhaps the pipe on Windows doesn't handle 128 endpoints well.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source