'Multiple Producer Single Consumer Multiprocessing Priority Queue

I am new to writing on StackExchange so thanks in advance for any replies!

I have the following set up

  1. Multiple processes, each of which is listening to a data stream.
  2. One final end consumer(on a different process to the producers).
  3. I want all of the producers to intake their data stream, do some stuff, and then pass on a result to the single consumer.
  4. The result they pass on to the consumer also contains a priority with which that information should be picked up by the consumer.

What would be perfect is a priority queue in the multiprocessing library, with all producers putting items in this queue and the one consumer getting items. This does not exist.

The final criteria is that this message passing has to be fast - really fast.

Currently the set up I have is that all producers and consumers run on the same process and I simply use a queue.PriorotyQueue. However, due to the number of producers, the consumer is just getting blocked due to the work being done by the producers.

Any ideas? Currently running the latest python also FYI.

Thanks!



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source