'Multiprocessing.queue - Transfering huge amounts of data between processes
i encountered a rather difficult problem regarding the multiprocessing module. The basic idea is that i have two processes. One gathers info and creates with that a matrix continuously (process: "create_matrix"). These matrices are then sent via multiprocessing.queue to the main process in which calculations on these matrices are performed. Now the problem might come from the sheer number of matrices sent through the queue which is between 100 to 500 per second. The size of the matrices are around 150col to 150rows. Checking the matrices which got through the queue marking them with timestamps entrying and exiting the queue the time delay was constant around 150 ms implying that there was no increasing amount of matrices stuck in the queue. However, using the code below the queue.qsize() showed an ever increasing number even though the queue.empty() was true. E.g. The q.size() = 12.399 and the q.empty() = True. When i change the size of the matices sent to e.g. 5col to 5 rows the q.size() goes up but is then successively worked down by queue.put() and q.empty is only true if q.size is 0 again. So somethings seems to be strange when sending so much data so fast through queues. Am i missing something here? Is there a better method to accomplish this task? Only other possibility i came up with was that i could change the standard queue to LiFo queue and then only get the last matrix and somehow delete the rest. Maybe there is a way to share a matrix between the two processes with one able to write and the other only able to read it?
while True:
'Process to create matrix'
.
.
.
queue_main.put(matrix)
if __name__ == "__main__":
v1 = 5
queue_main = multiprocessing.Queue()
p = multiprocessing.Process(target=create_matrix, args=(v1,queue_main))
p.start()
time.sleep(1) #so that queue is not empty
while True:
print(queue_main.qsize())
print(queue_main.empty())
while not queue_main.empty():
print(queue_main.qsize())
queue_main.get()
p.join()```
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
