'Python Threading Worker Closure

I'm writing a script that contains multiple threads interconnected with queues. The lowest level thread is taking data from a UDP port. The other threads do various levels of processing on that data e.g.

    Thread 3: Process application data

            Thread 2: Process layer 4 packet headers

                    Thread 1: Listen for UDP packets

I get that normally you should use .task_done() to indicate that the work has been completed and that .join() will wait until that all tasks are done. However, I need to start these threads before the UDP data is being sent. So what should I do in the case that the queues will remain empty for some time before UDP data is received?

I think I have a couple of options:

  1. The UDP listener (Thread 1) has a timeout. at the end of the timeout I could put something that signifies the end in the queue, e.g. udp_queue.put(-1). Then the following threads can add further -1s to the other queues. Is there something better to use than a -1?
  2. Implement timeouts that increment on Queue.Empty() which are larger than the time out in the UDP listener (Thread 1).

I assume option 1 is the preference? My concern is that if I have some valid data in a queue that happens to equal -1 then I could end up closing the threads unintentionally?

Any thoughts or pointer appreciated.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source