Queue.Queue vs semaphores, locks, etc. in multithreaded Python code

The principal challenge of multi-threaded applications is coordinating threads that share data or other resources. To that end, the threading module provides a number of synchronization primitives including locks, events, condition variables, and semaphores.

While those tools are powerful, minor design er开发者_C百科rors can result in problems that are difficult to reproduce. So, the preferred approach to task coordination is to concentrate all access to a resource in a single thread and then use the Queue module to feed that thread with requests from other threads. Applications using Queue.Queue objects for inter-thread communication and coordination are easier to design, more readable, and more reliable.

It, basically, states to use Queue.Queue for inter-thread communication and coordination, instead of the powerful tools such as semaphores, locks, etc.

My question is, what's the drawback of the suggested method? When should one use the more "powerful tools" instead, and why?

Edit

To be clear, I know what semaphores are. I was just wondering why the Python documentation suggests to use the Queue.Queue method instead of the "powerful tools" -- I'm simply using the documentation's own verbiage, not coming up with my own.

---

I'm not sure I'd consider semaphores and locks "more powerful methods", as you suggest.

Queues are generally a higher-order abstraction. In other words, you could use semaphores and locks to build thread-safe queues.

Which you'd use where depends on your application. Queues are good for passing "work" between threads and processes, and semaphores/locks are good for protecting critical sections or shared resources, so only one thread can access at a time.

---

Take a look at the source code for Python's thread-safe queue. The queue class builds a useful abstraction from 3 Conditions and a Lock, correctly.

I wouldn't say coordination is the hardest problem. In shared-state multithreading the hardest thing is preventing threads from "sharing". You always have to look out for non-deterministic behaviour due to threads accidentally sharing and stomping on each other's data.

So, I recommend you don't use threads at all. You should use the lower-level tools when you feel you haven't spent enough time tracking down heisenbugs, but if there's any way you can get away with using a simple queue, go for it.