The queue module implements multi-producer, multi-consumer queues that are thread-safe, meaning multiple threads can add and remove items without corrupting the queue or needing explicit locks. It provides FIFO (Queue), LIFO (LifoQueue), and priority (PriorityQueue) variants.
Safe hand-off between threads
What you will learn: why a thread-safe queue beats a shared list, and the basic put/get operations including the important blocking behaviour of get.
How to read this tab: get() on an empty queue blocks until an item arrives — that blocking is the feature that powers producer/consumer.
When several threads need to hand work to each other, sharing a plain list is dangerous — two threads modifying it at once can corrupt it. A queue.Queue is built to be used by many threads simultaneously and safely. One thread puts items in; another takes them out; the queue handles all the locking.
get() blocks by default — and that is the point. Calling get() on an empty queue waits until an item arrives; it does not return immediately or raise. This lets a consumer thread sit on get() and wake the instant work appears. Use get_nowait() or block=False to raise queue.Empty instead.
Why a thread-safe queue
The queue module's queues handle all the locking internally, so threads can put and get freely without stepping on each other. This is the recommended way to communicate between threads in Python.
import queue
# A FIFO queue — first in, first out
q = queue.Queue()
# Add items (put) — thread-safe
q.put("first")
q.put("second")
q.put("third")
# Remove items (get) — thread-safe, returns in FIFO order
print(q.get()) # "first"
print(q.get()) # "second"
# Check size and emptiness (approximate in multithreaded use)
print(q.qsize()) # 1
print(q.empty()) # False
# Optional maximum size — put() blocks when the queue is full
limited = queue.Queue(maxsize=10)
limited.put("item") # blocks if 10 items already waiting
# get() on an empty queue BLOCKS until an item is available —
# this is the key behaviour that makes producer/consumer work.Calling q.get() on an empty queue waits until something is available — it does not return immediately or raise. This blocking is the feature: a consumer thread can sit on get() and wake up the moment work arrives. Pass block=False or use get_nowait() to raise queue.Empty instead of waiting.
A sentinel value signals "no more work". Put a special marker (often None) to tell consumers to stop. The queue safely passes items from producers to consumers with no manual locks, even with multiple producers and consumers running at once.
The producer-consumer pattern
import queue
import threading
import time
work_queue = queue.Queue()
def producer():
for i in range(5):
item = f"job-{i}"
work_queue.put(item)
print(f"Produced {item}")
time.sleep(0.1)
work_queue.put(None) # sentinel: signal "no more work"
def consumer():
while True:
item = work_queue.get() # blocks until an item is available
if item is None: # sentinel received -> stop
break
print(f" Consumed {item}")
time.sleep(0.2)
# Run producer and consumer on separate threads
t1 = threading.Thread(target=producer)
t2 = threading.Thread(target=consumer)
t1.start(); t2.start()
t1.join(); t2.join()
print("All work done")
# The queue safely passes items from producer to consumer with no
# manual locks — even if you ran several producers and consumers.✅ Beginner tab complete
- I know queue.Queue is thread-safe; a plain list is not
- I can put and get items safely from multiple threads
- I know get() blocks on an empty queue by default
- I can use get_nowait() to raise queue.Empty instead of waiting
Variants, join, and task_done
What you will learn: LifoQueue and PriorityQueue, and coordinating completion with task_done and join.
How to read this tab: When using join(), every get() must be matched by a task_done() — use try/finally to guarantee it.
PriorityQueue returns the lowest value first — add a tiebreaker. Use (priority, data) tuples; lower number = higher priority. When priorities tie, Python compares the data, so include a counter — (priority, count, data) — when the data is not comparable.
Queue variants: LIFO and priority
Beyond the standard FIFO queue, the module offers a stack-like LIFO queue and a priority queue that always returns the smallest item first.
import queue
# LifoQueue — last in, first out (a thread-safe stack)
stack = queue.LifoQueue()
stack.put(1); stack.put(2); stack.put(3)
print(stack.get()) # 3 (most recent first)
print(stack.get()) # 2
# PriorityQueue — always returns the LOWEST value first
pq = queue.PriorityQueue()
pq.put((2, "medium priority"))
pq.put((1, "high priority")) # lower number = higher priority
pq.put((3, "low priority"))
print(pq.get()) # (1, 'high priority')
print(pq.get()) # (2, 'medium priority')
# Common pattern: (priority, data) tuples. If priorities tie, Python
# compares the data, so include a tiebreaker (e.g. a counter) when the
# data isn't comparable:
import itertools
counter = itertools.count()
pq = queue.PriorityQueue()
pq.put((1, next(counter), {"task": "a"})) # (priority, seq, data)
pq.put((1, next(counter), {"task": "b"})) # seq breaks the tie safely
print(pq.get()[2]) # {'task': 'a'} — first inserted at this priorityEvery get() needs a matching task_done() when using join(). put() increments a counter, task_done() decrements it, and join() blocks until it hits zero. Miss a task_done() and join() blocks forever; call it too many times and you get a ValueError. Use try/finally to guarantee the call.
Coordinating completion with join and task_done
import queue
import threading
q = queue.Queue()
def worker():
while True:
item = q.get() # get the next task
try:
print(f"Processing {item}")
# ... do the work ...
finally:
q.task_done() # signal THIS task is complete
# Start a pool of daemon worker threads
for _ in range(3):
threading.Thread(target=worker, daemon=True).start()
# Enqueue 10 tasks
for item in range(10):
q.put(item)
# Block until EVERY enqueued task has had task_done() called
q.join() # waits for the count to reach zero
print("All tasks completed")
# How it works: put() increments an internal counter; task_done()
# decrements it; join() blocks until the counter hits zero. This lets
# the main thread wait for all work to finish without a sentinel.When using q.join(), each item you get() must be followed by a task_done() call (use a try/finally to guarantee it). Miss one and join() blocks forever; call it too many times and you get a ValueError.
✅ Intermediate tab complete
- I can use LifoQueue (stack) and PriorityQueue (lowest first)
- I use (priority, counter, data) tuples to break priority ties
- I call task_done() after each get() when using join()
- I know join() blocks until every task is marked done
Blocking semantics, SimpleQueue, shutdown
What you will learn: block/timeout control, SimpleQueue for signal-safe use, the 3.13 shutdown method, and the relationship to multiprocessing and asyncio queues.
How to read this tab: Read to master non-blocking patterns and the modern shutdown-based producer/consumer.
Blocking semantics, SimpleQueue, and the shutdown method
Every get and put accepts block and timeout parameters giving precise control: get(block=False) (equivalently get_nowait()) raises queue.Empty immediately if nothing is available, while get(timeout=2) waits up to two seconds before raising. The symmetric behaviour applies to put on a bounded queue, raising queue.Full. These let you build responsive workers that poll, time out, or fall back rather than blocking indefinitely.
import queue
q = queue.Queue(maxsize=5)
# Non-blocking and timed operations
try:
q.put("x", timeout=1.0) # wait up to 1s if full, else queue.Full
item = q.get(timeout=2.0) # wait up to 2s if empty, else queue.Empty
except queue.Full:
print("queue was full")
except queue.Empty:
print("nothing available in time")
# SimpleQueue (3.7+) — a simpler, unbounded FIFO with fewer features
# but stronger guarantees: its put() never blocks and is reentrant-safe,
# making it usable from signal handlers and __del__ methods.
sq = queue.SimpleQueue()
sq.put("safe even from a signal handler")
print(sq.get())
# Queue.shutdown (3.13+) — cleanly stop a queue
# After shutdown(), put() raises ShutDownError; get() drains remaining
# items then raises ShutDownError, so worker loops exit without sentinels.
# q2 = queue.Queue()
# q2.shutdown() # signal no more work; replaces sentinels
# Subclassing to customise the underlying container — override _put,
# _get, _init. PriorityQueue and LifoQueue are themselves just Queue
# subclasses overriding these three methods over a list/heap.
class MaxPriorityQueue(queue.PriorityQueue):
def _put(self, item):
# invert priority to pop the LARGEST first
super()._put((-item[0], item[1]))
# Thread-safety note: qsize(), empty(), and full() are reliable at the
# instant called but may be stale immediately after in a multithreaded
# program. Never gate logic on them; rely on blocking get/put or the
# Empty/Full exceptions instead.A subtle but important design point distinguishes queue.Queue from queue.SimpleQueue: the former supports the full task_done/join task-tracking protocol and bounded sizes, while the latter is a minimal, unbounded FIFO whose put is guaranteed never to block and is safe to call from contexts like signal handlers and finalizers where the more complex queue could deadlock. The newer shutdown method (Python 3.13) modernises the producer-consumer pattern by replacing the traditional sentinel-value approach — instead of putting None markers to tell workers to stop, you call shutdown() and worker get() calls raise ShutDownError once the queue drains, giving a cleaner and less error-prone shutdown. For the relationship to other tools: queue is for threads, multiprocessing.Queue and multiprocessing.Manager().Queue() for processes, and asyncio.Queue for coroutines — each matching its concurrency model, all sharing the same conceptual put/get interface.
✅ Expert tab complete
- I can use block and timeout to control get/put waiting
- I know SimpleQueue is signal-safe and never blocks on put
- I know shutdown() (3.13) replaces sentinel values for clean stops
- I know queue is for threads; multiprocessing.Queue for processes