Concurrency and backpressure · Consumer

Size capacity by treating concurrency, prefetch, process buffers and broker retention as separate bounds.

Consumer capacity is not determined by one number. Unacknowledged broker deliveries, deliveries waiting inside the process and concurrently running work consume different resources. Each physical queue has its own plan, and replica count further increases total capacity.

Consider the three controls together#

# Fields in an existing queue entry, before its retention table.
concurrency = 4
prefetch_count = 10
delivery_buffer_capacity = 6
max_message_size_bytes = 1048576

Field	Effect	Bound
`concurrency`	Concurrent execution tasks/permits in the queue dispatcher	1–1024; default 1
`prefetch_count`	RabbitMQ QoS unacked window for the subscription	1–4096; default 10
`delivery_buffer_capacity`	Bounded buffer between receiver and dispatcher	1–prefetch; defaults to prefetch
`max_message_size_bytes`	Body admission limit	1 byte–16 MiB; default 1 MiB

This example defines four execution slots and a broker window of at most ten unacked messages; it does not open four subscriptions. Prefetch is not secretly multiplied by concurrency. Buffer size is neither broker queue retention nor an upper bound on all process memory.

Where does backpressure occur?#

When execution slots are full, the dispatcher waits before starting another delivery. A semaphore wakeup alone is not enough to admit new work; the dispatcher also observes completed task results. On terminal runtime failure it reconciles existing ownership before starting another handler.

After losing a broker stream, recovery first waits for termination evidence from the old subscription generation’s tasks and channels, then creates a new subscription. This prevents silently accumulating a second active owner behind a broken connection; it is not a promise of uninterrupted processing during network failures.

Body limits and retention#

An oversized body never enters the normal pipeline and takes a non-requeueing broker NACK/dead-letter path before application cloning or decoding. This is different evidence from a confirmed application DLQ handoff. The transport may already have received the body when its size is checked; the limit is not a process RSS quota.

Broker ready-message capacity is bounded by retention. Include retry bucket count: every bucket has its own retention budget. Average body size, expansion during decoding, external connection pools and processing time all contribute to overall capacity.

Ordering and throughput expectations#

concurrency=10 does not mean ten messages per second. Multiple slots, replicas, retries and broker redelivery can change completion order. The single-active-consumer option does not automatically reduce application concurrency to one. Design the data model and partitioning around any strict domain ordering requirements.

Choose production settings using actual processing times and operational snapshots. Do not assume higher prefetch improves every workload; more accepted work also affects shutdown and memory requirements.