Retries and dead letters

A retry allows the handler to run again for the same work. The application selects a retryable or permanent outcome; the framework handles waiting, handoff and settlement of the original delivery. A retry does not keep a handler slot occupied by sleeping.

Choose attempts and delays#

# Fields in the existing orders.created queue entry.
retry_attempts = 3
retry_backoff_millis = 1000
max_retry_backoff_millis = 60000
retry_jitter_ratio = 0.0

retry_attempts=3 means at most three framework retries after the initial attempt. With jitter disabled, delays are 1000, 2000 and 4000 ms. Exponential backoff stops at the configured cap. The default retry count is zero; even retryable errors then normally go directly to the dead-letter destination.

Broker redelivery and repetitions after process crashes are not bounded by those three attempts. Database transaction replay and outbox relay publish retry are separate mechanisms too. Do not interpret them as one shared attempt counter.

Waiting happens in a broker bucket#

orders.created
  -> orders.retry.v2
     -> orders.created.retry.v2.1000ms
     -> orders.created.retry.v2.2000ms
     -> orders.created.retry.v2.4000ms
  -> orders.dlx.v2
     -> orders.created.dlq.v2

These are the default names for the orders exchange and orders.created queue. Retry queues return messages to the main route when their queue-level TTL expires. No retry exchange or buckets are declared when retries are disabled. The v2 in the name is the topology format, not the message’s x-lily-schema-version.

Jitter is compiled in advance too#

retry_jitter_ratio must be finite and lie in 0.0..0.5. A positive ratio produces at most lower/nominal/upper buckets for each nominal delay, clamped to the initial and maximum delays. A fixed hash of the canonical event ID and retry attempt selects a bucket. Identical delays share one physical bucket.

One logical queue may contain at most 32 distinct retry buckets, and the complete topology at most 4096. Queues are not created per message; this does not use arbitrary delays or a delayed-message plugin. Retry retention calculations must include bucket count multiplied by capacity per bucket.

Handoff and overflow behavior#

Retryable failures with attempts remaining use mandatory publishing to a retry bucket; permanent failures or exhausted attempts use the DLQ. The original is not ACKed before broker confirmation. A handoff error, panic or timeout triggers one bounded NACK/requeue attempt within the remaining budget, while preserving the runtime failure. Settlement explains uncertain outcomes.

Framework-managed physical queues use reject-publish overflow. A handoff to a full destination is not reported as success. Broker TTL/dead-letter transfer does not carry an end-to-end exactly-once guarantee across all failure conditions.

Inspecting and replaying the DLQ#

There is no built-in DLQ replay dashboard or automatic replay scheduler. Inspect the cause, repair the data or policy, and make replay an explicit application operation. Preserve the stored event ID when replaying the same event; creating a new UUID changes the transactional deduplication boundary.

A missing x-retry-count means zero; when present it must be an AMQP LongInt in 0..100. String "0" and negative values are not canonical. The canonical publisher and framework handoff manage this metadata; the counter is not an application business-attempt identity.