Troubleshooting · Consumer

Diagnose startup, delivery and transaction failures, with four recorded source-document discrepancies.

First determine whether the failure occurs during compilation, startup, delivery, settlement or shutdown. ConsumerError retains typed causes, and a lifecycle failure can preserve both the primary error and secondary cleanup evidence. Do not assume the last visible cleanup error caused the entire failure.

Startup does not complete#

Symptom	What to check
Missing handler or duplicate contract	Linked module, queue name and `(version, content)` key; a configured queue cannot be left without a handler.
Unknown TOML field	An old queue section, a misspelled field or a transactional binding whose feature is disabled.
Retention/topology error	All six retention fields, external DLX/routing key, exclusive=false and derived identity limits.
Connection plan rejected	Explicit use_tls, matching URI scheme and no mixing a URI with separate connection fields.
Transactional preflight error	Feature, backend, single/factory cell, migration and the actual schema/index contract.
MongoDB transactions unsupported	A transaction-capable replica set or sharded deployment instead of standalone MongoDB.

Do not resolve preflight failures by calling runtime __private preparation functions. Apply the public migrator to the correct database cell and retry ordinary startup. Lily does not automatically run migrations at startup; do not attempt to bypass schema checks by editing a version marker manually.

A message arrives but the handler does not run#

Check the canonical event ID, positive schema version and content-kind headers; there is no headerless V1/JSON fallback. Then inspect the queue’s linked version/content contract, payload size, extraction/validation result and guard decision. For a transactional inbox, AlreadyCompleted is a successful deduplication result that deliberately skips business code.

If replicas sharing a queue support different contract sets, RabbitMQ does not select the correct binary by version header. Plan a separate physical queue or binding for an incompatible rollout. Adding another handler to one process’s local dispatch table does not automatically fix the whole cluster.

Retry, duplicate and outbox questions#

Situation	Source-backed interpretation
PostgreSQL duplicate goes straight to the DLQ	`InProgress` uses ordinary retryable handling; check whether retry_attempts=0.
MongoDB handler runs again within one delivery	TransientTransactionError can replay the whole pipeline in a fresh scope.
MongoDB requeue does not increase application retries	Lease contention or ambiguous commit may use special original-requeue paths.
An outbox record exists but is not published	Check destination routing, confirm failures, claim leases, attempt exhaustion and the terminal runtime result.
The same outbox event arrived twice	A crash can occur after confirm but before the delivered mark; downstream deduplication is required.
Policy conflict for the same cell	The complete TransactionalInboxConfig must match, not only the relay fields.

Outbox attempt exhaustion retains the record and propagates terminal relay failure to the Consumer lifecycle. Do not arbitrarily edit SQL or collection fields as if a public retry-reset API existed. Inspect failure evidence and your replay/idempotency policy first. Cleanup applies retention to completed inbox and delivered outbox records.

C01 — Trailing comma in #[queue]#

At reviewed source commit 78ca08b822aa192623c7bcffd90edaf06e4f8e55, the macro rustdoc first example and transactional example show a comma after the final argument. The actual parser rejects it with trailing comma is not accepted in #[queue].

#[queue("orders.created", version = 1, content = "json")]

The examples on this site omit the trailing macro-argument comma. A negative check against the published facade also confirmed the expected compile error for the comma-ending variant. Trailing commas in ordinary Rust function parameters are a separate language rule.

C02 — Deadlines and cancellation order#

QUEUE_MACROS.md and the Consumer configuration guide contain older descriptions of stage cutoffs, a second middleware reserve or cancellation after handler drop. The budget implementation, pipeline and cooperative execution are authoritative.

Current behavior uses one normal-pipeline cutoff and a shared bounded reserve. Cancellation is signaled first, normal execution is polled for a bounded cooperation period, and execution is dropped if necessary. Owner cleanup runs within the remaining deadline; any missing completion evidence is retained in the report. Normal after callbacks do not receive a second independent timeout. The deadline guide follows this model.

C03 — Pending DI rollback budget#

The Consumer README suggests a transition from an independent pending-DI rollback budget to the Consumer budget. The build transaction caps the DI rollback timeout by the Consumer timeout and adopts the existing rollback start into the same attempt root. The current LIFECYCLE contract describes this ownership.

Pending DI rollback therefore does not unconditionally add another 30 seconds to Consumer shutdown. A fallback budget may apply before configuration has been resolved; it does not create a new, unbounded independent cleanup owner. The same attempt retains its bounded receipt and evidence chain.

C04 — Force notification and settlement revocation#

Queue ARCHITECTURE suggests settlement authority is revoked immediately when the force reserve begins. In the implementation, force_drain first signals execution and closes admission; authority revocation and abort occur after the final forced cutoff.

A healthy channel can settle during the cooperative window. Immediate loss of authority for an old generation after transport failure is a different situation. Force notification, final revocation and confirmed resource termination should not be reported as one event. The shutdown and settlement guides preserve that distinction.

Interpret diagnostic evidence correctly#

These four records describe direct differences between documentation and source at the reviewed commit; this documentation work did not modify framework files. Recheck them against source when moving to a new version. These pages follow the source rather than assuming older prose describes current behavior.

An aggregate snapshot describes liveness/readiness, while a shutdown report records action and termination evidence. Read them alongside application error codes and bounded logs. Do not silently add payloads, broker URIs or terminal-observation event ID lists to a snapshot; your application chooses the diagnostic surface it exposes.