At-most-once, at-least-once and what your broker actually gives you

01The three semantics, precisely

At-most-once means a message is delivered zero or one times: the consumer acknowledges on receipt, before processing. If it crashes mid-processing, the message is gone and nobody will tell you. This is the right choice only when losing a message is genuinely cheaper than processing it twice - high-volume telemetry where a gap is invisible, or a periodic state broadcast where the next one supersedes it anyway.

At-least-once means one or more times: the consumer acknowledges only after processing succeeds. If it crashes before acknowledging, the broker redelivers, and if it crashes after processing but before the acknowledgement lands, you process again. This is what almost every system should use, and it makes duplicate handling the consumer's responsibility rather than an edge case.

02Why exactly-once delivery cannot exist across a network

The consumer processes a message and sends an acknowledgement. The acknowledgement is lost in transit. The broker cannot distinguish 'the consumer never processed it' from 'the consumer processed it and the acknowledgement was lost', because those two states look identical from its side. It must either redeliver, risking a duplicate, or not, risking a loss. There is no third option, and no amount of protocol sophistication produces one.

What systems like Kafka call exactly-once semantics is real, but it is scoped. It works by making the read, the processing output and the offset commit part of one atomic transaction inside the broker, so consume-transform-produce pipelines that stay entirely within Kafka genuinely get exactly-once. The moment your consumer writes to Postgres, calls a payment provider or sends an email, that side effect is outside the transaction and the guarantee no longer covers the part you care about.

03The acknowledgement is where systems break

Acknowledging before processing turns your at-least-once queue into at-most-once without anyone noticing, and a surprising number of client libraries do this by default - auto-commit, auto-ack, or a poll loop that commits offsets on a timer regardless of whether the handler finished. Check the actual default rather than the marketing description; a five-second auto-commit interval means up to five seconds of processed-but-not-really work is silently discarded on every crash.

The opposite failure is holding a message too long. Brokers redeliver after a visibility timeout or session timeout, so a handler that takes longer than that gets its message redelivered while it is still working on it - producing genuine concurrent duplicate processing rather than sequential retries. Either extend the timeout to comfortably exceed your slowest handler, or heartbeat during long work, and monitor how often you approach the limit.

• Disable auto-commit and acknowledge explicitly after the work is durable.
• Set visibility timeout above your p99 handler duration, with margin.
• Heartbeat during long-running handlers rather than raising the timeout globally.
• Never acknowledge in a `finally` block - it acknowledges failures too.

04Idempotent consumers, done properly

The mechanism is a deduplication store: derive a stable message identifier, insert it under a uniqueness constraint, and let the database reject the second attempt. The identifier must come from the producer's business event - an order ID, an event ID assigned at creation - not from the broker's message ID, which changes on redelivery in several brokers and defeats the whole exercise.

The ordering matters more than the store. Record the identifier and perform the effect in the same transaction; if they are separate, a crash between them leaves the effect applied with no record of it, and the redelivery applies it again. When the effect is in an external system that cannot join your transaction - a payment API, an email provider - push the idempotency requirement onto that call by sending it an idempotency key, so the duplicate is absorbed at the far end.

05Ordering guarantees are narrower than people assume

Brokers that guarantee ordering guarantee it per partition, per queue or per message group - never globally. Kafka orders within a partition, so two events for the same order must share a partition key or they can be processed out of sequence. Adding partitions changes the key-to-partition mapping and can reorder events across the boundary, which is a genuinely difficult incident to diagnose after the fact.

Concurrency inside the consumer destroys ordering just as effectively as the broker can. A handler that hands each message to a thread pool has given up sequencing entirely, whatever the broker promised. If order matters, process sequentially per key and parallelise across keys - and if you can design the events to be commutative so order does not matter, do that instead, because it removes an entire class of failure.

• Ordering is per partition or per group, never across the topic.
• Repartitioning reorders across the change; treat it as a migration, not a config tweak.
• Thread pools in the consumer discard ordering regardless of broker guarantees.
• Commutative events beat ordered events - prefer designs that do not need sequencing.

06Dead-letter queues need an owner and a route back

A message that fails repeatedly must eventually stop being retried, or it blocks the partition behind it and burns capacity forever. Retry a bounded number of times with exponential backoff and jitter, then move it to a dead-letter queue with the original payload, the failure reason and the full retry history attached.

The part teams skip is the return path. A dead-letter queue with no alerting, no owner and no replay tooling is a place messages go to be forgotten, and its depth is usually discovered during a customer complaint months later. Alert on non-zero depth, make replay a single documented command, and distinguish poison messages - which will never succeed and need a code fix - from transient failures caused by a downstream outage, which will succeed on replay once the dependency recovers.

07The producer side has the same problem

Publishing has an identical ambiguity: the producer sends, the broker persists, the acknowledgement is lost, the producer retries, and the message is now in the topic twice. Enabling idempotent producer mode where your broker supports it deduplicates within a producer session, which handles the common retry case but not a producer restart.

The harder failure is the dual-write: your service commits to the database and then publishes to the broker, and a crash between the two leaves the database changed with no event published. The fix is the transactional outbox - write the event to an outbox table in the same transaction as the state change, and have a separate relay publish from that table. The relay may publish twice, which is fine, because your consumers are already idempotent.

• Enable idempotent producers where available; understand it is per session.
• Never write to the database and the broker as two independent steps.
• Use a transactional outbox and accept at-least-once publishing from the relay.
• Monitor outbox lag - a stalled relay looks exactly like a quiet system.