Exactly-once is mostly a marketing slide

Kafka delivers at-least-once by default. There is an exactly-once story (the idempotent producer, transactions) and it's real, but it's exactly-once within Kafka. The second your consumer writes to a database, or calls another service, or sends an email, you've stepped outside that transaction and you're back to designing for duplicates.

Our pipeline read messages and wrote computed results to a database. That write is exactly the boundary where the exactly-once guarantee quietly stops applying. So during that rolling restart, a consumer picked up messages it had already processed but hadn't committed the offset for yet, and did the work again. At-least-once, working as designed. Doing exactly what it says on the tin, just not what I wanted.

The move that actually holds up is to make the consumer idempotent. Processing the same message twice should land you in the same place as processing it once. Get that, and at-least-once delivery stops being scary, because a duplicate is just a no-op you paid a little CPU for.

Every message carried a stable id. (If yours don't, derive one from the fields that make it unique.) Before doing the work, I try to record that id in a processed table with a unique constraint. If the insert conflicts, I've already handled this message, so I skip it. Crucially, the dedup row and the actual write go in the same database transaction, so they commit or roll back together.

-- same transaction as the result write
INSERT INTO processed (message_id) VALUES ($1)
ON CONFLICT (message_id) DO NOTHING;

-- 0 rows affected → we've seen this one before → skip the work

The other half is when you commit offsets, and it's easy to get backwards. Commit after your write is durable, never before. If you commit the offset first and then crash, the message is gone for good. That's at-most-once, which is worse than the problem you started with. Process, make it durable, then commit. Crash anywhere in that window and you simply reprocess, which the idempotency now shrugs off.

One more thing that trips people up: ordering only holds inside a partition, and a message only lands on the same partition if you key it consistently. I keyed by the entity id so every event for one record stayed in order on one partition. Get this wrong and your “duplicate” might actually be two related events racing on different partitions, which is a different and nastier bug wearing the same costume.

Some messages are poison. Wrong shape, a bug downstream, a record that references something that no longer exists. Retry a few times with backoff, and if it still won't process, move it to a dead-letter queue and raise an alert. Do not let one bad message wedge the partition while everything behind it waits. I've watched a single malformed record stall an entire pipeline because someone decided infinite retries were “safer.” They are not.

The result was exactly-once effects across partitions, even through the network blips that come free with every rolling deploy. Not because the transport promised it, but because I stopped trusting the transport to and pushed the guarantee down to a database constraint.

Exactly-once delivery is hard and often oversold. Exactly-once effect is a unique index and a bit of discipline. Build the second one.