Event-driven architecture moves records of things that happened between independently deployed components. That decoupling can make systems responsive and extensible, but it also exposes the ordinary failure of networks, processes, and acknowledgements. A sender may not know whether a receiver acted before a timeout. Reliable design accepts that ambiguity early and specifies what each event means, who owns its effects, and how repetition is handled.
This article is part of the software engineering technologies guide library.
Name the delivery promise precisely
Delivery language needs a boundary. “At least once” generally permits an event to arrive more than once, so consumers must tolerate repetition. “At most once” accepts possible loss to avoid redelivery. Stronger claims may apply only to a producer, a broker log, a transaction, or a particular read-process-write path. State the components and failure cases that a guarantee actually covers.
An event is an immutable record that something occurred; a command is a request to perform an action. Keeping the distinction clear helps teams select safe handling. A duplicated notification event and a duplicated request to charge an account do not have the same consequence. Include an event identifier, producer identity where useful, occurrence time, schema version, and a stable business key when the domain requires one.
Choose an ordering scope that matches the work
Ordering is rarely global in a distributed stream. Many systems preserve append order within a partition or comparable shard, while parallel processing across partitions intentionally trades a single sequence for throughput. Decide which entity truly needs order: perhaps one account, device, order, or conversation. Route events for that entity consistently and document that order does not extend beyond the chosen scope.
Even within an ordered stream, retries and parallel side effects can create apparent reordering in downstream systems. A consumer should not silently assume that arrival time equals business time. Carry sequence information or version numbers when the domain needs it, and define a response to an older event. Some cases require buffering; others can reject stale transitions. The correct choice depends on the cost of waiting and the domain rule.
Treat retries as normal execution
Retries recover from transient conditions such as unavailable dependencies, rate limits, or restarts. They also repeat work after an acknowledgement is lost. Design a bounded policy that records the reason, attempts, and next action. Backoff spaces attempts over time, and jitter varies delays so many workers do not retry in lockstep. Neither mechanism decides whether repeating the business effect is safe.
Classify failures before retrying indefinitely. A malformed payload or a violated invariant usually needs investigation rather than another identical attempt. A durable quarantine or dead-letter process can isolate such records while preserving diagnostic context. It should have an owner and a defined reprocessing procedure. Leaving failed events invisible merely transfers uncertainty to customers and on-call teams.
Make the consumer effect idempotent
Idempotency means that processing the same logical request more than once produces the intended final effect, rather than multiplying it. It is a property of the operation and its state boundary, not just of the message broker. Common approaches include storing a processed-event key with the resulting state, using a naturally unique business key, or applying a versioned state transition conditionally.
Choose the deduplication key deliberately. A transport identifier may change when a producer rebuilds an event, whereas a business operation identifier can express the user-visible action. Retain deduplication records long enough for the retry and replay horizon that matters. A short retention period can make a later replay dangerous; unlimited retention may create a growing operational burden. Make that trade-off explicit.
Align state changes and external effects
Atomicity is easiest inside one transactional boundary. A database update and a published event can otherwise disagree if one succeeds and the process stops before the other. An outbox pattern stores the intended event with the state change, then publishes it asynchronously with deduplication-aware handling. This does not erase retries; it gives recovery a durable source of truth.
External effects such as email, payments, or third-party API calls require their own idempotency and reconciliation design. Propagate a stable operation key where the receiving interface supports it, and record requests and outcomes sufficiently to resolve uncertainty. Do not claim end-to-end exactly-once behavior merely because one layer supports idempotent production. The system boundary includes every effect that matters to people.
Practice recovery as an operational capability
Runbooks should explain how to pause a consumer, inspect a failed record, correct a known issue, and reprocess safely. Operators need visibility into lag, retry volume, quarantine depth, and duplicate suppression without exposing sensitive payloads. Alert thresholds should distinguish a brief transient burst from a sustained inability to make progress. This converts delivery semantics into actions rather than architecture diagrams.
Keep replay separate from casual retry. Replaying a retained history can be valuable for rebuilding a projection or repairing a defect, but it deliberately revisits old records. Specify which consumers support it, what state is reset, and how downstream effects are guarded. Review these assumptions whenever schemas or business rules change. A recovery plan earns trust only if it respects the idempotency and ordering limits it declares.
Source notes
Reporting record
techduopulse stores source destinations privately. Public notes remain non-clickable so every visitor journey stays on this website.
Apache Kafka documentation
Primary source · Topics, partitions, and orderingApache Kafka producer documentation
Primary source · Idempotent and transactional producer behaviorImage updated: embedded writing removed; article content and factual claims unchanged.
