If you don't understand this concept, "eventually" you will be in trouble.
How to achieve Exactly-Once delivery?
One of the most serious problems a distributed system can have is duplicate or missed operations, and Exactly-Once delivery solves that problem.
At first glance, Exactly-Once delivery seems hard to tackle, but dividing the problem into two parts makes it much easier to solve.
An operation is executed Exactly-Once if:
• It is executed At-Least-Once.
• It is executed At-Most-Once.
At-Least-Once Delivery
An operation is executed at least once if, even in the presence of failures, the system retries it until it succeeds.
How?
1. Retry until the operation is processed or reaches maximum attempts. Note that not all operations are retryable.
2. Store the message and its state in a durable message queue to ensure it is not lost and can be retried.
3. Set up monitoring to detect and alert if retries exceed a threshold. It could be potential issues needing attention.
At-Most-Once Delivery
If the system prevents the same message from being processed more than once, even if it is retried, an operation is executed at most once.
How?
1. Idempotency is your friend. An idempotent operation can be applied multiple times without changing the result beyond the initial application.
2. Generate a unique idempotency key for each message/operation.
3. Before processing the message, check if the idempotency key already exists in a persistent store:
- If it exists, do not process the message again and return the previous result.
- If it does not exist, process the message and store the key.
Resilient and reliable systems need Exactly-Once delivery. Don't double charge your customers; 😉