Idempotent Consumers

Today we will talk about idempotent consumers. The idempotent consumer pattern is used to ignore duplicate messages. Let’s first give a brief introduction about what consumer and idempotency mean before going deeper.

Event-driven architecture consists of events, message brokers, producers, and consumers. Producer generates events, send events through message broker, and consumers consume these events from message brokers. An event may occur when there is a state change in data or an update.

Each message broker has different features while storing and streaming events. Messages could be delivered with “at-least once”, “at-most once”, or “exactly once” delivery strategies. At least once delivery is a good option in most cases, but it comes with some trade-offs.

At-least once means you never lose a message, but messages might be delivered more than once. So our consumers can consume these events repeatedly. It is not a good idea to charge a customer twice, right??

How can we protect our consumers from multiple consuming the same event? This is where idempotency helps.

Idempotency topic is not just related to consumers. It depends on your context. It could be a web service, a function, or anything. Here is the Wikipedia definition:

Idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application.

Let’s dive into possible solutions for that problem.

Storing processed event ids

We can make the consumer idempotent by storing unique id of each message. When processing this event, the consumer checks whether the id exists or not.

We can implement this logic with a transaction by trying to insert the processed event table first, then if the insert is successful, process other query. If message id already exists, transaction will fail.

Storing event revision count on the business entities

If we don’t have the capability to have another processed event table(or bucket, index, etc. depending on the solution) we can have a field that represent the version of entity and update that version on each document update.

There is a disadvantage of this solution. Let’s say you missed an event that has version 5678. Then consumer consumed the event that has version 5679. This event must have the latest version of the data. Otherwise, you can lose the consistency of data because of missing event 5678.

Consumers may consume same event repeatedly. We should be ready for that problem in event driven systems.