An asynchronous event based system is one where instead of making synchronous requests to communicate between different microservices, we publish an event to a message broker and hope whatever service next in line receives that event and processes it.

This architecture allows us to scale each service independently with its own local database and also allows for loose coupling between the services. Each service can operate without being aware of any other services. It can just fire and event and forget. Adding and removing services becomes extremely easy.

As you can imagine, this comes with its own challenges. One of the most common challenge faced in distributed event based systems is the problem of ‘Dual Write’.

Dual Write Problem:

The scenario is this. Imagine you have an Order service that creates an order and a separate Payment service that handles the payment for the order.

The order service needs to create an Order entry in its database and it also needs to publish an event through a queue so that the Payment service can start the payment process.

We are dealing with two different systems here. A database and a queue, each with its own transaction boundaries. How do we ensure that if publishing to queue fails, we roll back the database entry creation as well?

Transactional Outbox:

One solution to this is the Transactional Outbox Pattern. Instead of publishing to the queue directly, we’ll have a separate Outbox table in the same database where we will write the event.

A new Order Event Processing service will read from this table and push to the queue. This ensures two things,

  1. Write to both the Order table and Outbox Event table can happen inside a single transaction, since they are part of the same database transaction boundary and hence can be rolled back together.
  2. Event data is not lost if publishing fails due to Event Processing service going down as it will be always available in the Outbox table and we can retry later when the service comes back up.

A sample schema for the Outbox table might look like this,

CREATE TABLE outbox (
    id SERIAL PRIMARY KEY,
    event_type VARCHAR(255) NOT NULL,
    event_payload JSONB NOT NULL,
    status VARCHAR(50) NOT NULL DEFAULT 'PENDING',
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    processed_at TIMESTAMP NULL,
    retry_count INTEGER NOT NULL DEFAULT 0
);
  • id: A unique identifier for each event.
  • event_type: A string to specify the type of event. This helps in identifying what kind of event it is (e.g., OrderCreated).
  • event_payload: A JSONB column to store the actual event data. Using JSONB allows for flexible schema and efficient querying.
  • status: A string to indicate the status of the event. It can be PENDING, PROCESSED, FAILED, etc.
  • created_at: A timestamp to record when the event was created.
  • processed_at: A timestamp to record when the event was successfully processed.
  • retry_count: An integer to keep track of the number of times the event has been retried in case of failures.

Duplicate Messages:

There is still one issue with this implementation. That is, what if the Order Event Processing service itself goes down? There is a chance that the same message might get published twice to the queue.

The solution to this is to add de-duplication of messages at the Payments Service. One way to achieve this is to use the Transactional Inbox Pattern. Similar to the Outbox Pattern we will have a new table in the Payments DB which will store the messages and their timestamp. When the Payment Service receives a message from the queue, it can check if the message is already present in its database. If so, we can ignore it.

Here’s a sample schema for the inbox table:

CREATE TABLE inbox (
    id SERIAL PRIMARY KEY,
    event_id UUID NOT NULL,
    event_type VARCHAR(255) NOT NULL,
    event_payload JSONB NOT NULL,
    received_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(event_id)
);
  • id: A unique identifier for each row in the table.
  • event_id: A UUID to uniquely identify each event. This should be the same UUID used in the outbox table to ensure consistency across services.
  • event_type: A string to specify the type of event. This helps in identifying what kind of event it is (e.g., OrderCreated, PaymentProcessed).
  • event_payload: A JSONB column to store the actual event data.
  • received_at: A timestamp to record when the event was received.

Together, these patterns create a resilient architecture that can gracefully handle the complexities of distributed systems. They enable services to scale independently, maintain loose coupling, and simplify adding or removing services, all while ensuring data consistency and reliability.

“There’s no better guarantee of failure than convincing yourself that success is impossible, and therefore never even trying.”
― Max Tegmark, Our Mathematical Universe: My Quest for the Ultimate Nature of Reality