← Back to Blog

When Microservices Fail: Sagas, Outbox, and Recovery

By Yasiru Samarasekara · 8 Sept 2026

A business operation can look like a single action to a user while spanning several independently managed services behind the scenes. Consider a checkout flow:

%%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 24, "diagramPadding": 4}}}%%
flowchart LR
    customer(["Customer"])
    order["Order Service"]
    inventory["Inventory Service"]
    payment["Payment Service"]

    customer -->|"  Place Order  "| order
    order -->|"  OrderCreated  "| inventory
    inventory -->|"  InventoryReserved  "| payment
    payment -->|"  PaymentCompleted  "| order

From the customer’s perspective, this is one operation: place an order. From the system’s perspective, it may involve several services, several databases, and several independently committed transactions. That difference is where consistency becomes difficult. In a monolith backed by a single database, related changes can often participate in the same database transaction. If something fails before the transaction commits, the whole operation can be rolled back. Microservices change that model. Each service normally owns its own state and commits changes independently.

%%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 24, "diagramPadding": 4}}}%%
flowchart LR
    order["Order Service"] --> orderDb[("Order DB")]
    inventory["Inventory Service"] --> inventoryDb[("Inventory DB")]
    payment["Payment Service"] --> paymentDb[("Payment DB")]

The technical transaction boundaries no longer match the business transaction boundary. So the problem becomes:

How do we keep the overall business operation consistent when individual services succeed or fail independently?

The Happy Path Is Easy

Read the numbered actions in order. Each box is a service; the arrows show the events that trigger the next step. The flow returns to the Order Service for confirmation:

%%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 24, "diagramPadding": 4}}}%%
flowchart LR
    order["Order Service<br/>1. Create order<br/>4. Confirm order"]
    inventory["Inventory Service<br/>2. Reserve stock"]
    payment["Payment Service<br/>3. Charge customer"]

    order -->|"&nbsp;&nbsp;OrderCreated&nbsp;&nbsp;"| inventory
    inventory -->|"&nbsp;&nbsp;InventoryReserved&nbsp;&nbsp;"| payment
    payment -->|"&nbsp;&nbsp;PaymentCompleted&nbsp;&nbsp;"| order

When everything succeeds, the final state may look like this:

The interesting part begins when only some of those steps succeed.

Partial Failure

Suppose the same order sequence fails at the payment step:

%%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 24, "diagramPadding": 4}}}%%
flowchart LR
    order["Order Service<br/>1. Create order"]
    inventory["Inventory Service<br/>2. Reserve stock"]
    payment["Payment Service<br/>3. Charge fails"]

    order -->|"&nbsp;&nbsp;OrderCreated&nbsp;&nbsp;"| inventory
    inventory -->|"&nbsp;&nbsp;InventoryReserved&nbsp;&nbsp;"| payment

Payment never produces PaymentCompleted, so the Order Service cannot proceed to confirmation. The Order Service may now contain a pending order, while the Inventory Service has already committed the reservation.

There is no single database transaction covering all three services. The Inventory Service has already committed its own local transaction. The Order Service cannot simply issue a database rollback and undo what happened elsewhere. This is a partial failure: part of the distributed operation succeeded while another part failed. Distributed systems need an explicit strategy for dealing with that state.

Rollback Becomes Compensation

Inside a local database transaction, rollback is straightforward.

None of those changes become committed state. Across services, the situation is different.

%%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 24, "diagramPadding": 4}}}%%
flowchart LR
    inventory["Inventory Service<br/>Reservation committed"]
    payment["Payment Service<br/>Payment fails"]

    inventory -->|"&nbsp;&nbsp;InventoryReserved&nbsp;&nbsp;"| payment

The inventory reservation is already committed. To reverse its business effect, the system needs another operation:

%%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 24, "diagramPadding": 4}}}%%
flowchart LR
    inventory["Inventory Service"]
    payment["Payment Service"]
    inventory -->|"  1. InventoryReserved  "| payment
    payment -->|"  2. PaymentFailed  "| inventory

The Inventory Service first commits the reservation. After receiving PaymentFailed, it releases the inventory in a new local transaction. That second operation is a compensating transaction. Compensation is not the same as rollback. A rollback prevents an uncommitted change from becoming permanent. A compensation performs a new business operation intended to counteract an operation that has already committed. Examples include:

The distinction matters because compensation is often domain-specific. A refund does not erase the original payment. It creates another financial operation that reverses its effect. Some actions may not even have a perfect inverse. An email cannot be unsent. A notification may already have been seen. An external system may have performed an irreversible action. For that reason, compensating behaviour needs to be designed as part of the business workflow rather than treated as a generic technical rollback.

Sagas: A Distributed Business Transaction

A Saga models a distributed business transaction as a sequence of local transactions. Each service commits its own state independently. When a later step fails, previously completed steps can be compensated. For the order example:

The happy path is simple:

If payment fails, the Saga follows a different path:

The Saga does not create one large ACID transaction across the services. Instead, it defines how independently committed operations collectively complete or recover. There are two common ways to coordinate that workflow: choreography and orchestration.

Saga Choreography

With choreography, there is no central component controlling the entire process. Services react to events produced by other services.

%%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 24, "diagramPadding": 4}}}%%
flowchart LR
    order["Order Service"]
    inventory["Inventory Service"]
    payment["Payment Service"]

    order -->|"  OrderCreated  "| inventory
    inventory -->|"  InventoryReserved  "| payment
    payment -->|"  PaymentCompleted  "| order

The failure path can also be event-driven.

%%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 24, "diagramPadding": 4}}}%%
flowchart LR
    payment["Payment Service"]
    inventory["Inventory Service"]
    order["Order Service"]

    payment -->|"  PaymentFailed  "| inventory
    inventory -->|"  InventoryReleased  "| order

This can work well for smaller workflows because services remain loosely coupled and there is no central workflow coordinator. As the workflow grows, however, its rules are spread across multiple services: each service decides what to do when it receives an event, while no single component necessarily tracks the entire order process.

Choreography therefore needs deliberate monitoring and recovery rules to make the overall workflow visible. An explicit orchestrator can make this easier by tracking completed steps and deciding which action or compensation should happen next.

Saga Orchestration

With orchestration, a coordinator controls the sequence of operations.

%%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 24, "diagramPadding": 4}}}%%
flowchart LR
    saga["Order Saga Orchestrator"]
    order["Order Service"]
    inventory["Inventory Service"]
    payment["Payment Service"]

    saga -->|"  Create Order  "| order
    saga -->|"  Reserve Inventory  "| inventory
    saga -->|"  Charge Payment  "| payment

The orchestrator can maintain the workflow state and decide what should happen next. For example:

If payment fails:

A useful distinction is:

Choreography is primarily based on services reacting to events describing what has happened.

Orchestration introduces an explicit component responsible for directing the business workflow.

Neither approach is universally better. Choreography can keep simple workflows lightweight, while orchestration can make complex workflows, retries, state transitions, and compensation easier to observe and control.

What If Compensation Fails?

Compensation is itself a distributed operation, so it can fail too. Suppose payment fails and the system attempts to release inventory:

The system is still inconsistent. That means compensating actions need many of the same reliability properties as normal operations:

• retryability
• idempotency
• persistent workflow state
• monitoring
• clear recovery states
• escalation when automatic recovery cannot complete

A Saga may therefore remain in an intermediate state such as COMPENSATING while the system continues attempting recovery. This leads to an important shift in how consistency is viewed in distributed systems. Consistency does not always mean that every service changes atomically at the same instant. Often it means that the system has a reliable path from an intermediate state toward a valid business state.

Sagas Do Not Solve Reliable Event Publishing

Suppose the Inventory Service successfully reserves stock and then publishes an event. Conceptually:

There is a failure window between the database commit and the message publication.

The Inventory Service now believes the stock is reserved, but the next participant in the workflow may never receive InventoryReserved. Reversing the order does not solve the problem:

Now another service may react to an inventory reservation that does not actually exist. The service is attempting to update two independent systems:

%%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 24, "diagramPadding": 4}}}%%
flowchart LR
    inventory["Inventory Service"]
    database[("Inventory DB")]
    kafka["Kafka"]

    inventory -->|"  Write State  "| database
    inventory -->|"  Publish Event  "| kafka

This is commonly called the dual-write problem. The local database transaction cannot normally make both the database change and the Kafka publication atomic.

Transactional Outbox

The Transactional Outbox pattern changes how the event is produced. Instead of updating the business state and immediately publishing to Kafka, the service writes both the business change and a representation of the event into the same database transaction. For example:

BEGIN;

UPDATE inventory
SET status = 'RESERVED'
WHERE item_id = 123;

INSERT INTO outbox (
    event_id,
    event_type,
    payload,
    status
)
VALUES (
    'evt-7821',
    'InventoryReserved',
    '{ ... }',
    'NEW'
);

COMMIT;

Because both writes use the same database transaction, they succeed or fail together.

Kafka is not contacted during this transaction. A separate outbox publisher is responsible for reading pending records and publishing them.

%%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 24, "diagramPadding": 4}}}%%
flowchart LR
    service["Inventory Service"]
    database[("Inventory DB<br/>Inventory + Outbox")]
    publisher["Outbox Publisher"]
    kafka["Kafka"]

    service -->|"&nbsp;&nbsp;Single DB Transaction&nbsp;&nbsp;"| database
    database -->|"&nbsp;&nbsp;Read Pending Events&nbsp;&nbsp;"| publisher
    publisher -->|"&nbsp;&nbsp;Publish&nbsp;&nbsp;"| kafka

One implementation is a polling publisher. It periodically reads unpublished events:

SELECT *
FROM outbox
WHERE status = 'NEW'
ORDER BY created_at
LIMIT 100;

It then publishes each event to Kafka and marks the corresponding outbox record as processed. Another approach is Change Data Capture (CDC). A CDC system can observe changes in the database transaction log and turn new outbox records into Kafka messages without requiring the application to poll the table directly. The exact publishing mechanism can vary. The important guarantee comes from writing the business state and the outbox record atomically.

The Outbox Publisher Can Still Fail

The Outbox pattern does not eliminate every failure. Consider this sequence:

Kafka has already accepted the message, but the publisher fails before recording that fact in the database. After restarting, the publisher still sees the outbox event as pending. It may publish the same event again.

This illustrates an important property of reliable distributed messaging:

It is often easier to guarantee that an event is delivered at least once than to guarantee that it is delivered exactly once across every participating system.

If publishing fails before Kafka accepts the event, the publisher can simply retry.

The more difficult case is when Kafka succeeds but the publisher cannot reliably record that success. The result is possible duplicate delivery. This is why Outbox and idempotency are closely related. The Outbox pattern changes the failure mode from:

The business state was committed, but the event may be lost.

to:

The event should eventually be delivered, but it may be delivered more than once.

Duplicates are usually a much safer problem to handle than permanently missing events.

Idempotent Consumers

A consumer should therefore be prepared to receive the same event more than once. One common strategy is to give every event a unique identifier:

eventId = evt-7821

The consumer keeps track of events it has already processed.

Another option is to design the operation itself so repeating it produces the same result. For example, setting an order to CONFIRMED can be naturally idempotent if applying the same valid state transition twice does not create additional side effects. The correct strategy depends on the domain, but the goal is the same:

Reprocessing the same event should not incorrectly apply the same business effect twice.

Idempotency deserves a deeper discussion of its own, but it is an essential part of making reliable event-driven workflows safe.

Putting the Pieces Together

The complete architecture may now contain several different reliability mechanisms.

%%{init: {"flowchart": {"nodeSpacing": 40, "rankSpacing": 50, "diagramPadding": 16}}}%%
flowchart TB
    kafka["Kafka"]

    order["Order Service"]
    orderDb[("Order DB<br/>Orders + Outbox")]
    orderPublisher["Order Outbox<br/>Publisher"]

    inventory["Inventory Service"]
    inventoryDb[("Inventory DB<br/>Inventory + Outbox")]
    inventoryPublisher["Inventory Outbox<br/>Publisher"]

    payment["Payment Service"]
    paymentDb[("Payment DB<br/>Payments + Outbox")]
    paymentPublisher["Payment Outbox<br/>Publisher"]

    kafka --> order
    kafka --> inventory
    kafka --> payment

    order --> orderDb --> orderPublisher
    inventory --> inventoryDb --> inventoryPublisher
    payment --> paymentDb --> paymentPublisher

    orderPublisher --> kafka
    inventoryPublisher --> kafka
    paymentPublisher --> kafka

Each mechanism solves a different problem, and the patterns work together:

Following a Complete Failure

Consider a case where payment succeeds, but the final order confirmation cannot complete.

At this point, the Saga decides that the business operation must be compensated.

Recovery must also handle duplicate messages and failures along the way:

• Duplicate refund request: If RefundPayment is delivered twice, the Payment Service must handle it idempotently so the customer is not refunded twice.
• Refund completed, event publication delayed: The Payment Service stores its local refund state and a PaymentRefunded outbox record in the same transaction. The outbox publisher can retry publication later.
• Inventory release temporarily fails: The Saga remains in a recovery state and retries the compensating action.
• Inventory Service stays unavailable: Monitoring must flag the stalled workflow, with an operational escalation path when automatic recovery cannot complete.

No single mechanism solves all these cases. Reliability comes from designing each failure boundary deliberately.

Eventual Consistency Is a Recovery Model

During compensation, the services may temporarily disagree. For example:

Later:

The intermediate state is not automatically a defect. In an eventually consistent design, the more important questions are:

• Is the intermediate state understood?
• Is there a durable record of what should happen next?
• Can failed operations be retried safely?
• Can duplicate operations be tolerated?
• Can the system eventually reach a valid business state?
• Can operators identify workflows that are unable to recover automatically?

Eventual consistency should not mean hoping that the data eventually looks right. It should mean having an explicit recovery model.

Questions to Ask When Designing the Workflow

When a business operation crosses service boundaries, it helps to work through the failure modes before implementing the happy path.

The design becomes much clearer when failure is treated as part of the workflow rather than as an exceptional path added later.

Final Thoughts

Microservices give individual services autonomy over their data and their transactions, but business processes rarely stop at those boundaries. A checkout, booking, payment, claim, or provisioning flow may still need several services to work together as one logical operation. Trying to treat that workflow as one enormous database transaction is usually not the model used in loosely coupled microservice architectures. Instead, consistency is built from several smaller guarantees:

• Local transactions protect state changes within a service.
• Sagas coordinate the overall business workflow.
• Compensation handles work that has already committed when later steps fail.
• Transactional Outbox makes the relationship between local state changes and outgoing events reliable.
• Idempotency makes retries and duplicate delivery safe.

Together, these patterns allow a distributed system to move through partial failures and recover toward a valid business state without requiring one global transaction across every service involved.

References

• Microservices.io — Saga pattern (choreography and orchestration)
• Microsoft Azure Architecture Center — Compensating Transaction pattern
• Microservices.io — Transactional Outbox pattern
• Microservices.io — Idempotent Consumer pattern
• Debezium — Outbox Event Router (publishing outbox events with CDC)