Jesse's Notes

Outbox Pattern

4 min read

The Outbox Pattern ensures reliable message delivery in distributed systems by storing events in the same database transaction as business data changes.

The Problem

In distributed systems, you often need to:

  1. Update your database
  2. Publish an event to a message queue

The challenge: These are two separate operations that can fail independently:

// This is NOT atomic
database.Save(order);           // ✓ Succeeds
messageQueue.Publish(orderCreated); // ✗ Fails
 
// Now: Database has the order, but no event was published
// Other services never know the order was created

You can’t use traditional database transactions because the message queue is external.

The Solution

The Outbox Pattern makes both operations atomic by storing events in the same database:

  1. Write business data and events in a single database transaction
  2. A separate process reads from the outbox table and publishes events
  3. Mark events as published after successful delivery

Database Schema

CREATE TABLE orders (
    id UUID PRIMARY KEY,
    customer_id UUID,
    total DECIMAL,
    -- ... other columns
);
 
CREATE TABLE outbox (
    id UUID PRIMARY KEY,
    aggregate_type VARCHAR(255),
    aggregate_id UUID,
    event_type VARCHAR(255),
    payload JSONB,
    created_at TIMESTAMP,
    published_at TIMESTAMP NULL
);

Writing to Outbox

public class CreateOrderHandler
{
    public async Task Handle(CreateOrderCommand cmd)
    {
        using var transaction = await db.BeginTransaction();
 
        // 1. Save business data
        var order = new Order(cmd.CustomerId, cmd.Items);
        await db.Orders.Add(order);
 
        // 2. Save event to outbox (same transaction!)
        var outboxEvent = new OutboxEvent
        {
            AggregateType = "Order",
            AggregateId = order.Id,
            EventType = "OrderCreated",
            Payload = JsonSerializer.Serialize(new OrderCreatedEvent(order))
        };
        await db.Outbox.Add(outboxEvent);
 
        // 3. Commit both together - atomic!
        await transaction.Commit();
    }
}

Publishing from Outbox

A background worker continuously polls and publishes:

public class OutboxPublisher : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        while (!ct.IsCancellationRequested)
        {
            var unpublishedEvents = await db.Outbox
                .Where(e => e.PublishedAt == null)
                .OrderBy(e => e.CreatedAt)
                .Take(100)
                .ToListAsync();
 
            foreach (var evt in unpublishedEvents)
            {
                try
                {
                    // Publish to message queue
                    await messageQueue.Publish(evt.EventType, evt.Payload);
 
                    // Mark as published
                    evt.PublishedAt = DateTime.UtcNow;
                    await db.SaveChangesAsync();
                }
                catch (Exception ex)
                {
                    // Log and retry later
                    logger.LogError(ex, "Failed to publish event {EventId}", evt.Id);
                }
            }
 
            await Task.Delay(TimeSpan.FromSeconds(5), ct);
        }
    }
}

Guarantees

At-Least-Once Delivery

  • Events may be published multiple times if the publisher crashes after sending but before marking as published
  • Consumers must be idempotent

Ordering

  • Events are ordered per aggregate (by created_at)
  • Global ordering across aggregates is not guaranteed
  • Consider using partition keys for ordering guarantees

Benefits

Atomicity

  • Database changes and event publishing are atomic
  • No lost events due to partial failures

Reliability

  • Events persist in database until successfully published
  • Survives application crashes and message queue outages

Auditability

  • Complete record of all events ever published
  • Can replay or analyze event history

Tradeoffs

Eventual Consistency

  • Small delay between database write and event publication
  • Consuming services see updates slightly later

Complexity

  • Need background worker to publish events
  • Database cleanup strategy required (archive old published events)
  • Must handle duplicate events in consumers

Database Load

  • Additional writes to outbox table
  • Polling adds read load (use change data capture for efficiency)

Optimizations

Change Data Capture (CDC)

Instead of polling, use database CDC features:

  • PostgreSQL: Logical replication, pg_logical
  • MySQL: Binlog
  • SQL Server: Change Data Capture

Tools like Debezium can read CDC streams and publish to Kafka automatically.

Batching

Publish multiple events in a single message queue transaction to reduce overhead.

Cleanup

Archive or delete published events after a retention period:

DELETE FROM outbox
WHERE published_at < NOW() - INTERVAL '30 days';

When to Use

Use the Outbox Pattern when:

  • You need reliable event publishing from a transactional database
  • You’re building microservices with event-driven communication
  • You need to maintain consistency across service boundaries
  • You can tolerate eventual consistency (small delay)

Don’t use when:

  • Working within a monolith (direct method calls are simpler)
  • Real-time synchronous consistency is required
  • You don’t have distributed services
  • Event Sourcing - Store all state as events
  • Idempotency - Required for consuming outbox events
  • Message Queues - Target for published events
  • Transactional Outbox (alternative name for same pattern)
  • Inbox Pattern (dual pattern for consuming events reliably)

Resources

See Also

Graph View

Backlinks