Outbox Pattern and the Two-Stage Commit Problem

Outbox Pattern and the Two-Stage Commit Problem

In the world of microservices and distributed systems, ensuring reliable message delivery while maintaining data consistency is one of the most challenging problems developers face. The Outbox Pattern provides an elegant solution to this dual-write problem, guaranteeing that business operations and event publishing happen atomically. This comprehensive guide will walk you through the implementation of the Outbox Pattern step-by-step, discuss the requirements for success, and highlight critical considerations.

Understanding the Problem: The Dual-Write Challenge

Before diving into the solution, let's understand the problem the Outbox Pattern solves. Consider this common scenario in a microservices architecture:

  1. A service processes a business operation (e.g., creating an order)
  2. The service needs to persist data to a database
  3. The service must publish an event to notify other services

The challenge arises when trying to make both operations atomic. What happens if the database transaction succeeds but the message publishing fails? Or vice versa? This dual-write problem can lead to:

  • Data inconsistency between services
  • Lost events that other services never receive
  • Duplicate processing when retries occur
  • **System-wide reliability issues

The Outbox Pattern Solution

The Outbox Pattern solves this by storing outbound messages in the same database transaction as the business data. A separate background process then reliably publishes these messages to the message broker. This approach ensures exactly-once delivery semantics and maintains consistency between local state changes and event publishing.

System Components

The solution consists of five main components:

  1. OrderCoordinator - ASP.NET Core Web API (.NET 9)
  2. InventoryApi - ASP.NET Core Web API (.NET 9)
  3. PaymentsApi - ASP.NET Core Web API (.NET 9)
  4. RollbackWorker - Azure Functions (.NET 8)
  5. Contracts - Shared contract library (.NET 9)
  6. Azure Service Bus - Message broker for async communication

Architectural Patterns

1. Microservices Architecture

  • Each service owns its data (SQLite databases)
  • Services communicate via HTTP APIs and async messaging
  • Loose coupling through well-defined contracts

2. Outbox Pattern Implementation

The Outbox Pattern is implemented in the OrderCoordinator service to ensure reliable message publishing:

public sealed class OutboxMessage
{
    public Guid Id { get; init; } = Guid.NewGuid();
    public DateTime OccurredUtc { get; init; } = DateTime.UtcNow;
    public string Type { get; init; } = default!;
    public string PayloadJson { get; init; } = default!;
    public int AttemptCount { get; set; }
    public DateTime? LastAttemptUtc { get; set; }

    public static OutboxMessage From<T>(T payload) where T : class
        => new OutboxMessage
        {
            Type = typeof(T).FullName!,
            PayloadJson = JsonSerializer.Serialize(payload)
        };
}

Key Implementation Details:

  1. Transactional Consistency: Outbox messages are stored in the same database transaction as business operations
  2. Background Publisher: OutboxPublisher runs as a BackgroundService polling for pending messages
  3. Retry Logic: Failed messages track attempt counts and timestamps
  4. At-Least-Once Delivery: Messages are only removed after successful publishing

3. Orchestration-Based Saga Pattern

The OrderCoordinator implements an orchestration saga with compensating transactions:

// Try-Confirm-Cancel pattern
try
{
    // Try: Reserve inventory
    var resResp = await inventory.PostAsJsonAsync("inventory/reservations", ...);

    // Try: Authorize payment
    var authResp = await payments.PostAsJsonAsync("payments/authorize", ...);

    // Confirm both operations
    await inventory.PostAsJsonAsync("inventory/reservations/confirm", ...);
    await payments.PostAsJsonAsync("payments/capture", ...);
}
catch (Exception ex)
{
    // Compensate on failure
    db.Outbox.Add(OutboxMessage.From(new OrderRollbackRequested(...)));
}

4. Idempotency Pattern

Both the order coordinator and downstream services implement idempotency:

// Order coordinator uses idempotency keys
var idem = await db.Idempotency.FirstOrDefaultAsync(i => i.Key == idempotencyKey, ct);
if (idem is not null)
    return Results.Accepted($"/orders/{idem.ResultOrderId}", new { orderId = idem.ResultOrderId });

// Downstream services check for existing operations
var existing = await db.Authorizations.FirstOrDefaultAsync(a => a.OrderId == req.OrderId);
if (existing is not null)
    return Results.Ok(new AuthorizePaymentResponse(existing.AuthorizationId));

Step-by-Step Implementation Guide

Step 1: Design the Outbox Message Entity

The foundation of the Outbox Pattern is a database table that stores pending messages. Here's a well-designed outbox entity:

public sealed class OutboxMessage
{
    public Guid Id { get; init; } = Guid.NewGuid();
    public DateTime OccurredUtc { get; init; } = DateTime.UtcNow;
    public string Type { get; init; } = default!;
    public string PayloadJson { get; init; } = default!;
    public int AttemptCount { get; set; }
    public DateTime? LastAttemptUtc { get; set; }

    public static OutboxMessage From<T>(T payload) where T : class
        => new OutboxMessage
        {
            Type = typeof(T).FullName!,
            PayloadJson = JsonSerializer.Serialize(payload)
        };
}

Key Design Decisions:

  • Immutable core properties: Id, OccurredUtc, Type, and PayloadJson are set once and never changed
  • Tracking metadata: AttemptCount and LastAttemptUtc support retry logic and monitoring
  • Generic serialization: The From<T> method provides type-safe message creation
  • Type information: Storing the full type name enables proper deserialization

Step 2: Integrate with Your Data Model

Add the outbox table to your existing DbContext:

public sealed class AppDb(DbContextOptions<AppDb> opt) : DbContext(opt)
{
    public DbSet<OrderRecord> Orders => Set<OrderRecord>();
    public DbSet<IdempotencyRecord> Idempotency => Set<IdempotencyRecord>();
    public DbSet<OutboxMessage> Outbox => Set<OutboxMessage>();

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        // Add indexes for efficient querying
        modelBuilder.Entity<OutboxMessage>()
            .HasIndex(x => x.OccurredUtc)
            .HasDatabaseName("IX_OutboxMessage_OccurredUtc");

        modelBuilder.Entity<OutboxMessage>()
            .HasIndex(x => new { x.AttemptCount, x.LastAttemptUtc })
            .HasDatabaseName("IX_OutboxMessage_Retry");
    }
}

Step 3: Implement Transactional Message Storage

The critical aspect is storing outbox messages within the same database transaction as your business operations:

public async Task<IResult> PlaceOrderAsync(CreateOrderDto dto, string idempotencyKey, CancellationToken ct)
{
    // Check idempotency first
    var idem = await db.Idempotency.FirstOrDefaultAsync(i => i.Key == idempotencyKey, ct);
    if (idem is not null)
        return Results.Accepted($"/orders/{idem.ResultOrderId}", new { orderId = idem.ResultOrderId });

    var order = new OrderRecord { Id = Guid.NewGuid(), Amount = dto.Amount, Status = "Pending" };
    db.Orders.Add(order);
    await db.SaveChangesAsync(ct);

    try
    {
        // Business logic with external services
        await ProcessInventoryReservation(order, dto.Items, ct);
        await ProcessPaymentAuthorization(order, ct);
        await ConfirmOrderComponents(order, ct);

        order.Status = "Confirmed";
    }
    catch (Exception ex)
    {
        order.Status = "Canceled";
        // Store rollback event in the same transaction
        db.Outbox.Add(OutboxMessage.From(new OrderRollbackRequested(
            order.Id, 
            order.InventoryReservationId, 
            order.PaymentAuthId, 
            ex.Message)));
    }

    // Store idempotency record and save everything atomically
    db.Idempotency.Add(new IdempotencyRecord { Key = idempotencyKey, ResultOrderId = order.Id });
    await db.SaveChangesAsync(ct);

    return Results.Accepted($"/orders/{order.Id}", new { orderId = order.Id });
}

Critical Points:

  • Outbox messages are added to the same DbContext as business entities
  • A single SaveChangesAsync() call ensures atomicity
  • If the transaction fails, neither business data nor outbox messages are persisted

Step 4: Build the Background Publisher

The background service continuously polls for pending messages and publishes them:

public class OutboxPublisher(IServiceProvider sp, IConfiguration cfg, ILogger<OutboxPublisher> log) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var pollSeconds = cfg.GetValue<int?>("Outbox:PollSeconds") ?? 1;
        var batchSize = cfg.GetValue<int?>("Outbox:BatchSize") ?? 100;
        var topicName = cfg["ServiceBus:TopicName"] ?? "orders";

        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await ProcessPendingMessages(batchSize, topicName, stoppingToken);
            }
            catch (Exception ex)
            {
                log.LogError(ex, "Outbox processing loop error");
            }

            await Task.Delay(TimeSpan.FromSeconds(pollSeconds), stoppingToken);
        }
    }

    private async Task ProcessPendingMessages(int batchSize, string topicName, CancellationToken stoppingToken)
    {
        using var scope = sp.CreateScope();
        var db = scope.ServiceProvider.GetRequiredService<AppDb>();
        var sbClient = scope.ServiceProvider.GetRequiredService<ServiceBusClient>();
        var sender = sbClient.CreateSender(topicName);

        var pending = await db.Outbox
            .OrderBy(x => x.OccurredUtc)
            .Take(batchSize)
            .ToListAsync(stoppingToken);

        foreach (var message in pending)
        {
            try
            {
                await PublishMessage(sender, message, stoppingToken);
                db.Outbox.Remove(message);
            }
            catch (Exception ex)
            {
                await HandlePublishFailure(message, ex);
            }
        }

        await db.SaveChangesAsync(stoppingToken);
    }

    private async Task PublishMessage(ServiceBusSender sender, OutboxMessage outboxMessage, CancellationToken ct)
    {
        var message = new ServiceBusMessage(outboxMessage.PayloadJson)
        {
            Subject = outboxMessage.Type,
            MessageId = outboxMessage.Id.ToString(),
            ContentType = "application/json"
        };

        await sender.SendMessageAsync(message, ct);
        log.LogInformation("Published message {MessageId} of type {Type}", 
            outboxMessage.Id, outboxMessage.Type);
    }

    private async Task HandlePublishFailure(OutboxMessage message, Exception ex)
    {
        message.AttemptCount++;
        message.LastAttemptUtc = DateTime.UtcNow;

        log.LogWarning(ex, "Failed to publish message {MessageId} (attempt {AttemptCount})", 
            message.Id, message.AttemptCount);

        // Implement exponential backoff or dead letter logic here
        if (message.AttemptCount >= 5)
        {
            log.LogError("Message {MessageId} exceeded max retry attempts", message.Id);
            // Consider moving to dead letter storage
        }
    }
}

Step 5: Configure Dependency Injection

Register all components in your DI container:

var builder = WebApplication.CreateBuilder(args);

// Database
builder.Services.AddDbContext<AppDb>(opt => 
    opt.UseSqlite(builder.Configuration.GetConnectionString("AppDb")));

// Message Broker
builder.Services.AddSingleton(sp =>
{
    var cfg = sp.GetRequiredService<IConfiguration>();
    var conn = cfg["ServiceBus:ConnectionString"]!;
    return new ServiceBusClient(conn);
});

// Background Service
builder.Services.AddHostedService<OutboxPublisher>();

// HTTP Clients for external services
builder.Services.AddHttpClient("Inventory", c =>
{
    c.BaseAddress = new Uri(builder.Configuration["Services:Inventory"]!);
});
builder.Services.AddHttpClient("Payments", c =>
{
    c.BaseAddress = new Uri(builder.Configuration["Services:Payments"]!);
});

var app = builder.Build();

// Ensure database is created
await using (var scope = app.Services.CreateAsyncScope())
{
    var db = scope.ServiceProvider.GetRequiredService<AppDb>();
    await db.Database.EnsureCreatedAsync();
}

Step 6: Add Monitoring and Observability

Create an endpoint to monitor outbox health:

app.MapGet("/outbox", async (AppDb db) =>
{
    var messages = await db.Outbox
        .AsNoTracking()
        .OrderBy(x => x.OccurredUtc)
        .ToListAsync();

    return Results.Ok(messages.Select(m => new 
    { 
        m.Id, 
        m.Type, 
        m.AttemptCount, 
        m.LastAttemptUtc, 
        m.OccurredUtc,
        Status = m.AttemptCount == 0 ? "Pending" : 
                m.AttemptCount < 5 ? "Retrying" : "Failed"
    }));
});

app.MapGet("/outbox/health", async (AppDb db) =>
{
    var stats = await db.Outbox
        .GroupBy(m => m.AttemptCount == 0 ? "Pending" : 
                     m.AttemptCount < 5 ? "Retrying" : "Failed")
        .Select(g => new { Status = g.Key, Count = g.Count() })
        .ToListAsync();

    var oldestPending = await db.Outbox
        .Where(m => m.AttemptCount == 0)
        .OrderBy(m => m.OccurredUtc)
        .Select(m => m.OccurredUtc)
        .FirstOrDefaultAsync();

    return Results.Ok(new 
    { 
        Statistics = stats,
        OldestPendingMessage = oldestPending,
        HealthStatus = oldestPending == default || 
                      DateTime.UtcNow - oldestPending < TimeSpan.FromMinutes(5) 
                      ? "Healthy" : "Warning"
    });
});

Requirements for Success

1. Transactional Database Support

The Outbox Pattern requires a database that supports ACID transactions. This includes:

  • Relational databases: PostgreSQL, SQL Server, MySQL, SQLite
  • Document databases with transactions: MongoDB (with replica sets), Azure Cosmos DB

Note: Simple key-value stores or eventually consistent databases won't work for this pattern.

2. Exactly-Once Processing Logic

Your business logic must be idempotent to handle potential duplicate processing:

// Always check for existing records
var existing = await db.Orders.FirstOrDefaultAsync(o => o.Id == orderId);
if (existing != null)
{
    return existing; // Idempotent response
}

// Proceed with creation only if not exists
var order = new OrderRecord { Id = orderId, /* ... */ };

3. Message Ordering Considerations

If message order matters, consider these strategies:

  • Partition by entity: Group related messages using partition keys
  • Sequence numbers: Add ordering metadata to messages
  • Single-threaded processing: Process messages sequentially (impacts throughput)

4. Error Handling and Dead Letter Management

Implement robust retry logic with exponential backoff:

private static readonly TimeSpan[] RetryDelays = 
{
    TimeSpan.FromSeconds(1),
    TimeSpan.FromSeconds(5),
    TimeSpan.FromSeconds(30),
    TimeSpan.FromMinutes(5),
    TimeSpan.FromMinutes(30)
};

private bool ShouldRetry(OutboxMessage message)
{
    if (message.AttemptCount >= RetryDelays.Length)
        return false;

    var nextRetryTime = message.LastAttemptUtc?.Add(RetryDelays[message.AttemptCount - 1]);
    return nextRetryTime == null || DateTime.UtcNow >= nextRetryTime;
}

5. Performance and Scalability

For high-throughput scenarios:

  • Batch processing: Process multiple messages in each iteration
  • Parallel processing: Use multiple worker threads with proper coordination
  • Database indexes: Ensure efficient querying of pending messages
  • Message archival: Remove or archive successfully processed messages

Words of Caution

1. Storage Growth Management

Critical Warning: The outbox table can grow rapidly in high-volume systems. Without proper cleanup, it can:

  • Consume excessive storage space
  • Degrade query performance
  • Impact application startup times

Solution: Implement message cleanup after successful publishing:

// Remove successful messages immediately
db.Outbox.Remove(message);

// Or archive for audit purposes
message.ProcessedAt = DateTime.UtcNow;
message.Status = "Completed";

2. Message Broker Dependency

The background publisher creates a dependency on your message broker. Consider:

  • Circuit breaker patterns for broker outages
  • Graceful degradation when messaging is unavailable
  • Health checks to monitor broker connectivity

3. Delayed Message Processing

Messages aren't published immediately; there's a delay based on your polling interval. This means:

  • Near real-time requirements may not be suitable for this pattern
  • Dependent services must handle eventual consistency
  • User expectations should account for processing delays

4. Database Transaction Size

Including outbox messages in business transactions can:

  • Increase transaction size and duration
  • Impact database performance under high load
  • Create contention on the outbox table

Mitigation: Consider using separate transactions for outbox writes in extreme high-volume scenarios, accepting the risk of lost messages for performance gains.

5. Message Duplication Scenarios

Despite best efforts, duplicate messages can occur due to:

  • Publisher retries after timeout (message was actually sent)
  • Database connection issues during acknowledgment
  • Process crashes between publishing and cleanup

Solution: Always implement idempotent message handlers:

public async Task HandleOrderCreated(OrderCreated orderEvent)
{
    // Check if already processed
    var existing = await db.ProcessedEvents
        .FirstOrDefaultAsync(e => e.EventId == orderEvent.Id);

    if (existing != null)
    {
        log.LogInformation("Event {EventId} already processed", orderEvent.Id);
        return; // Idempotent exit
    }

    // Process the event
    await ProcessOrder(orderEvent);

    // Record processing
    db.ProcessedEvents.Add(new ProcessedEvent 
    { 
        EventId = orderEvent.Id, 
        ProcessedAt = DateTime.UtcNow 
    });

    await db.SaveChangesAsync();
}

Conclusion

The Outbox Pattern is a powerful solution for ensuring reliable message delivery in distributed systems. When implemented correctly, it provides:

  • Guaranteed message delivery with exactly-once semantics
  • Data consistency between local and distributed state
  • Resilience against partial failures
  • Auditability of all system events

However, success requires careful attention to transactional boundaries, idempotency, error handling, and performance considerations. The pattern is particularly valuable in mission-critical systems where data consistency and reliability are paramount.

Remember that the Outbox Pattern is one tool in the distributed systems toolkit. Evaluate whether its benefits outweigh the added complexity for your specific use case. For simple scenarios with relaxed consistency requirements, direct message publishing might suffice. For complex, business-critical systems, the Outbox Pattern provides the reliability guarantees that enable robust distributed architectures.

By following this guide and heeding the cautions outlined, you'll be well-equipped to implement a production-ready Outbox Pattern that serves as a reliable foundation for your microservices communication strategy.

Comments ()