RabbitMQ in .NET: retries, duplicate messages and the outbox pattern

With manual acks and publisher confirms, RabbitMQ makes "at-least-once" delivery possible, but it does not guarantee exactly-once delivery. That is why reliable message processing in .NET has several layers: durable queues and persistent messages, publisher confirms, manual acks with a bounded prefetch, a consumer that safely skips duplicate messages, delayed retries through a dead-letter exchange, a parking queue for poison messages, and the outbox pattern for when you have to both write to the database and publish a message. The code samples use the asynchronous API of RabbitMQ.Client 7.

Where can a message get lost?

A message is at risk in three places: between the publisher and the broker, inside the broker itself, and between the broker and the consumer. Start with the broker: the queue must be durable and the message persistent. The two are independent, and a transient message is discarded when the node restarts, even if it sits in a durable queue. To make a message persistent in RabbitMQ.Client 7, set Persistent to true in BasicProperties. A quorum queue is always durable, is replicated across several nodes and keeps all its messages persistent regardless of their delivery mode; the examples in this guide use this queue type.

On the publisher side, if publisher confirms are not enabled, a completed BasicPublishAsync only means that the client has sent the message, not that the broker has accepted it. In RabbitMQ.Client 7 you turn confirms on when you create the channel, with CreateChannelOptions. If publisherConfirmationTrackingEnabled is enabled as well, an await on BasicPublishAsync does not complete until the broker's confirm arrives. If the broker nacks the message, or a message published with mandatory: true reaches no queue and is returned, a PublishException is thrown. For a persistent message in a durable queue, a confirm means the message has been written to disk; in a quorum queue it means a majority of replicas have accepted it. The default for mandatory is false, and a message that reaches no queue is silently dropped unless an alternate exchange is configured.

If the connection drops before the confirm arrives, the publisher cannot tell whether the message got through, and the only safe option is to publish it again. That is the first source of duplicate messages.

Manual ack, autoAck and prefetch

In autoAck mode the broker counts a message as delivered the moment it sends it, and if the process fails halfway through processing or the connection drops, the message is gone. For important messages, call BasicConsumeAsync with autoAck: false and send BasicAckAsync only after the result has been stored in the database. Any delivery that has not been acked goes back to the queue automatically when the channel or connection closes.

BasicQosAsync sets how many unacknowledged messages a consumer can hold at once, and with global: false RabbitMQ applies that limit to each consumer separately. A value of zero means no limit, and the documentation warns that consuming without acks and without a limit drives up memory use on the broker node. According to the documentation, values in the 100 to 300 range usually give the best throughput, and 1 is the most conservative choice, one that noticeably reduces throughput; find the right number by measuring under real load.

Two more defaults matter. The ack timeout is 30 minutes by default. If a delivery is not acked within that time, then depending on the broker version and queue type, either the channel is closed with a PRECONDITION_FAILED error or (for quorum queues since 4.3) the consumer is canceled; either way, the unacked deliveries go back to the queue. ConsumerDispatchConcurrency in RabbitMQ.Client 7 also defaults to 1, which means a channel's handlers run one at a time. With a higher value, processing runs in parallel, but the processing order of messages is no longer guaranteed and the handler has to be thread-safe.

Why do duplicate messages arrive?

With manual acks, delivery is at-least-once, and duplicate messages have to be treated as normal system behavior. Three common scenarios:

• The consumer committed the database transaction but failed before its ack reached the broker.
• The publisher did not receive the confirm and published the message again, while the first copy had already reached the queue.
• The relay in the outbox pattern sent the message but stopped before marking the row as sent.

You cannot rely on the Redelivered flag. The RabbitMQ documentation treats it only as a hint that the message may have been seen before, and in the second scenario it is not set at all, because to the broker the second copy is a new message. The documentation's own advice is to design consumers to be idempotent.

The first option is to make the operation itself idempotent: "mark order 42 as paid" can run many times, but "add 100 units to stock" cannot. The second option, for operations that are not idempotent by nature, is a table of processed messages with MessageId as its primary key, filled in the same transaction that records the main change. This only works if the producer sets the MessageId when it creates the message and keeps it on every republish; if each publish generates a fresh Guid, the table is useless.

public sealed class OrderPlacedHandler(BillingDbContext db)
{
    public async Task HandleAsync(string messageId, OrderPlaced order, CancellationToken ct)
    {
        if (await db.ProcessedMessages.AnyAsync(p => p.MessageId == messageId, ct))
            return;

        var invoice = Invoice.CreateFor(order);
        db.Invoices.Add(invoice);
        db.ProcessedMessages.Add(new ProcessedMessage(messageId, DateTime.UtcNow));
        db.OutboxMessages.Add(OutboxMessage.Create("invoice.created", new InvoiceCreated(invoice.Id)));

        try
        {
            await db.SaveChangesAsync(ct);
        }
        catch (DbUpdateException ex) when (ex.InnerException is SqlException { Number: 2627 or 2601 })
        {
            // Same MessageId already saved by a concurrent delivery: treat as processed.
        }
    }
}

All three inserts happen in one SaveChangesAsync call, and by default EF Core applies all the changes of one SaveChanges in a single transaction, so the invoice, the message record and the outbox row are written together or not at all. AnyAsync is just a cheap shortcut; the real guarantee comes from the primary key. If two consumers process the same message concurrently, the second save fails with error 2627 or 2601 (a primary key or unique index violation). The empty catch is deliberate: the second transaction has been rolled back completely, the first one has recorded this message, and the handler returns without an error so the consumer acks the message. If other tables in this transaction also have unique constraints, check the constraint name as well, so that a real error is not hidden. And create a new DbContext for each message, because after a failed SaveChanges the added entities are still tracked by that DbContext.

Delayed retries: a DLX and a wait queue

The simplest reaction to an error is BasicNackAsync with requeue: true, but the requeued message may be redelivered immediately. If the cause of the error does not go away immediately, for example if the database is unavailable for a few minutes, you get a loop of deliveries and requeues that the RabbitMQ documentation describes as expensive in network bandwidth and CPU. Attempts need a gap between them.

The classic pattern, which works on all supported versions, is built from a dead-letter exchange (DLX) and a TTL. The work queue hands rejected messages to its DLX, and that exchange is bound to a wait queue that has no consumers, a fixed TTL, and the original exchange as its own DLX. When processing fails, the consumer rejects the message with requeue: false; the message sits in the wait queue until the TTL runs out and then returns to the work queue. The code below is the start of the ExecuteAsync method of a BackgroundService; read the user name and password from configuration rather than writing them in code.

var factory = new ConnectionFactory
{
    HostName = "rabbitmq.internal",
    ClientProvidedName = "billing-worker"
};
await using var connection = await factory.CreateConnectionAsync(stoppingToken);
await using var channel = await connection.CreateChannelAsync(cancellationToken: stoppingToken);

await channel.ExchangeDeclareAsync("orders", ExchangeType.Direct, durable: true);
await channel.ExchangeDeclareAsync("orders.retry", ExchangeType.Fanout, durable: true);

await channel.QueueDeclareAsync("orders.work", durable: true, exclusive: false, autoDelete: false,
    arguments: new Dictionary<string, object?>
    {
        ["x-queue-type"] = "quorum",
        ["x-dead-letter-exchange"] = "orders.retry"
    });
await channel.QueueBindAsync("orders.work", "orders", "order.placed");

await channel.QueueDeclareAsync("orders.wait", durable: true, exclusive: false, autoDelete: false,
    arguments: new Dictionary<string, object?>
    {
        ["x-queue-type"] = "quorum",
        ["x-message-ttl"] = 30_000,
        ["x-dead-letter-exchange"] = "orders"
    });
await channel.QueueBindAsync("orders.wait", "orders.retry", "");

await channel.QueueDeclareAsync("orders.parking", durable: true, exclusive: false, autoDelete: false,
    arguments: new Dictionary<string, object?> { ["x-queue-type"] = "quorum" });

The wait queue has no x-dead-letter-routing-key, so the message goes back with its original routing key, order.placed, to the orders exchange and from there to orders.work. RabbitMQ detects dead-letter cycles, but it drops a message only if there was no rejection anywhere in the cycle; here every round starts with a rejection by the consumer. The orders.parking queue has neither an automatic consumer nor a TTL, so messages stay there for inspection. A few more points:

• Put the TTL on the queue, not on individual messages. An expired message is dead-lettered only when it reaches the head of the queue; with a separate TTL per message, expired messages can get stuck behind ones that have not expired yet. For stepped delays, create several wait queues with different TTLs and have the consumer publish the message to the right one itself.
• The RabbitMQ documentation recommends policies instead of x-arguments for DLX and TTL, because x-arguments cannot be changed without redeploying the application, while a policy can be changed at any time. I used x-arguments in the code above for readability; only x-queue-type has to be set when the queue is declared.
• By default, dead-lettering happens without internal publisher confirms (the at-most-once strategy), and in a cluster a message can be lost along this path. In a quorum queue, setting dead-letter-strategy to at-least-once and overflow to reject-publish in a policy makes this path at-least-once, at the cost of more memory and CPU.
• Since RabbitMQ 4.3, quorum queues have built-in delayed retry with a linear delay: with the x-delayed-retry-type, x-delayed-retry-min and x-delayed-retry-max arguments (or the matching keys without the x- prefix in a policy), a message that the consumer returns to the queue is held before it is redelivered, and the delay grows with the number of failed deliveries. If your broker version allows it, this feature can replace the wait queue.
• According to the README of the delayed message exchange plugin's repository, the RabbitMQ team no longer maintains it; do not build new designs on it.

Poison messages and the parking queue

A poison message never processes successfully: its format is broken, it points to a record that does not exist, or it hits a bug in the handler. Retrying it forever only wastes resources and hides the problem. Attempts need a cap, after which the message goes to a parking queue for someone to look at.

Quorum queues also have a built-in safety net. Since RabbitMQ 4.0 the default delivery-limit is 20, and the number of failed deliveries is carried in the x-delivery-count header. A message that goes over the limit is dead-lettered if a DLX is configured and dropped otherwise; in other words, a quorum queue without a DLX silently deletes poison messages. Since 4.3 this counter goes up only on real failures, such as basic.reject or a closed channel or connection, and it does not count basic.nack. According to the documentation, disabling the limit with -1 is not recommended.

In this design the attempt count comes from the x-death header: each time the message is dead-lettered from orders.work, RabbitMQ increments the counter for that queue and reason in this header. The consumer reads the number before processing, and if it has reached the cap, sends the message to parking without running it. A message that cannot be read or has no MessageId goes straight there too. This snippet continues the same method:

const int MaxAttempts = 5;

await using var parking = await connection.CreateChannelAsync(
    new CreateChannelOptions(publisherConfirmationsEnabled: true, publisherConfirmationTrackingEnabled: true));
await channel.BasicQosAsync(prefetchSize: 0, prefetchCount: 20, global: false);

var consumer = new AsyncEventingBasicConsumer(channel);
consumer.ReceivedAsync += async (_, ea) =>
{
    var id = ea.BasicProperties.MessageId;
    OrderPlaced? order = null;
    try { order = JsonSerializer.Deserialize<OrderPlaced>(ea.Body.Span); }
    catch (JsonException) { }

    if (id is null || order is null || DeathCount(ea.BasicProperties, "orders.work") >= MaxAttempts)
    {
        var props = new BasicProperties(ea.BasicProperties) { Persistent = true };
        await parking.BasicPublishAsync("", "orders.parking", mandatory: true,
            basicProperties: props, body: ea.Body.ToArray());
        await channel.BasicAckAsync(ea.DeliveryTag, multiple: false);
        return;
    }

    try
    {
        await using var scope = scopes.CreateAsyncScope();
        var handler = scope.ServiceProvider.GetRequiredService<OrderPlacedHandler>();
        await handler.HandleAsync(id, order, stoppingToken);
        await channel.BasicAckAsync(ea.DeliveryTag, multiple: false);
    }
    catch (Exception)
    {
        await channel.BasicNackAsync(ea.DeliveryTag, multiple: false, requeue: false);
    }
};

await channel.BasicConsumeAsync("orders.work", autoAck: false, consumer: consumer);
await Task.Delay(Timeout.Infinite, stoppingToken);

static long DeathCount(IReadOnlyBasicProperties props, string queue)
{
    if (props.Headers is null || !props.Headers.TryGetValue("x-death", out var raw)
        || raw is not IList<object> deaths)
        return 0;

    long total = 0;
    foreach (var d in deaths.OfType<IDictionary<string, object>>())
        if (d.TryGetValue("queue", out var q) && q is byte[] name && Encoding.UTF8.GetString(name) == queue)
            total += Convert.ToInt64(d["count"]);
    return total;
}

Publishing to parking happens on a separate channel with publisher confirms, so the original message is acked only after the broker has confirmed the copy; if the process fails between the two, the message is parked twice, which is harmless. scopes is the injected IServiceScopeFactory. DeathCount adds up all reasons for the work queue, so a message that reached the wait queue by hitting delivery-limit is counted as well. The body is copied with ToArray because, according to the client documentation, using the Body memory outside ReceivedAsync without copying it is not safe.

With MaxAttempts = 5, a message that keeps failing is processed five times and, on the sixth delivery, parked without being run. Put an alert on the length of the parking queue; a parking queue that nobody checks is, in practice, the same as silently dropping messages.

Outbox: saving to the database and publishing a message together

Suppose a service saves an invoice and also has to publish an InvoiceCreated event. SQL Server and RabbitMQ share no transaction. If you save first and the process fails before publishing, the invoice is left without an event; if you publish first and the save fails, you have broadcast an event about data that does not exist.

In the transactional outbox pattern, the message is not published directly; it is written to an outbox table in the same transaction as the main change, which is exactly what OrderPlacedHandler did with OutboxMessages.Add. A separate process, usually called the relay, reads the unsent rows, publishes them with publisher confirms and then marks them as sent. So a message is published only once its transaction has committed. The code below is the body of ExecuteAsync in a separate BackgroundService:

await using var channel = await connection.CreateChannelAsync(
    new CreateChannelOptions(publisherConfirmationsEnabled: true, publisherConfirmationTrackingEnabled: true),
    stoppingToken);

while (!stoppingToken.IsCancellationRequested)
{
    await using var scope = scopes.CreateAsyncScope();
    var db = scope.ServiceProvider.GetRequiredService<BillingDbContext>();

    var batch = await db.OutboxMessages
        .Where(m => m.SentAt == null)
        .OrderBy(m => m.CreatedAt)
        .Take(100)
        .ToListAsync(stoppingToken);

    foreach (var m in batch)
    {
        var props = new BasicProperties
        {
            MessageId = m.Id.ToString(),
            Type = m.Type,
            ContentType = "application/json",
            Persistent = true
        };
        await channel.BasicPublishAsync("billing", m.Type, mandatory: true,
            basicProperties: props, body: m.Payload, cancellationToken: stoppingToken);
        m.SentAt = DateTime.UtcNow;
        await db.SaveChangesAsync(stoppingToken);
    }

    if (batch.Count == 0)
        await Task.Delay(TimeSpan.FromSeconds(1), stoppingToken);
}

Here connection and scopes come from DI. This relay publishes each message on its own and waits for its confirm. The official RabbitMQ tutorial shows that publishing in batches is faster, but when something goes wrong it is harder to tell which message did not arrive. The outbox row's ID is the MessageId; if the relay stops after publishing and before SaveChangesAsync, it sends the same message again after a restart, and the consumer skips the second copy thanks to the processed-messages table. That is why an outbox is not complete without an idempotent consumer. In real code, catch PublishException and connection errors, log them and carry on after a short pause.

If several instances of the service run, either only one of them runs the relay or the rows are locked as they are read. The common approach in SQL Server is to read with the UPDLOCK and READPAST hints inside a transaction; the SQL Server documentation itself presents READPAST mainly as a way to implement work queues on a table. In EF Core these hints have to be written in raw SQL (FromSql), and if READ_COMMITTED_SNAPSHOT is on, READPAST at the READ COMMITTED level is accepted only together with READCOMMITTEDLOCK.

A filtered index on the unsent rows keeps the relay's query cheap (see the SQL Server performance guide), and sent rows have to be deleted or archived periodically. If you would rather not build these pieces yourself, MassTransit and NServiceBus provide retries, an outbox and duplicate detection out of the box; their licensing terms differ, and you should check them before choosing.

Checklist before production

At Gozine2, RabbitMQ was part of our stack alongside .NET 6, SQL Server and Redis. The list below sums up this guide, and you can go through it before launching any new consumer:

• A quorum queue, or at least a durable one; messages with Persistent = true; publisher confirms and mandatory: true for important messages.
• autoAck: false, acks only after commit, and a bounded prefetch.
• Processing that finishes before the ack timeout (30 minutes by default).
• A stable MessageId set by the producer, and a processed-messages table written in the same transaction.
• Delayed retries, a cap on attempts, and a parking queue with an alert on its length.
• A DLX for every quorum queue, so that delivery-limit does not silently drop messages.
• An outbox wherever saving and publishing must happen together.

Not every system needs a broker. When messages do not have to survive a process restart, an in-process queue is simpler; in the watch-time pipeline I used a bounded in-memory Channel with 202 and 503 responses. If you want help designing or reviewing a path like this, the Work with me page explains how I can help.

Frequently asked questions

Does RabbitMQ guarantee exactly-once delivery?

No. Durable queues and persistent messages, publisher confirms and manual acks give you at-least-once delivery, and duplicate messages can still arrive. An exactly-once effect has to be built into the application itself, for example with a table of processed MessageIds that is written in the same transaction as the main change.

autoAck or manual ack?

With autoAck, a message counts as delivered the moment it is sent, and if the consumer fails halfway through processing, the message is lost. For important messages, send the ack manually after the result has been stored, and limit the number of unacknowledged messages with BasicQosAsync.

Why not retry with requeue: true?

The requeued message may be redelivered immediately, and until the cause of the error is fixed you get an expensive loop of deliveries and requeues. It is better to reject the message with requeue: false into a wait queue with a TTL, or, on RabbitMQ 4.3 and later, to enable the quorum queue's own delayed retry, which holds a returned message before delivering it again.

What does delivery-limit do in a quorum queue?

Since RabbitMQ 4.0 its default is 20, and a message that fails delivery more times than that is dead-lettered if a DLX is configured and dropped otherwise. Since 4.3 only real failures, such as basic.reject or a lost connection, are counted, and basic.nack is not. That is why the documentation recommends a dead-letter configuration for every quorum queue.

What problem does the outbox pattern solve?

When you need to both save data in the database and publish a message, the outbox writes the message to a table in the same transaction, and a separate process publishes it. The message is published only if the transaction commits, but it may arrive more than once, so the consumer must be idempotent.

More in this section

• Caching in ASP.NET Core: IMemoryCache, Redis and HybridCache
• API load testing with Locust for ASP.NET Core
• SQL Server performance tuning for high-traffic APIs