RabbitMQ Data Pipelines: Millions of Records a Day

Every data pipeline works fine until the day it doesn't. For a verification workload I re-architected recently, that day arrived at about 40,000 records a day. Nothing was broken. The server had CPU to spare. It was just doing one heavy, I/O-bound job at a time, in a single line, and no single line runs fast enough to hit the volume the business suddenly needed.

The fix wasn't a bigger machine. It was a RabbitMQ data pipeline: a queue between the code that creates work and a pool of workers that do it. Same logic, same servers, but the work now ran in parallel. Throughput went from tens of thousands of records a day to more than two million. This is the general pattern behind that jump, and how to tell when a message queue is the right tool for the job.

link_off The real bottleneck: work welded to the request

In a synchronous pipeline, the thing that receives or generates work also does the work. A record arrives, the same process validates it, calls an API, waits, writes the result, then reaches for the next one. While it waits on that I/O, it does nothing else. Throughput is capped by how fast one process can grind through a backlog it is holding in memory.

That design has two failure modes. It can't use more than one machine, so your only lever is a bigger box, which runs out fast and gets expensive. And a burst of input has nowhere to sit, so the process either falls behind or falls over.

A message queue breaks the weld. The producer's only job becomes "describe the work and hand it off." Everything after that happens somewhere else, on its own schedule.

hub The pattern: producer, queue, worker pool

RabbitMQ calls this a work queue, and it is the single most useful pattern in the whole system. A producer publishes messages to a queue. Several workers subscribe to that same queue. RabbitMQ hands each message to one worker, round-robin by default, so the workers share the load without knowing about each other.

The producer side is small. Declare a durable queue, publish a persistent message, done.

import pika
 
conn = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = conn.channel()
 
# durable=True so the queue definition survives a broker restart
channel.queue_declare(queue="records", durable=True)
 
channel.basic_publish(
    exchange="",
    routing_key="records",
    body=record_json,
    properties=pika.BasicProperties(delivery_mode=2),  # persist the message to disk
)
conn.close()

The worker side is where the real pipeline logic lives. Each worker is an identical process. Run one, run fifty. They all read from the same records queue.

import pika
 
conn = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = conn.channel()
channel.queue_declare(queue="records", durable=True)
 
# hand this worker one message at a time (see "backpressure" below)
channel.basic_qos(prefetch_count=1)
 
def on_message(ch, method, properties, body):
    process(body)                                    # the slow, I/O-bound work
    ch.basic_ack(delivery_tag=method.delivery_tag)   # ack only after it succeeds
 
channel.basic_consume(queue="records", on_message_callback=on_message)
channel.start_consuming()

Two lines carry most of the reliability: prefetch_count=1 and the basic_ack after process returns. Both matter enough to get their own sections below.

groups Scale by adding workers, not servers

Here is the part that felt like magic the first time. To double throughput, you don't tune anything. You start more workers. Each new process is another consumer on the same queue, and RabbitMQ spreads messages across all of them. This is scaling out, with more workers, instead of up, with a bigger machine. For an I/O-bound pipeline that is both cheaper and far less limited.

Because the workers are stateless and identical, they run anywhere: more processes on one host, or spread across a fleet. On Kubernetes you can make it automatic with an autoscaler that watches queue depth rather than CPU. When the backlog grows past a threshold it launches more worker pods; when the queue drains it scales them back down. That one rule, scale on queue length and not CPU, is the thing most teams get wrong.

The workload I opened with went from roughly 40K to 2M+ records a day mostly by moving the heavy per-record I/O off the request path and onto a fleet of workers pulling from RabbitMQ. Same code doing the checks. The queue just let a lot of them happen at once.

shield Don't lose a single record: durability and acks

A queue that loses messages is worse than no queue, because you trust it. RabbitMQ can be genuinely durable, but only if you set four things together. Miss one and there is a silent hole.

1. Durable queue (durable=True): the queue definition survives a broker restart.
2. Persistent messages (delivery_mode=2): the messages themselves are written to disk, not just held in memory.
3. Manual acknowledgements: a worker says "done" only after the work succeeds. If it dies mid-task, the unacked message is redelivered to another worker.
4. Publisher confirms: the broker tells the producer it actually persisted the message, so a publish that quietly failed can be retried.

The rule that ties them together: acknowledge work only after the result is safely handled, never the moment you receive it. In the pipeline above, a worker acked a record only once its result had been published downstream and confirmed. Ack too early and a crash erases work you promised to do. This is what let the system claim zero lost records across worker crashes, with no separate database tracking in-flight state. The broker was the ledger.

replay Handle the messages that fail

Some records will fail no matter how many times you retry: malformed input, a row that trips a bug, a dependency that is gone for good. Requeue them blindly and they loop forever, clogging the pipeline. That is a poison message, and the fix is a dead-letter queue.

You tell the main queue to route rejected messages to a separate dead-letter exchange. Transient failures get retried a few times. Anything that still fails is nacked without requeue, so it lands in the dead-letter queue for a human to inspect instead of poisoning the fleet.

# a separate exchange + queue for messages that cannot be processed
channel.exchange_declare("dlx", exchange_type="direct", durable=True)
channel.queue_declare("records.dead", durable=True)
channel.queue_bind("records.dead", "dlx", routing_key="records")
 
# the main queue forwards rejects to the dead-letter exchange
channel.queue_declare(
    queue="records",
    durable=True,
    arguments={
        "x-dead-letter-exchange": "dlx",
        "x-dead-letter-routing-key": "records",
    },
)
# on an unrecoverable message:
#   ch.basic_nack(delivery_tag=method.delivery_tag, requeue=False)

A dead-letter queue that is filling up is also your best early warning that something upstream changed.

speed Backpressure: the small setting that keeps it stable

prefetch_count=1 looked like a throwaway line earlier. It is actually the flow-control valve. Prefetch is how many unacknowledged messages RabbitMQ will push to a worker at once. Leave it at the default and a fast broker shoves hundreds of messages at a single worker, which then sits on a backlog it can't clear while other workers starve. Worse, if that worker dies, every message it was hoarding gets redelivered.

Set prefetch low (1 for long tasks, a small number for short ones) and RabbitMQ only gives a worker the next message when it is ready. Slow workers naturally pull less, fast workers pull more, and the queue itself absorbs bursts. That is backpressure, and it is most of what keeps a busy pipeline from tipping over.

compare_arrows RabbitMQ or Kafka? Match the tool to the work

This is the question that derails most "which message system" debates, so here is the short of it. RabbitMQ is a message broker built to hand discrete tasks to workers. Kafka is a distributed log built to stream and retain ordered events. They overlap, but they are good at different jobs.

For a pipeline whose job is "take these records and process each one," RabbitMQ's work-queue model fits like a glove, and it is simpler to run. If you also need to replay the last week of events into three different consumers, that is Kafka's world. Plenty of teams reach for Kafka by reflex when a work queue was the honest answer. Pick by the shape of the work, not the hype.

engineering What this looks like in production

The verification pipeline is one example. The other is a dynamic ETL pipeline I built on AWS that moves millions of records a day off external APIs into a warehouse. Different domain, same backbone idea: decouple the stages, let each scale on its own, and never hold in-flight work only in memory where a crash can lose it.

Two lessons carried across both. First, the broker is your reliability layer, not just a pipe. Durable queues plus ack-after-work replaced what would otherwise have been a database of in-flight state. Second, throughput is a pacing problem more than a compute problem. Once the work is parallel, the game becomes feeding workers steadily without burying them, which is exactly what prefetch and queue-depth autoscaling are for.

If you are deciding how to orchestrate the stages around a queue, a broker pairs with a scheduler rather than replacing it. I weighed the main options in Dagster vs Airflow vs Prefect.

quiz Frequently asked questions

rocket_launch The takeaway

Scaling a pipeline rarely means writing faster code. It usually means changing the shape of the work so more of it can happen at once, then feeding that parallelism without drowning it. A message broker does exactly that. It decouples who makes the work from who does it, holds the backlog safely, and lets you add capacity by adding workers. Get durability and prefetch right and the same pipeline that stalled at tens of thousands a day clears millions.

If you have a pipeline hitting a ceiling and you are not sure whether the fix is a queue, a re-architecture, or just better backpressure, that is the kind of work I do. Or tell me what you are running and I will take a look.