Skip to content
All Projects
Data EngineeringHigh-Throughput

Scaling Data Pipeline with RabbitMQ

Scaling a verification workload from ~40K to 2M+ records a day with an event-driven worker fleet

Re-architected a bottlenecked, per-request processing system into a distributed, message-driven pipeline on RabbitMQ. A fleet of stateless async workers offloads the heavy I/O off the request path, lifting sustained throughput from tens of thousands of records a day to more than two million, with zero lost work across crashes.

Python 3.12asyncioRabbitMQaio-pikaMongoDBMySQL 8httpxdnspythonStreamlit
Problem Statement

A high-volume data-verification platform was capped at tens of thousands of records a day. Verification ran a full browser session per record, synchronously, so throughput was bounded by how many heavy, I/O-bound checks a single process could grind through in sequence.

  • A per-request browser-automation step made every check heavy, synchronous and slow.
  • Throughput hit a hard ceiling near 30 to 40K records a day and could not grow with demand.
  • A crash or restart risked losing in-flight work, with no durable record of what was already done.
Headline Outcomes
2M+/day~50×

Daily throughput

~30–40K/day

Zerodurable settlement

Lost records across crashes & outages

Horizontalmore workers, more throughput

Scaling model

The Solution

A message-driven micro-worker architecture with RabbitMQ as the single durable backbone. Work, results and telemetry each travel on their own queues, and a horizontally-scalable fleet of stateless async workers consumes them, so adding workers adds throughput and the heavy I/O runs off the request path.

RabbitMQ carries three separated flows (work, results and events), decoupling producers from a fleet of stateless consumers.

Heavy per-record I/O is offloaded to async workers, so throughput scales with worker count instead of one process.

Exactly-processed settlement: durable queues, persistent messages, publisher confirms and ack-after-downstream-confirm mean a result is never lost and never acknowledged before it lands.

A prefetch of one message per worker adds backpressure, and a dedicated retry queue re-queues rate-limited work, so a slow or crashed worker never hoards or drops records.

Polyglot persistence keeps hot operational state in MongoDB and analytics in a separate MySQL warehouse fed by an event stream, so the analytics plane can never slow production.

System Architecture

How the data flows

01

Enqueue

Durable RabbitMQ work queue

02

Worker Fleet

Stateless async, prefetch 1

03

I/O Offload

Heavy work off request path

04

Settle

Confirms + retry, zero loss

05

Event Stream

MySQL rollups + dashboard

Result 01

Broke a hard throughput ceiling, lifting sustained processing from tens of thousands to millions of records a day.

Result 02

Guaranteed zero lost work across worker crashes and datastore outages through broker-anchored settlement.

Result 03

Delivered elastic, horizontal scale, so throughput grows with worker count rather than a single process.

Build Something Like ThisServices & PricingMore Case Studies
Further reading

From the blog

Data Engineering

AWS Lambda 15-Minute Timeout and How to Work Around It

Hit AWS Lambda's 15-minute timeout? Durable functions did not raise it. Which of the three fixes you need depends on whether your job computes or waits.

AWS LambdaServerlessAWSData Engineering
Read Article13 min read
Career

How to Manage Multiple Projects as a Developer

How to manage multiple projects as a developer: WIP limits, project-day blocking, landing notes, and pointing AI at re-entry cost instead of writing more code.

ProductivityDeep WorkContext SwitchingDeveloper Experience
Read Article10 min read
Data Engineering

How to Build Your First ETL Pipeline in Python

New to data engineering? Build your first ETL pipeline in Python: pull data from an API, clean it with pandas, and load it into a database, the right way.

ETLPythonData EngineeringPandas
Read Article13 min read
Taking on new projects · Outside IR35

Have a data pipeline or warehouse problem worth solving?

From messy source data to analytics-ready warehouses that cut cost. Let's scope it. I reply within one business day.

Start a ProjectConnect on LinkedIn