What Building a Job Queue From Scratch Actually Teaches You

The real lesson isn't the code — it's what breaks

Building a production-grade async job queue from scratch in Python, FastAPI, and Redis Streams sounds like a "why would you do that" decision. Celery exists. RQ exists. Dozens of battle-tested options exist. But here's the insight worth internalizing: the engineer who builds a queue from scratch understands distributed systems in a way that library users simply don't. Every abstraction Celery hides — backpressure, worker recovery, priority starvation, job reaping — becomes a bug you write yourself before you understand why it matters.

This is the real value of reading a build log like this one: not to copy the implementation, but to internalize the checklist of things that can go wrong.

---

Why "just use Redis" is only the beginning

Most developers' mental model of a job queue is:

1. Push job into Redis

2. Worker pulls job out

3. Job runs

That works in a demo. Production adds a few complications:

• What if the worker crashes mid-job?
• What if producers enqueue 10,000 jobs per second and workers process 100?
• What if one slow job type monopolizes all workers and starves urgent work?
• What if a job fails repeatedly and poisons the queue forever?

Redis Streams (as opposed to plain Redis lists) actually gives you primitives to handle some of this — consumer groups, pending entry lists (PEL), and acknowledgment semantics. But the primitives don't hand you a working system. You have to wire them together correctly.

1# Simplified: reading from a Redis Stream consumer group
2messages = await redis.xreadgroup(
3    groupname="workers",
4    consumername="worker-1",
5    streams={"jobs:high": ">"},  # '>' means only new, undelivered messages
6    count=10,
7    block=2000  # block for 2s if queue is empty
8)

Notice what happens on failure: you simply don't acknowledge. The message stays in the Pending Entry List. But who reclaims it? Nobody — unless you build a Reaper.

---

The Reaper: the service nobody tells you to build

A Reaper is a background process that periodically scans for jobs that have been "in flight" too long — claimed by a worker that has since crashed, stalled, or died — and reclaims them for retry. It's the difference between a queue that recovers from worker failure and one that silently loses work.

1# Simplified Reaper logic
2async def reap_stale_jobs(redis, stream, group, stale_after_ms=30_000):
3    pending = await redis.xpending_range(
4        stream, group,
5        min="-", max="+",
6        count=100
7    )
8    now = int(time.time() * 1000)

Without this, a single worker crash during a high-load period can leave dozens of jobs orphaned in the PEL indefinitely. They won't fail visibly. They just... disappear.

---

Backpressure: the feature that protects everything else

Backpressure is the mechanism that makes a producer slow down when consumers are saturated. Without it, a spike in traffic means your queue depth grows unbounded, your Redis memory climbs, and eventually your whole system degrades under the weight of work it will never finish fast enough.

Implementing backpressure in a custom queue typically means:

• Setting a maximum queue depth per priority tier
• Returning a 503 / retry-after response (or raising an exception) when the queue is full rather than accepting the job
• Optionally, implementing token bucket or leaky bucket rate limiting on the producer side

1async def enqueue_job(job_data: dict, priority: str = "normal"):
2    stream_key = f"jobs:{priority}"
3    queue_length = await redis.xlen(stream_key)
4
5    if queue_length >= MAX_QUEUE_DEPTH[priority]:
6        raise QueueFullError(
7            f"{priority} queue at capacity ({queue_length} jobs). Try later."
8        )
9
10    await redis.xadd(stream_key, job_data)

This is simple but powerful. The alternative — accepting every job no matter what — feels more "reliable" until your queue has 500,000 unprocessed jobs and your SLA is on fire.

---

Priority scheduling and the starvation trap

Priority queues seem straightforward: always pick from the high-priority queue first. But naive implementations starve low-priority work entirely during sustained high load. A user-triggered job that came in at "normal" priority might wait forever because "high" jobs keep arriving.

The fix is weighted fair scheduling or aging: low-priority jobs that have been waiting long enough get temporarily promoted. It's a small design decision with big operational consequences.

---

47 tests and 85% coverage: what that actually signals

Queue bugs are concurrency bugs. They show up under:

• simultaneous enqueue and dequeue
• worker crashes during job execution
• Redis connection drops mid-acknowledgment
• rapid priority shifts under load

You cannot manually test these. 47 tests and 85% coverage for a queue implementation is the minimum signal that the author took correctness seriously. The remaining 15% is almost certainly the hardest-to-test paths: network partitions, partial writes, race conditions in the Reaper. That's not laziness — that's an honest acknowledgment of what automated tests can't easily cover.

---

Why this matters for your work

You probably won't build a queue from scratch. But you will use one — or you'll evaluate one — and you'll need to know what questions to ask:

• Does it handle worker failure? Ask about the Reaper equivalent.
• Does it have backpressure? Or will it accept infinite work until Redis OOMs?
• How does it handle priority starvation? Does low-priority work ever complete during peaks?
• What are the acknowledgment semantics? Can a job be lost between dequeue and ack?
• What does the dead-letter queue look like? How do you inspect and replay poison jobs?

If you're using Celery, know that it answers most of these — but with defaults that may not match your assumptions. If you're using Redis Streams directly, you're responsible for building the Reaper yourself.

And if you ever want to deeply understand a distributed system primitive, build it once from scratch. Not for production — for understanding. The bugs you write are the concepts you'll never forget.