Unbounded Queues Are Memory Leaks in Disguise

Here's the counterintuitive truth: When your system is drowning in traffic, adding a queue is like trying to fix a flood by building a bigger reservoir upstream. It doesn't solve the problem—it just delays and amplifies the eventual collapse.

I've seen this pattern destroy production systems countless times. Traffic spikes hit, engineers panic and add queues to "absorb the load," and suddenly what should have been a 5-minute blip becomes hours of downtime with mysterious memory exhaustion.

The Physics of System Overload

The problem isn't queues themselves—it's unbounded queues. They violate what you might call the "physical laws" of distributed systems.

Here's why: Little's Law states that concurrency = arrival_rate × average_latency. When your database starts choking and response times jump from 100ms to 10 seconds, your thread pool usage explodes from 10 to 1,000 threads at the same request rate. Your system starves itself.

1// This is a recipe for disaster
2class UnboundedQueue<T> {
3  private items: T[] = [];
4  
5  enqueue(item: T): void {
6    this.items.push(item); // No limit = eventual OOM
7  }
8  

The Latency Death Spiral

Unbounded queues create a vicious cycle I call the "latency death spiral":

1. Traffic spike hits → Database/downstream service slows down

2. Queues start growing → Everything "looks fine" in monitoring

3. Caches empty → Cold cache misses increase load further

4. Memory pressure builds → GC pauses make everything worse

5. System becomes unresponsive → Even when traffic drops, the backlog keeps you down

The cruel irony? Your system stays broken long after the original spike ends because it's still churning through the massive queue backlog.

What Actually Works: The Three-Pronged Approach

Successful systems combine three strategies:

1. Bounded Queues with Explicit Limits

1import queue
2import threading
3
4class BackpressureQueue:
5    def __init__(self, max_size: int):
6        self.q = queue.Queue(maxsize=max_size)
7        self.rejected_count = 0
8    

2. Load Shedding at the Ingress

Drop requests early when you detect overload. It's harsh but necessary:

1class LoadShedder {
2  private recentLatencies: number[] = [];
3  
4  shouldShed(currentLoad: number): boolean {
5    const avgLatency = this.getAverageLatency();
6    const latencyThreshold = 5000; // 5 seconds
7    const loadThreshold = 0.8;
8    

3. Explicit Backpressure Propagation

When downstream services are overwhelmed, slow down the upstream callers instead of buffering infinitely:

1type BackpressurePool struct {
2    semaphore chan struct{}
3    timeout   time.Duration
4}
5
6func (p *BackpressurePool) Execute(task func()) error {
7    select {
8    case p.semaphore <- struct{}{}: // Acquire
9        defer func() { <-p.semaphore }() // Release
10        task()
11        return nil
12    case <-time.After(p.timeout):
13        return errors.New("backpressure: timeout acquiring slot")
14    }
15}

The Hard Truths About Production

Implementing proper backpressure is politically challenging. Stakeholders hate hearing "we're dropping requests," but the alternative—total system failure—is worse.

The key insight: queues are for handling temporary mismatches between supply and demand, not for infinite buffering. When demand consistently exceeds supply, you need capacity planning, not bigger queues.

Monitoring That Actually Helps

Track these metrics to catch queue problems early:

• Queue depth over time (not just current size)
• Age of oldest item in queue
• Rejection/drop rates
• End-to-end latency percentiles

Set alerts when queue depth trends upward or oldest item age exceeds acceptable bounds.

Why This Matters

Unbounded queues are seductive because they make problems disappear temporarily. But they're technical debt that compounds with interest. When that traffic spike hits at 3 AM, you'll either have a system that gracefully sheds load and recovers quickly, or you'll have one that falls over completely and stays down for hours.

The next time someone suggests "just add a queue" to handle load, remember: queues are tools for smoothing, not magic wands for capacity. Bound them, monitor them, and always have a plan for when they fill up.

Your move: Audit your existing queues. Find the unbounded ones. Fix them before they fix you.