Your Async Runtime Is a First-Class Performance Variable, Not a Free Abstraction

The lesson that took 2.5 million events per second to learn

Here's the insight you can take from this post-mortem without reading the whole thing: async is not free, and at scale, the runtime itself becomes part of your performance budget. That sounds obvious in retrospect, but it's the kind of thing that only really lands when you've watched a system fall over at 2.5M events/sec while your CPU sits at 30% and you've already ruled out the usual suspects.

The author inherited a Rust-based game event engine—think treasure-hunt mechanics, millions of in-game events, real-time state updates. Benchmarks looked clean: 3 ms median latency, 12 ms p99. But every time traffic crossed 2.5 million events per second, the JVM control-plane ground to a halt. The instinct was to blame GC pressure. That was wrong.

The real culprit was async runtime overhead.

---

Why this confusion happens

There's a mental model most developers carry that goes something like: async = faster because no blocking threads. And in a lot of contexts that's true enough. For I/O-bound workloads with thousands of concurrent connections, an async runtime genuinely wins—it multiplexes efficiently, avoids the memory overhead of thread-per-connection, and keeps the CPU busy.

But that model breaks down when the workload shifts. This engine wasn't waiting on sockets or database calls. It was processing events at massive throughput—which is fundamentally compute-bound work dressed up as I/O orchestration. And compute-bound work has a different relationship with an async executor.

At 2 million events/sec, the runtime overhead was a rounding error. Cross 2.5 million, and suddenly task wakeups, queue contention, and executor scheduling become measurable costs. The CPU wasn't idle because the system was relaxed—it was idle because the scheduler was spending cycles on runtime bookkeeping rather than application work.

---

The async/thread tradeoff in concrete terms

Rust's async model requires an executor—Tokio being the most common—to drive futures to completion. Without it, nothing actually runs. That executor is doing real work:

• Polling futures and deciding which ones are ready
• Managing task wakeups when I/O events arrive
• Handling task migration across worker threads
• Allocating and deallocating task state on the heap

For a system doing 2M+ events/second, those aren't negligible. Consider what a tight event loop might look like in an async context versus a thread-based one:

1// Async approach: every event goes through the executor
2async fn handle_event(event: GameEvent) -> Result<(), Error> {
3    let state = fetch_state(&event.player_id).await;  // wakeup cost
4    let outcome = compute_outcome(event, state);       // CPU work
5    persist_outcome(&outcome).await;                   // wakeup cost
6    Ok(())
7}
8

In the async version, every .await point is a potential wakeup, a potential task migration, a potential scheduler decision. At 2.5M events/sec those "potentials" compound. The thread-based version has its own costs—context switching, synchronization—but those costs are predictable, well-understood, and don't grow in the same non-linear way under throughput pressure.

This doesn't mean async is wrong. It means the concurrency model has to match the workload shape.

---

The diagnostic trap: low CPU, poor throughput

One of the more instructive parts of this story is the symptom profile: 30% CPU, 512 MB RSS, and still hitting a wall. That combination is a classic sign that the system isn't compute-bound in the obvious sense—it's not maxing out cores on hot application code. It's spinning on infrastructure.

When you see that pattern, the questions to ask are:

• What's the wakeup rate? If futures are waking up faster than they can be processed, your executor queue grows and latency spikes without CPU going up.
• What's the task allocation rate? Each spawned task allocates. At high concurrency that adds up and can cause subtle heap pressure.
• Is there lock contention in the runtime itself? Tokio's work-stealing scheduler is good, but it's not immune to contention when task counts are very high.
• Are tasks migrating between threads frequently? Task migration is necessary for work-stealing but has cache-coherency costs that show up at extreme throughput.

None of this shows up in standard application-level profiling. You have to instrument the runtime.

---

Practical design adjustments

If you're building or maintaining a high-throughput event pipeline, here's what this post-mortem actually implies in practice:

Keep async scope tight. Don't make the entire program async just because one part needs it. Async is for concurrent I/O waits. If your event processing is mostly computation with occasional I/O, only the I/O layer should be async.

Separate concerns explicitly. A common and effective pattern: run your I/O layer (ingestion, network, persistence) on an async runtime, and hand off compute-heavy processing to a dedicated thread pool using spawn_blocking or a channel-based worker pool.

1// Hybrid: async for I/O, thread pool for compute
2async fn ingest(event: RawEvent, tx: Sender<GameEvent>) {
3    let parsed = tokio::task::spawn_blocking(move || {
4        parse_and_validate(event)  // CPU work, off the async executor
5    }).await?;
6    tx.send(parsed).await?;
7}

Profile the runtime, not just the app. Tools like tokio-console exist specifically to surface executor-level metrics: task counts, poll times, wakeup rates. If you're at scale and haven't used it, you're flying partially blind.

Validate at production-like load. The benchmarks looked fine. They were fine—at 2M events/sec. The cliff appeared at 2.5M. Benchmarks that don't exercise the scaling boundary will miss runtime costs that only emerge at high task concurrency.

---

Why this matters beyond this one story

The broader principle here applies well past Rust. Any system with a managed execution model—Node.js event loop, Go runtime scheduler, JVM thread pools—has runtime costs that are load-dependent and non-linear. The mistake isn't choosing async or a managed runtime. The mistake is treating the runtime as a transparent layer rather than a component with its own performance characteristics.

At low and moderate load, that abstraction holds. At scale, it leaks. And the systems that perform reliably at scale are the ones where the engineers understood what the runtime was actually doing and designed around it—not through it.

The author spent three years blaming the wrong thing. The good news: once they stopped looking at the application and started looking at the infrastructure under it, the path forward was clear. That reframe—from "why is my code slow" to "what is my runtime doing"—is one of the more valuable shifts you can make as a systems engineer.