AgentDesk: The System Behind the Scalable Backends Series

Every post in the Scalable Backends series threads through one system instead of a dozen disconnected toy examples: AgentDesk, a fictional platform where customer teams deploy AI agents that handle customer chats, call external tools, run long sessions, and spin up sub-agents for parallel research. It’s scaling from roughly 100 to 100,000 concurrent sessions — which is exactly the kind of growth that forces every decision below.

This post is the reference diagram the rest of the series points back to. Click any box for details — ownership, technology, and design notes.

(The diagram is a self-contained interactive page — click a component to open its detail drawer. It renders in its own light theme by design, generated by Farah’s Architecture Agent rather than hand-drawn. The preview above is cropped for space — open the full page to see the whole canvas and use the highlight filters.)

Why AgentDesk looks the way it does

None of the decisions baked into this baseline are arbitrary — each one exists because an earlier, simpler version of AgentDesk broke under load in a specific way. Here’s the problem behind each one, and the general lesson it’s meant to teach.

Session affinity, not statelessness

The problem: Early on, AgentDesk’s orchestrator was stateless by the book — any instance could handle any request, and session context lived only in the database. That’s the textbook “scalable” design. In practice, it meant every single agent turn paid a full round-trip to reload conversation history and tool state from Cosmos DB before it could do anything, because the instance that happened to pick up the next request usually wasn’t the one that handled the last one. Latency scaled with session length instead of staying flat.

The fix: Route by session id at the gateway (L7, not round-robin L4) so a session stays pinned to the orchestrator instance that already has its context warm in memory. The database write is still there for durability, but it’s no longer on the hot path for every turn.

What it teaches: “Stateless is more scalable” is true in general, but it’s not free — you’re trading compute-side simplicity for a data-layer tax on every request. Long-lived, chatty sessions are exactly the case where that tax adds up fastest, and where session affinity (with a plan for what happens when an instance dies) usually wins. This is the subject of the load-balancing post later in the series.

One event bus, four independent consumers

The problem: The first version of task completion had the orchestrator call billing, webhooks, and analytics directly, one after another, before returning. It worked fine in testing. In production, one customer’s webhook endpoint started timing out intermittently — and because the orchestrator waited on it synchronously, billing updates for every customer started lagging behind it, not just that one customer’s.

The fix: The orchestrator publishes one task.completed event to the event bus and moves on. Billing, the webhook dispatcher, and the analytics pipeline each subscribe independently and process at their own pace. A slow or broken webhook endpoint now only delays that one delivery queue, not the entire platform.

What it teaches: Synchronous fan-out couples the reliability of your core path to the reliability of your least reliable downstream consumer, even when that consumer has nothing to do with the request that’s waiting. If a side effect isn’t needed to answer the current request, it doesn’t belong in the current request’s critical path.

Governance lives in the sandbox, not a policy doc

The problem: This is the same failure mode covered in the IRREGULAR containment-failure post — eval harnesses across several frontier labs assumed a sandboxed sub-agent would “stay in its lane” because the prompt told it to, and in a small fraction of long-running sessions, it didn’t. A sub-agent with real tool access and no hard infrastructure limit is one bad instruction (or one bad session) away from becoming a very expensive, very real problem.

The fix: The Sub-Agent Sandbox enforces a network-egress allowlist and a hard turn/spend budget cap at the runtime level — not as a system prompt instruction, but as something the infrastructure actually refuses to let the agent exceed.

What it teaches: Governance framed as “tell the model not to” is a suggestion, not a control. If a boundary matters, it has to be enforced somewhere the model’s own output can’t talk its way around — which puts it squarely in backend infrastructure design, not prompt engineering.

Why this baseline matters for the rest of the series

The point of using one running system instead of a fresh example per post is that these decisions don’t exist in isolation — they interact. Session affinity changes how you’d shard the conversation database. The event bus’s delivery guarantees determine whether the outbox pattern is optional or mandatory. The sandbox’s budget cap is also the mechanism the cost-control post relies on. Later posts will drill into specific slices of this same system: data-flow views for the caching and sharding posts, a sequence view for the outbox pattern, and a dedicated threat-model view for the sandbox itself.