"Does It Scale?" Is Three Questions Pretending to Be One

Someone asks “does it scale?” in a design review and everyone nods, and I’ve stopped trusting the nod. Usually each person in the room is picturing a different thing, and the system was only ever tested for one of them.

This is stop 1 in the Scalable Backends series. Every post threads through AgentDesk, a fictional platform where customer teams deploy AI agents that chat with their customers, call tools and run long sessions. It’s about to have a very good, very bad week.

The night AgentDesk goes viral

Picture it. A customer’s support agent gets shared everywhere, and overnight AgentDesk goes from about 100 concurrent sessions to about 100,000. By morning three different people are telling you three different things are wrong.

The product lead says it’s slow: customers wait eight seconds for an answer that used to take two. The platform engineer says it’s fine, look, agent turns per second are at an all-time high. And the on-call engineer says nothing, because they’re busy figuring out why a few thousand sessions just vanished when two orchestrator pods restarted at 3am.

All three are right. They’re just answering different questions.

The three questions

Latency is how long one request takes, start to answer. For AgentDesk the number that matters is p99 time per agent turn, including the tool calls inside it, because the slowest 1% is what customers actually remember.

Throughput is how much work you finish per unit of time. Here that’s agent turns per second across the whole fleet.

Availability is whether the system still answers when a piece of it dies. For an agent platform it’s a bit stricter than “is the endpoint up”: a session that loses its context mid-task is unavailable to that customer even if every health check is green.

Why you can’t max all three

They pull against each other, and the pull isn’t subtle.

Batch your model calls and throughput goes up, since the GPU does more useful work per trip, and every single user waits a little longer for their answer because their turn now sits in a queue until the batch fills. That’s AgentDesk’s product lead and platform engineer looking at the same change and both being correct.

Add aggressive retries to protect availability and you can turn one bad minute at the model provider into twice the load, right when you can least afford it, which hurts latency for everyone and can knock over the thing you were retrying against.

Keep everything in memory on one instance to make latency great, and you’ve quietly made availability depend on that instance never restarting.

None of these are bad decisions. They’re trades, and the trouble starts when nobody says out loud which side of the trade they picked.

Why agent backends feel it first

A typical web request is over in a few hundred milliseconds and remembers nothing, so it’s easy to get away with only thinking about throughput.

Agent turns don’t work like that. One turn can run for minutes while the model thinks, tools get called and sub-agents fan out, and the session holds state the whole time. So AgentDesk can post record throughput numbers and still drop a customer’s half-finished task the moment the pod holding it restarts. That failure never shows up on the throughput chart, which is exactly why the 3am pod restarts surprised everyone.

Long turns also make latency lumpy in a way averages hide. Most turns are quick, a few take forever, and the p99 is where the angry support tickets come from.

What I’d ask in any design review

Instead of “does it scale?”, I’d ask which of the three a change is for, and what it costs the other two. Something like:

• Which one is this change trying to improve?
• What does it do to the other two, even a little?
• How would we notice if one of them got worse? Is there a chart for it?

If nobody can answer the third question, that’s usually the one that breaks first.

What this sets up

The rest of the series keeps coming back to these three. Next up is load balancing, where AgentDesk’s long sessions run straight into the oldest trick in the book, round-robin, and the latency side of this trade gets very concrete: Where does turn 12 go?

When something broke in your system, which of the three was it? I’m curious whether it’s the same one for everybody.