Where Does Turn 12 Go? Load Balancing Agent Sessions

I keep coming back to turn 12 of an agent session.

By turn 12 a session is carrying a lot: the whole conversation so far, whatever tools it loaded, maybe a streaming connection that’s still open to the customer’s chat window. And the most “scalable” load balancer setup there is, plain round-robin, hands that turn to whichever orchestrator instance happens to be next. Usually that’s a cold one, and it has to reload all of that context before it can say a single word back.

This is stop 2 in the Scalable Backends series, where AgentDesk, a fictional agent platform, grows from about 100 to 100,000 concurrent sessions. Stop 1 split “does it scale?” into latency, throughput and availability; this one is mostly about latency, and what protecting it costs availability. The overview post made one decision without fully defending it: route by session id at the gateway instead of round-robin. This post is that defense, plus the part the overview skipped, which is how pinning fights back.

Round-robin is fair, and that’s the problem

Round-robin is the friend who splits the bill evenly when you only had chai. Every instance gets the same number of requests, which sounds great until you notice the requests aren’t the same size.

For a normal web API that’s fine. A request comes in, it’s handled, nothing is remembered, and any instance can take the next one. An agent session isn’t one request though, it’s a long conversation where every turn depends on everything before it. When turn 12 lands on an instance that has never seen this session, the orchestrator has to go fetch history from Conversation History, rebuild the context window, reload tool definitions, and only then call the model.

That reload is the tax, and it gets worse the longer the session runs. Latency stops being flat and starts growing with session length, which is exactly backwards from what you want on a platform whose whole selling point is long, useful sessions.

sequenceDiagram
  participant C as Customer chat
  participant G as API Gateway
  participant A as Orchestrator A (warm)
  participant B as Orchestrator B (cold)
  participant H as Conversation History
  C->>G: turn 11
  G->>A: round-robin picks A
  A-->>C: answers right away
  C->>G: turn 12
  G->>B: round-robin picks B
  B->>H: load history, tools, context
  H-->>B: everything from turns 1-11
  B-->>C: answers, eventually

  

L4 can’t help you here, L7 can

A layer 4 load balancer sees connections: an IP, a port, some bytes. It can spread connections around, but it has no idea that two requests belong to the same agent session, so it can’t keep them together on purpose.

A layer 7 load balancer reads the actual request, which means it can look at a header or a path segment carrying the session id and make a decision based on it. That’s where AgentDesk’s API Gateway sits, and it’s what makes “send this session back to the instance that already has it warm” possible at all.

There are two common ways to do it.

Sticky sessions. The gateway remembers which instance served a session (usually with a cookie or a routing table) and keeps sending it there. Simple, and it works out of the box on most L7 load balancers.

Consistent hashing on the session id. Instead of remembering, the gateway hashes the session id onto a ring of instances, so the same id always lands on the same instance with no lookup table. The nice part shows up when you scale: adding an instance only moves the slice of sessions that hash next to it, not everyone. With a naive hash(id) % N, going from 10 to 11 instances reshuffles almost every session at once, and every one of them pays the cold reload in the same minute.

Either way, turn 12 goes back to the instance that handled turn 11, and the reload tax mostly disappears.

And then the pinning fights back

Pinning is faster right up until something about the fleet changes, and in a real system the fleet is always changing.

Picture a deploy draining orchestrator 3 in the middle of someone’s 40-turn support chat. Every session pinned to that instance either goes cold on whichever instance picks it up next or, if the streaming connection was held there, just drops. The customer sees their agent forget everything or disconnect mid-sentence, during a routine deploy that nobody thought of as risky.

Load stops being even, too. Pinning spreads sessions, not work, and agent sessions are wildly uneven: one chatty enterprise session can keep its instance busy all afternoon while the instances next to it sit mostly idle. Round-robin would have smoothed that out. Pinning can’t, because moving the session is the thing it exists to avoid.

Autoscaling gets quieter about it but it’s the same issue. When AgentDesk scales out during a spike, the new orchestrator pods only get new sessions. The busy pods stay exactly as busy as they were, holding their pinned sessions, so the scale-out helps much less than the dashboard suggests. (That cold-start story gets its own post in Level 4.)

Where AgentDesk lands

AgentDesk keeps the pinning, but stops treating the instance as the only place a session can live.

Session context, the working state an orchestrator needs to pick up a session mid-conversation, lives in the Session State Store (Azure Managed Redis), with the full record in Conversation History for durability. The gateway still routes by session id, so on a normal turn the session lands on a warm instance and never touches Redis on the hot path. When that instance goes away, whether it’s a deploy, a crash or a scale-in, the next instance reads the context from the shared store instead of rebuilding it from the full history, and the customer mostly doesn’t notice.

flowchart LR
  C[Customer chat] --> G[API Gateway
L7, routes by session id]
  G -->|normal turn| A[Orchestrator A
warm, pinned]
  G -.->|A drained or gone| B[Orchestrator B]
  A <--> S[(Session State Store
Azure Managed Redis)]
  B --> S
  S --> H[(Conversation History
durable record)]

  

That changes what the pin means. It’s a cache hint now, a way to make the common case fast, instead of something the session’s survival depends on. Draining an instance for a deploy becomes boring again: stop sending it new sessions, let in-flight turns finish, write state, and let the next turn land wherever the ring says.

It isn’t free. Every turn now writes its working state somewhere shared, Redis becomes something you have to size and keep healthy, and you still have to decide what “context” means for each session (the full window, or a compact summary that’s cheaper to move). For long, chatty sessions that trade is worth it. For short, mostly stateless requests, plain round-robin is still the right answer, and it’s worth saying that out loud before anyone pins everything by default.

What this sets up

Caching is next in Level 1, and it picks up right where this leaves off: once session context lives in a shared store, the hard question stops being “where does turn 12 go?” and becomes “is the copy turn 12 reads still correct?”

If you’re routing long-running agent sessions today, I’d genuinely like to know what you landed on. Sticky, a shared store, something smarter, or round-robin and a lot of patience?