The cluster scaled to zero before its first request

I gave a simulated cluster four warm replicas of Llama 3.1 70B on H200 GPUs and sent it 200 requests. It completed none of them.

The autoscaler ran its first check at time zero. No request had arrived, so it saw no load, and it drained all four replicas. The first request landed 50 milliseconds later and found nothing to route to. So did the other 199.

The simulator is BLIS. I filed the bug as inference-sim#2015 and sent a fix as inference-sim#1997. As of 31 October 2026, the fix is open and waiting for maintainer review.

The bug is a few lines. The lesson is general: an autoscaler that sizes a cluster to its demand will, at zero demand, size it to zero. Something has to set a floor.

What BLIS is

BLIS (Blackbox Inference Simulator) is a discrete-event simulator of an LLM serving stack. Its goal is parity with llm-d, an LLM serving stack for Kubernetes: the router, admission control, the prefill and decode split, KV caching, and autoscaling. Its README lists three jobs:

• Configuration search: which parallelism, engine limits and router settings serve a workload best.
• Capacity planning: "how many GPUs and instances a target rate and SLO need".
• Policy discovery: comparing routing, scheduling and autoscaling policies before they ship.

It runs on a CPU and needs no GPU. The same seed gives byte-identical results. Each run below finished in about a second on my machine.

If you plan capacity with a simulator, its autoscaler is part of your plan.

Reproduce it

You need Go 1.27.2 or later. Build upstream main at the commit I tested:

git clone https://github.com/inference-sim/inference-sim
cd inference-sim
git checkout 762f160
go build -o blis main.go

Write the policy from the issue. It sets up a pool of four H200 nodes with eight GPUs each, replicas warm at start, and an autoscaler that ticks every second:

cat > policy.yaml <<'EOF'
node_pools:
  - name: pool-h200
    gpu_type: h200
    gpus_per_node: 8
    gpu_memory_gib: 139.80072021
    initial_nodes: 4
    min_nodes: 1
    max_nodes: 4
    cost_per_hour: 30.0
instance_lifecycle:
  loading_delay:
    mean: 1.0
    stddev: 0.0
  warm_start_initial_instances: true
autoscaler:
  interval_us: 1000000
  analyzer:
    kv_cache_threshold: 0.8
    scale_up_threshold: 0.8
    scale_down_boundary: 0.4
    avg_input_tokens: 1000
EOF

Run Llama 3.1 70B at tensor parallel 4 on four replicas, with 200 requests at 20 per second, each 1,000 tokens in and 100 out:

LEN="--prompt-tokens 1000 --prompt-tokens-stdev 0 --prompt-tokens-min 1000 --prompt-tokens-max 1000 \
  --output-tokens 100 --output-tokens-stdev 0 --output-tokens-min 100 --output-tokens-max 100"
./blis run --catalog testdata/catalog --registry testdata/registry --scenarios testdata/scenarios \
  --scenario llama-3.1-70b-instruct-h200-tp4-4node.yaml --seed 11 --horizon 900000000 \
  --num-instances 4 --rate 20 --num-requests 200 $LEN --policy-config policy.yaml

The JSON result has "completed": 0 and "routing_rejections": 200. Add --log debug and you can watch it happen. Simulated time is in microseconds, so the first request arrives about 50 ms in, after all four drains:

[actuator] scale-down: draining instance instance_0 for model "llama-3.1-70b-instruct"
[actuator] scale-down: draining instance instance_1 for model "llama-3.1-70b-instruct"
[actuator] scale-down: draining instance instance_2 for model "llama-3.1-70b-instruct"
[actuator] scale-down: draining instance instance_3 for model "llama-3.1-70b-instruct"
[cluster] req request_0 arrived at tick 49999

To try the fix, merge the PR on top and rebuild:

git fetch origin pull/1997/head:fix-1997
git merge --no-edit fix-1997
go build -o blis main.go

I ran both configs from the issue on 31 October. The second row is the config of BLIS's own capability test: the same policy without the instance_lifecycle block, run with --num-instances 2 --rate 10 --num-requests 60.

With the fix, the debug log shows three drains, not four.

Nothing went missing on main either. BLIS counts every rejection, so its request conservation invariant still holds. The model just had no replica left.

Why it drained everything

Three pieces of code meet at the first tick.

1. Zero demand makes all supply spare. The analyzer is adapted from the saturation analyzer in llm-d's Workload Variant Autoscaler (WVA). It counts a replica's supply as its KV cache capacity times kv_cache_threshold. Demand is the KV tokens in use plus queued requests times avg_input_tokens. Then it decides whether scale-down is safe (saturation_analyzer.go):

spareCapacity := result.TotalSupply - (result.TotalDemand / a.config.ScaleDownBoundary)
// ...
supplyAfterRemoval := result.TotalSupply - maxPerReplicaSupply
if supplyAfterRemoval > (result.TotalDemand / a.config.ScaleDownBoundary) {
	result.SpareCapacity = spareCapacity
}

At t=0 nothing is in use and nothing is queued. Demand is 0, so spare capacity equals total supply. The N-1 check asks one question: after removing the biggest replica, does supply still cover demand? With zero demand, yes. That proves one removal is safe. It says nothing about four.

2. The engine turns spare capacity into a count. Before the fix, scaleDownN divided the spare capacity by one replica's supply (engine.go):

prc := vc.Supply / float64(vc.ReplicaCount)
// ...
n := int(math.Floor(spareCapacity / prc))
if n < 1 {
	n = 1
}
if n > vc.ReplicaCount {
	n = vc.ReplicaCount
}
return n

Four replicas, and spare capacity worth four replicas: floor(4) = 4. The only upper clamp is the replica count, so the answer is all of them. A unit test on main even expects this: spare 100, three replicas, want Delta: -3 (engine_test.go).

The math is consistent. It removes replicas only while the ones that stay cover demand divided by the boundary. At zero demand, zero replicas pass that test.

3. At zero replicas, the autoscaler goes blind. Demand is measured on running replicas. With none left, Analyze returns an empty result (saturation_analyzer.go):

if len(metrics.Replicas) == 0 {
	// ...
	return result
}

No demand signal means no scale-up. Each request reaches the router, finds no endpoint, and is counted as RejectedRouting, which the code defines as "no endpoint could serve the request" (gateway.go). The debug log also shows that flow control is off in this run, so requests have no queue to wait in.

Why the tests missed it

The floor used to come for free. Before inference-sim#1199 (merged 1 May 2026), the engine removed one replica per decision (Delta: -1). The analyzer stops sending a scale-down signal at one replica (if initReplicas <= 1), so one at a time stopped at one. #1199 made scale-down exact: remove as many replicas as the spare capacity covers, in one step. That was a sensible speed-up, and it removed a floor nobody had written down.

Then zero capacity hid it. Until inference-sim#2000 (merged 21 October 2026), the analyzer saw zero KV capacity on every replica (inference-sim#1916). With no measured supply, it never sent a scale-down signal. #2000 fixed that. In the issue's words: "With live capacity, the first tick now sees full supply and zero demand, so this drain is the default."

Then a window hid it again. With #2000, the capability test got a 5-minute scale-down stabilization window, next to a comment that without it the analyzer "would drain every replica at the first tick" (capability_matrix_test.go). The test's 60 requests arrive within six seconds, far inside the window. Outside the test, BLIS defaults the window to 0 (deployment.go).

A stabilization window changes when a decision lands, not how big it is. BLIS holds a scale-down until its signal has lasted the full window, then passes the whole decision (autoscaler.go). An idle cluster with a 5-minute window still drops to zero. It just does it five minutes later.

The fix

inference-sim#1997 caps the count at the model's replicas minus one. The new scaleDownN:

func scaleDownN(spareCapacity float64, vc VariantCapacity, modelReplicas int) int {
	if vc.ReplicaCount <= 0 {
		return 0
	}
	n := 1
	if prc := vc.Supply / float64(vc.ReplicaCount); prc > 0 {
		n = max(1, int(math.Floor(spareCapacity/prc)))
	}
	return min(n, vc.ReplicaCount, modelReplicas-1)
}

Both engines call it the same way, and emit a decision only when there is something to remove:

if n := scaleDownN(r.SpareCapacity, vc, totalReplicas(vcs)); n > 0 {
	decisions = append(decisions, ScaleDecision{ModelID: r.ModelID, Variant: vc.Variant, Delta: -n})
}

Three choices sit in those lines:

• The floor is per model, not per variant. totalReplicas counts replicas across every variant of the model. A model on A100s and H100s can still drain every H100 while the A100s serve. Only the model's last replica is protected.
• "Do nothing" means "say nothing". A single-replica model gets n = 0, and the engine emits no decision. BLIS treats a zero-delta decision as an engine bug (INV-19).
• The test stops hiding the bug. The capability test drops its 5-minute window, so it now tests the floor.

The PR adds an engine test for each case: spare capacity covers all three replicas and the engine removes two; a whole variant drains while another serves; a single-replica model gets no decision, in both engines. Per the PR's test plan, the capability test and the engine cases fail on main's engine.go.

The fix does not scale to zero on purpose. That stays a separate, opt-in feature, planned in inference-sim#909: drain a model's replicas after a set idle period, and only when no replica has requests in flight.

Status as of 31 October 2026: inference-sim#2015 and inference-sim#1997 are open. The PR has no reviews yet, and its CI waits for maintainer approval because I am a first-time contributor. As the PR notes, I wrote it with GitHub Copilot and reviewed the diff myself.

How real autoscalers set the floor

BLIS models llm-d, so the useful question is how production autoscalers handle the same moment. Each one I checked puts the floor in an explicit place, and treats zero as its own policy.

Kubernetes HPA. minReplicas "defaults to 1 pod" (API reference). The default scale-down policy can remove 100% of running replicas, which "means the scaling target can be scaled down to the minimum allowed replicas", after a 5-minute stabilization window (HPA docs). The window slows the drop. minReplicas stops it.

HPA scale to zero. It is beta since Kubernetes v1.37 and on by default. You set minReplicas: 0, and you must scale on an object or external metric. CPU and memory do not qualify, "because those can only be measured on running Pods" (HPA docs). That is the blind spot from step 3 above. The signal that wakes a workload has to live outside it.

llm-d WVA v0.9.0. BLIS adapted WVA's analyzer. WVA also runs an enforcer on each model after its optimizer (enforcer.go). With scale-to-zero off, if a model's replicas would total zero, it sets the cheapest variant to one. With scale-to-zero on, it goes to zero only when no requests arrived in a retention period, 10 minutes by default. A separate loop polls every 100 ms and scales a model from zero to one when the gateway's flow-control queue for it is not empty (scale-from-zero engine).

llm-d today. WVA is deprecated on main, and KEDA now drives llm-d autoscaling (README). The prefill and decode blueprint sets minReplicas: 1 for both roles, and scales down one pod per 300 seconds after a 300-second window (pd-disaggregation.yaml).

Why the floor is worth paying for. A model at zero replicas must load its weights again before it serves. In ServerlessLLM's own benchmark on H100 GPUs with NVMe SSDs, loading Qwen3-32B takes 20.6 seconds with SafeTensors and 3.2 seconds with ServerlessLLM's loader (README, vendor numbers). That covers loading the weights, nothing else. Requests that arrive in that gap need a queue to wait in. The BLIS run above had no queue and no replica coming back.

Check your own autoscaler

1. Feed it zero demand. Read what scale-down returns. If the answer is "every replica", you have this bug.
2. Put the floor where the count is chosen, or in an explicit step right after it. A stabilization window is not a floor.
3. Build scale-to-zero as its own policy: an idle timer, a demand signal that lives outside the replicas, and a queue for requests during the cold start.

Zero demand at the first tick is a measurement, not a forecast. Two hundred requests were already on their way.