Say which half of the wait a llama turn is in
A turn has two waits in front of the first token and they were one word. `SessionStatus::Loading` was already the model coming off disk; this adds `SessionStatus::Reading` for llama-server processing the prompt -- emitted when the request goes out, cleared by the first thing the model says of any kind, so it covers every generate in a tool loop rather than only the first. Prefill is the expensive half on this machine: measured 9.5s for 6,068 tokens and 22s for 14,068 on the 27B with the GPU to itself. Reported as `running` that was indistinguishable from a model thinking, which is the thing the reader is waiting for. The phone draws both with the working spinner and its own words -- "loading model" and "reading prompt" -- and the session screen's status row now spins for all three busy states instead of only `running`, which is also how `loading` stops being a bare word with nothing moving. Measured while checking the tok/s figure, and recorded in the rigs skill: the 27B holds 55.5 to 50.3 tok/s between 1.5k and 14k of context, so decode decays gently, while the 0.6B on the CPU falls 30.1 to 11.5 over 6k. A shared GPU is a different failure -- the model does not load at all. Verified on the emulator against a real llama session: "loading model" while the server started, then "reading prompt" with the spinner through prompt processing, then the thinking card. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
bb5ac1a242
commit
b660905098
8 files changed
+114
-6
No files matched your search
@@ -258,6 +258,31 @@ where it was instead of half-deleted.
|
||||
worth another 7% in a single sample and is deliberately *not* passed — one
|
||||
sample on a virtualised GPU is not a number to hardcode.
|
||||
|
||||
- **Prompt processing is the expensive part of a llama turn here, and decode
|
||||
speed falls only slowly with context.** Taken 2026-09-19 on a free GPU, the
|
||||
27B with `--spec-type draft-mtp -np 1`, generating 160 tokens each time:
|
||||
|
||||
| context | decode | prefill of that prompt |
|
||||
| --- | --- | --- |
|
||||
| 88 | 43.4 tok/s (cold) | 21s |
|
||||
| 1,569 | 55.5 tok/s | (model still warming) |
|
||||
| 6,068 | 53.2 tok/s | 9.5s |
|
||||
| 14,068 | 50.3 tok/s | 22s |
|
||||
|
||||
So a turn on a long conversation spends tens of seconds before the first
|
||||
token, and that is what `SessionStatus::Reading` exists to say. The same
|
||||
sweep on the 0.6B **on the CPU** falls much harder -- 30.1 tok/s at 44
|
||||
tokens of context to 11.5 at 6,024 -- which is the shape somebody means by
|
||||
"it gets slower as the conversation goes on". The figure the app draws is
|
||||
`timings.predicted_per_second`, decode only, so prefill is never mixed into
|
||||
it.
|
||||
|
||||
- **A busy GPU is a model that will not load at all**, not a slow one:
|
||||
`radv/amdgpu: Failed to allocate a buffer` and `failed to load model` while
|
||||
something else holds VRAM. A 0.6B that had been decoding at 149 tok/s ran at
|
||||
16.7 in that window before its server died, so a tok/s figure taken while
|
||||
the card is shared says nothing about the model.
|
||||
|
||||
- **Asking for the head when the file has none is fatal**, not ignored:
|
||||
`context type MTP requested but model doesn't contain MTP layers` and the
|
||||
server exits. Without the flag the same file logs `unused tensor
|
||||
|
||||
Reference in new issue
Block a user