Queue a llama message sent while its model loads

A message sent into a loading session was recorded as *read* the moment it
arrived: the phone drew it as sent, nothing read it for the next minute, and
the turn then folded the conversation out of a transcript that by then held
that same message and appended it again -- so the model was sent it twice.
It queues now, exactly as a message sent into a running turn does: drawn as
waiting, takeable back, and opening the first turn when the model arrives.
The conversation is read before the message is announced, which is what makes
"everything before this message" true rather than a race against the pump.
`await_ready` is left for the one case that still needs it, a turn whose model
was changed under it, and the `idle` that used to close a load is now decided
beside that first turn rather than racing it.

Two silent endings found while reproducing it, both of which look on the phone
like a message that was sent and never answered: an `{"error": ...}` chunk
arriving mid-stream on an otherwise successful response (the GPU out of memory
mid-decode), and a stream that stops without its `[DONE]` (the model unloaded
under the session). Neither is an ordinary end; the turn fails for both, and
keeps whatever arrived before it.

Ran against a real llama session on this VM's Qwen3-0.6B: a message sent
during the load now queues and is answered when the model lands, and
unloading the model mid-reply now says so instead of going quietly idle.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
iris-aiandClaude Opus 5 committed 2026-09-19 21:20:58 -04:00
1 parent c3c6ab0ecf
commit ef788b0405
3 files changed
+129 -26

No files matched your search

+12
View File
@@ -454,6 +454,18 @@ deliberate and easy to undo by accident:
not ready" is a fact only a driver can have. The third state matters as
much as the first two: a model that will never load has to answer a waiting
message with what went wrong rather than holding it for ever.
**Where it waits is the queue** (corrected 2026-09-20). It first waited
inside the turn, which recorded the message as *read* the moment it arrived
-- so the phone drew it as sent and answered nothing for the next minute,
and the turn then folded the conversation out of a transcript that by then
held that same message, sending it to the model **twice**. A message
arriving during a load now queues exactly as one arriving during a turn
does: drawn as waiting, takeable back, and opening the first turn when the
model arrives (`LlamaDriver::open_queued`, which is also what decides
whether the end of a load is `idle`). What reaches `await_ready` is now
only a turn whose model was changed under it. The conversation is read
*before* the message is announced, which is what makes "everything before
this message" true rather than a race against the pump.
- **Which tools a session offers is a filter here, not a flag there**
(2026-09-19). The router is always started with `--tools all` and hosts one
set of tools for the machine — one per session is not a thing a shared