Queue a llama message sent while its model loads
A message sent into a loading session was recorded as *read* the moment it
arrived: the phone drew it as sent, nothing read it for the next minute, and
the turn then folded the conversation out of a transcript that by then held
that same message and appended it again -- so the model was sent it twice.
It queues now, exactly as a message sent into a running turn does: drawn as
waiting, takeable back, and opening the first turn when the model arrives.
The conversation is read before the message is announced, which is what makes
"everything before this message" true rather than a race against the pump.
`await_ready` is left for the one case that still needs it, a turn whose model
was changed under it, and the `idle` that used to close a load is now decided
beside that first turn rather than racing it.
Two silent endings found while reproducing it, both of which look on the phone
like a message that was sent and never answered: an `{"error": ...}` chunk
arriving mid-stream on an otherwise successful response (the GPU out of memory
mid-decode), and a stream that stops without its `[DONE]` (the model unloaded
under the session). Neither is an ordinary end; the turn fails for both, and
keeps whatever arrived before it.
Ran against a real llama session on this VM's Qwen3-0.6B: a message sent
during the load now queues and is answered when the model lands, and
unloading the model mid-reply now says so instead of going quietly idle.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c3c6ab0ecf
commit
ef788b0405
3 files changed
+129
-26
No files matched your search
@@ -420,14 +420,27 @@ written, and the fold uses that same predicate to decide a reply is settled.
|
||||
`loading` for ever and nothing appeared in the log. `Routers` holds a
|
||||
`tokio::runtime::Handle` and enters it around the spawn.
|
||||
|
||||
- **A llama session reports `loading`, and a message sent into it waits.**
|
||||
- **A llama session reports `loading`, and a message sent into it queues.**
|
||||
Before 2026-09-19 the session showed `running` from the moment the process
|
||||
started, so a minute of reading a model off disk was indistinguishable from
|
||||
a minute of thinking -- and anything sent in that window came back as an
|
||||
error, because `llama-server` refuses everything until the model is in
|
||||
memory. `SessionStatus::Loading` is the state and `Shared::await_ready` is
|
||||
the waiting. A driver that reports `Loading` owes the holding as well as the
|
||||
word.
|
||||
memory. `SessionStatus::Loading` is the state. A driver that reports
|
||||
`Loading` owes the holding as well as the word, and **the queue is where it
|
||||
holds**: held inside the turn instead (until 2026-09-20) the message was
|
||||
recorded as read on arrival, so the phone drew it as sent while nothing was
|
||||
reading it, and the turn then folded it out of the transcript *and*
|
||||
appended it, sending it to the model twice. `Shared::await_ready` is now
|
||||
only for a turn whose model was changed under it.
|
||||
|
||||
- **A llama turn that says nothing said something that was thrown away.** Two
|
||||
silent endings were found on 2026-09-20 and both looked, on the phone, like
|
||||
a message that was sent and never answered: an `{"error": ...}` chunk
|
||||
arriving mid-stream on an otherwise successful response (a GPU that ran out
|
||||
of memory mid-decode), and a stream that simply stops without its `[DONE]`
|
||||
(the model unloaded under the session). Neither is an ordinary end, and
|
||||
`generate` now fails the turn for both -- a reply that stops early is not a
|
||||
reply, and the transcript keeps whatever arrived before it.
|
||||
|
||||
- **A transcript outlives the enum.** Removing `Event::TaskNote` hours after
|
||||
adding it made every transcript that had recorded one unreadable, so
|
||||
|
||||
Reference in new issue
Block a user