Pause a llama turn at once, and let Stop take the model with it
A turn waits on three things that look nowhere at all: a permission question, a tool call with a minute to run, and the completion itself, which says nothing while the prompt is read -- tens of seconds on a long conversation. Setting a flag left the turn exactly where it was until whichever it was came back. The wait is now what ends, not the work. Each of those runs on a thread of its own and the interrupt answers the wait; the abandoned thread finishes into a channel nobody is reading. 43ms to end a turn in every state, measured against a real model -- including mid prompt-processing, which used to be a twenty-second wait. That makes cancellation a token per turn rather than a flag on the session: the abandoned thread wakes up some time later, and a flag the next turn had reset would let it write into a conversation it is no longer part of. The open thinking block moves to Shared for the same reason -- the thread that knows one is open is no longer the thread that ends the turn. Stop, meanwhile, did nothing at all to a llama session: it signals the session's recorded process and process::stop refuses a Shared one, which is the whole point of that record -- so the session sat at idle. A Shared record routes to the driver now, because what stopping means for a session that borrows the machine's process is the driver's to say. It ends the turn, says exited itself, and gives up its claim on the model; each live session claims the model it is on, and the model is unloaded when the last claim goes. A model another session is using stays where it is.
This commit is contained in:
1 parent
849c3b599f
commit
386c1c4def
6 files changed
+419
-137
No files matched your search
@@ -477,6 +477,30 @@ deliberate and easy to undo by accident:
|
||||
only a turn whose model was changed under it. The conversation is read
|
||||
*before* the message is announced, which is what makes "everything before
|
||||
this message" true rather than a race against the pump.
|
||||
- **An interrupt ends the wait, not the work** (2026-09-21, `awaiting`,
|
||||
`Cancel`). Everything a llama turn waits on is on the far side of something
|
||||
that cannot be told to stop -- a permission nobody has answered, a shell
|
||||
command with a minute to run, a completion that says nothing until the
|
||||
prompt has been read. Each now runs on a thread of its own and the
|
||||
interrupt answers the *wait*; the abandoned thread finishes into a channel
|
||||
nobody is reading. Measured at 43ms to end a turn in every state, against
|
||||
up to a minute before. Two things fall out of it. Cancellation is a **token
|
||||
per turn** rather than a flag on the session, because the abandoned thread
|
||||
wakes later and a flag the next turn had reset would let it speak into a
|
||||
conversation it is no longer part of. And the open thinking block is closed
|
||||
by the interrupt, not by that thread, since the thread that knows is not
|
||||
the one that ends the turn.
|
||||
- **Stop takes the model with it when nobody else wants it** (2026-09-21).
|
||||
Stop did nothing to a llama session at all: `stop_session` signals the
|
||||
session's own recorded process and refuses a `Shared` one, which is the
|
||||
whole point of that record -- so the session sat at `idle`. A `Shared`
|
||||
record routes to `Driver::stop` now, because what stopping means for a
|
||||
session that borrows the machine's process is the driver's to say. Each
|
||||
live session claims the model it is on (`Router::claim`) and the model is
|
||||
unloaded when the last claim goes, which is the honest reading of "give me
|
||||
my GPU back": a model another session is on is nobody's to take. Rejected:
|
||||
unloading unconditionally, which is the thing the shared router exists to
|
||||
prevent.
|
||||
- **A wait that can be measured says how far along it is** (2026-09-21,
|
||||
`GET /sessions/{id}/progress`, `Driver::progress`). Both of this
|
||||
driver's waits have a real number behind them and neither used to reach
|
||||
|
||||
Reference in new issue
Block a user