Pause a llama turn at once, and let Stop take the model with it

A turn waits on three things that look nowhere at all: a permission
question, a tool call with a minute to run, and the completion itself,
which says nothing while the prompt is read -- tens of seconds on a long
conversation. Setting a flag left the turn exactly where it was until
whichever it was came back.

The wait is now what ends, not the work. Each of those runs on a thread of
its own and the interrupt answers the wait; the abandoned thread finishes
into a channel nobody is reading. 43ms to end a turn in every state,
measured against a real model -- including mid prompt-processing, which
used to be a twenty-second wait. That makes cancellation a token per turn
rather than a flag on the session: the abandoned thread wakes up some time
later, and a flag the next turn had reset would let it write into a
conversation it is no longer part of. The open thinking block moves to
Shared for the same reason -- the thread that knows one is open is no
longer the thread that ends the turn.

Stop, meanwhile, did nothing at all to a llama session: it signals the
session's recorded process and process::stop refuses a Shared one, which
is the whole point of that record -- so the session sat at idle. A Shared
record routes to the driver now, because what stopping means for a session
that borrows the machine's process is the driver's to say. It ends the
turn, says exited itself, and gives up its claim on the model; each live
session claims the model it is on, and the model is unloaded when the last
claim goes. A model another session is using stays where it is.
This commit is contained in:
iris-ai committed 2026-09-21 03:16:55 -04:00
1 parent 849c3b599f
commit 386c1c4def
6 files changed
+419 -137

No files matched your search

+24
View File
@@ -477,6 +477,30 @@ deliberate and easy to undo by accident:
only a turn whose model was changed under it. The conversation is read
*before* the message is announced, which is what makes "everything before
this message" true rather than a race against the pump.
- **An interrupt ends the wait, not the work** (2026-09-21, `awaiting`,
`Cancel`). Everything a llama turn waits on is on the far side of something
that cannot be told to stop -- a permission nobody has answered, a shell
command with a minute to run, a completion that says nothing until the
prompt has been read. Each now runs on a thread of its own and the
interrupt answers the *wait*; the abandoned thread finishes into a channel
nobody is reading. Measured at 43ms to end a turn in every state, against
up to a minute before. Two things fall out of it. Cancellation is a **token
per turn** rather than a flag on the session, because the abandoned thread
wakes later and a flag the next turn had reset would let it speak into a
conversation it is no longer part of. And the open thinking block is closed
by the interrupt, not by that thread, since the thread that knows is not
the one that ends the turn.
- **Stop takes the model with it when nobody else wants it** (2026-09-21).
Stop did nothing to a llama session at all: `stop_session` signals the
session's own recorded process and refuses a `Shared` one, which is the
whole point of that record -- so the session sat at `idle`. A `Shared`
record routes to `Driver::stop` now, because what stopping means for a
session that borrows the machine's process is the driver's to say. Each
live session claims the model it is on (`Router::claim`) and the model is
unloaded when the last claim goes, which is the honest reading of "give me
my GPU back": a model another session is on is nobody's to take. Rejected:
unloading unconditionally, which is the thing the shared router exists to
prevent.
- **A wait that can be measured says how far along it is** (2026-09-21,
`GET /sessions/{id}/progress`, `Driver::progress`). Both of this
driver's waits have a real number behind them and neither used to reach