Pause a llama turn at once, and let Stop take the model with it
A turn waits on three things that look nowhere at all: a permission question, a tool call with a minute to run, and the completion itself, which says nothing while the prompt is read -- tens of seconds on a long conversation. Setting a flag left the turn exactly where it was until whichever it was came back. The wait is now what ends, not the work. Each of those runs on a thread of its own and the interrupt answers the wait; the abandoned thread finishes into a channel nobody is reading. 43ms to end a turn in every state, measured against a real model -- including mid prompt-processing, which used to be a twenty-second wait. That makes cancellation a token per turn rather than a flag on the session: the abandoned thread wakes up some time later, and a flag the next turn had reset would let it write into a conversation it is no longer part of. The open thinking block moves to Shared for the same reason -- the thread that knows one is open is no longer the thread that ends the turn. Stop, meanwhile, did nothing at all to a llama session: it signals the session's recorded process and process::stop refuses a Shared one, which is the whole point of that record -- so the session sat at idle. A Shared record routes to the driver now, because what stopping means for a session that borrows the machine's process is the driver's to say. It ends the turn, says exited itself, and gives up its claim on the model; each live session claims the model it is on, and the model is unloaded when the last claim goes. A model another session is using stays where it is.
This commit is contained in:
1 parent
849c3b599f
commit
386c1c4def
6 files changed
+419
-137
No files matched your search
@@ -511,14 +511,36 @@ written, and the fold uses that same predicate to decide a reply is settled.
|
||||
reply, and the transcript keeps whatever arrived before it.
|
||||
|
||||
- **A cancel flag is only as prompt as the next place somebody looks.** A
|
||||
llama turn waits on two things that look nowhere at all: a permission
|
||||
question, and a tool call `llama-server` is running -- a shell command there
|
||||
runs to its own timeout, up to a minute. Setting `cancel` left the turn
|
||||
exactly where it was until that came back, so Pause did nothing on screen
|
||||
for as long as the command took. `Shared::abandon_turn` sets the flag *and*
|
||||
releases both waits, the tool call by answering its channel with
|
||||
`tools::UNFINISHED` -- the call is left running over there, because nothing
|
||||
in that protocol takes one back, and its answer is dropped.
|
||||
llama turn waits on three things that look nowhere at all: a permission
|
||||
question, a tool call `llama-server` is running (a shell command there runs
|
||||
to its own timeout, up to a minute), and the completion itself, which says
|
||||
nothing for as long as the prompt takes to read -- tens of seconds on a long
|
||||
conversation. Setting `cancel` left the turn exactly where it was until
|
||||
whichever it was came back, so Pause did nothing on screen for all of it.
|
||||
**The wait is what ends, not the work**: `awaiting` runs each of those on a
|
||||
thread of its own and `Shared::abandon_turn` answers the wait, so the turn
|
||||
ends in milliseconds (measured 43ms in every state) and the abandoned thread
|
||||
finishes into a channel nobody is reading. Nothing here can stop a shell
|
||||
command or a model mid-reply, and pretending otherwise is what the old code
|
||||
did.
|
||||
**Which is why cancellation is a token per turn** (`Cancel`), not a flag on
|
||||
the session: the abandoned thread wakes up some time later, and a flag the
|
||||
next turn had reset would let it write into a conversation it is no longer
|
||||
part of. Its own token stays set for ever, so it says nothing -- and the
|
||||
open thinking block is closed by `Shared::abandon_turn` rather than by that
|
||||
thread, since the one that knows is not the one that ends the turn.
|
||||
|
||||
- **Stop ends a llama session and takes the model with it, if nobody else
|
||||
wants it** (2026-09-21). Until then Stop did *nothing at all* to one:
|
||||
`stop_session` signals the session's recorded process, and `process::stop`
|
||||
refuses a `Shared` record -- so the session sat at `idle` with no sign
|
||||
anything had happened. A `Shared` record now routes to `Driver::stop`,
|
||||
because what stopping means for a session that borrows somebody else's
|
||||
process is the driver's to say. The llama driver ends the turn, says
|
||||
`exited` itself (nothing else will -- there is no process of its own to
|
||||
die), and asks `Router::release`: each live session claims the model it is
|
||||
on, and the model comes out of memory only when the last claim goes. A model
|
||||
another session is using stays.
|
||||
|
||||
- **A path is stored as it was typed, and `~` is expanded where it is used.**
|
||||
`~/repos/x` and `/home/someone/repos/x` are a path and a snapshot of where it
|
||||
|
||||
Reference in new issue
Block a user