Pause a llama turn at once, and let Stop take the model with it

A turn waits on three things that look nowhere at all: a permission
question, a tool call with a minute to run, and the completion itself,
which says nothing while the prompt is read -- tens of seconds on a long
conversation. Setting a flag left the turn exactly where it was until
whichever it was came back.

The wait is now what ends, not the work. Each of those runs on a thread of
its own and the interrupt answers the wait; the abandoned thread finishes
into a channel nobody is reading. 43ms to end a turn in every state,
measured against a real model -- including mid prompt-processing, which
used to be a twenty-second wait. That makes cancellation a token per turn
rather than a flag on the session: the abandoned thread wakes up some time
later, and a flag the next turn had reset would let it write into a
conversation it is no longer part of. The open thinking block moves to
Shared for the same reason -- the thread that knows one is open is no
longer the thread that ends the turn.

Stop, meanwhile, did nothing at all to a llama session: it signals the
session's recorded process and process::stop refuses a Shared one, which
is the whole point of that record -- so the session sat at idle. A Shared
record routes to the driver now, because what stopping means for a session
that borrows the machine's process is the driver's to say. It ends the
turn, says exited itself, and gives up its claim on the model; each live
session claims the model it is on, and the model is unloaded when the last
claim goes. A model another session is using stays where it is.
This commit is contained in:
iris-ai committed 2026-09-21 03:16:55 -04:00
1 parent 849c3b599f
commit 386c1c4def
6 files changed
+419 -137

No files matched your search

+30 -8
View File
@@ -511,14 +511,36 @@ written, and the fold uses that same predicate to decide a reply is settled.
reply, and the transcript keeps whatever arrived before it.
- **A cancel flag is only as prompt as the next place somebody looks.** A
llama turn waits on two things that look nowhere at all: a permission
question, and a tool call `llama-server` is running -- a shell command there
runs to its own timeout, up to a minute. Setting `cancel` left the turn
exactly where it was until that came back, so Pause did nothing on screen
for as long as the command took. `Shared::abandon_turn` sets the flag *and*
releases both waits, the tool call by answering its channel with
`tools::UNFINISHED` -- the call is left running over there, because nothing
in that protocol takes one back, and its answer is dropped.
llama turn waits on three things that look nowhere at all: a permission
question, a tool call `llama-server` is running (a shell command there runs
to its own timeout, up to a minute), and the completion itself, which says
nothing for as long as the prompt takes to read -- tens of seconds on a long
conversation. Setting `cancel` left the turn exactly where it was until
whichever it was came back, so Pause did nothing on screen for all of it.
**The wait is what ends, not the work**: `awaiting` runs each of those on a
thread of its own and `Shared::abandon_turn` answers the wait, so the turn
ends in milliseconds (measured 43ms in every state) and the abandoned thread
finishes into a channel nobody is reading. Nothing here can stop a shell
command or a model mid-reply, and pretending otherwise is what the old code
did.
**Which is why cancellation is a token per turn** (`Cancel`), not a flag on
the session: the abandoned thread wakes up some time later, and a flag the
next turn had reset would let it write into a conversation it is no longer
part of. Its own token stays set for ever, so it says nothing -- and the
open thinking block is closed by `Shared::abandon_turn` rather than by that
thread, since the one that knows is not the one that ends the turn.
- **Stop ends a llama session and takes the model with it, if nobody else
wants it** (2026-09-21). Until then Stop did *nothing at all* to one:
`stop_session` signals the session's recorded process, and `process::stop`
refuses a `Shared` record -- so the session sat at `idle` with no sign
anything had happened. A `Shared` record now routes to `Driver::stop`,
because what stopping means for a session that borrows somebody else's
process is the driver's to say. The llama driver ends the turn, says
`exited` itself (nothing else will -- there is no process of its own to
die), and asks `Router::release`: each live session claims the model it is
on, and the model comes out of memory only when the last claim goes. A model
another session is using stays.
- **A path is stored as it was typed, and `~` is expanded where it is used.**
`~/repos/x` and `/home/someone/repos/x` are a path and a snapshot of where it