Pause a llama turn at once, and let Stop take the model with it

A turn waits on three things that look nowhere at all: a permission
question, a tool call with a minute to run, and the completion itself,
which says nothing while the prompt is read -- tens of seconds on a long
conversation. Setting a flag left the turn exactly where it was until
whichever it was came back.

The wait is now what ends, not the work. Each of those runs on a thread of
its own and the interrupt answers the wait; the abandoned thread finishes
into a channel nobody is reading. 43ms to end a turn in every state,
measured against a real model -- including mid prompt-processing, which
used to be a twenty-second wait. That makes cancellation a token per turn
rather than a flag on the session: the abandoned thread wakes up some time
later, and a flag the next turn had reset would let it write into a
conversation it is no longer part of. The open thinking block moves to
Shared for the same reason -- the thread that knows one is open is no
longer the thread that ends the turn.

Stop, meanwhile, did nothing at all to a llama session: it signals the
session's recorded process and process::stop refuses a Shared one, which
is the whole point of that record -- so the session sat at idle. A Shared
record routes to the driver now, because what stopping means for a session
that borrows the machine's process is the driver's to say. It ends the
turn, says exited itself, and gives up its claim on the model; each live
session claims the model it is on, and the model is unloaded when the last
claim goes. A model another session is using stays where it is.
This commit is contained in:
iris-ai committed 2026-09-21 03:16:55 -04:00
1 parent 849c3b599f
commit 386c1c4def
6 files changed
+419 -137

No files matched your search

+27
View File
@@ -22,6 +22,33 @@ one in place when it turns out to need a decision.
## Session settings
Asked for by Bryan on 2026-09-21, in one run while other work was in flight.
The first two are one change; the rest can land separately.
- [ ] **A screen, not a modal.** The settings dialog has outgrown one: it
scrolls inside itself and covers the session it is about.
- [ ] **Two tabs on that screen, the way the main screen has three.** One is
the session's own settings; the other is *the same provider screen*
reached from the machines tab (`ProviderScreen`), for this session's
provider -- two ways in, one screen, no second copy of the truth.
- [ ] **Compact fields everywhere.** The label goes above the box rather than
floating inside it, and the padding around the value comes down. The
value's own text size does not change: what is costing a row its height
is the framing, not the text.
- [ ] **A system prompt per session.** For llama.cpp it is a `system` message
on each request, so it is one entry in `DriverKind::params` and no app
change; whether the CLI drivers get one (`--append-system-prompt`) is a
separate question.
- [ ] **Stop unloads the model where nothing else is using it.** Today
`Driver::stop` deliberately leaves it in memory, because the server is
the machine's and another session may be on the same model. The answer
is a claim per live session on the router, and an unload when the last
one goes -- not an unconditional unload.
- [ ] Autocompact belongs in session settings; empty disables it, which is the
default. Iris chose "hand it to the driver" — only where a driver has
auto-compaction of its own. **That option was offered on a false premise