Pause a llama turn at once, and let Stop take the model with it
A turn waits on three things that look nowhere at all: a permission question, a tool call with a minute to run, and the completion itself, which says nothing while the prompt is read -- tens of seconds on a long conversation. Setting a flag left the turn exactly where it was until whichever it was came back. The wait is now what ends, not the work. Each of those runs on a thread of its own and the interrupt answers the wait; the abandoned thread finishes into a channel nobody is reading. 43ms to end a turn in every state, measured against a real model -- including mid prompt-processing, which used to be a twenty-second wait. That makes cancellation a token per turn rather than a flag on the session: the abandoned thread wakes up some time later, and a flag the next turn had reset would let it write into a conversation it is no longer part of. The open thinking block moves to Shared for the same reason -- the thread that knows one is open is no longer the thread that ends the turn. Stop, meanwhile, did nothing at all to a llama session: it signals the session's recorded process and process::stop refuses a Shared one, which is the whole point of that record -- so the session sat at idle. A Shared record routes to the driver now, because what stopping means for a session that borrows the machine's process is the driver's to say. It ends the turn, says exited itself, and gives up its claim on the model; each live session claims the model it is on, and the model is unloaded when the last claim goes. A model another session is using stays where it is.
This commit is contained in:
1 parent
849c3b599f
commit
386c1c4def
6 files changed
+419
-137
No files matched your search
@@ -22,6 +22,33 @@ one in place when it turns out to need a decision.
|
||||
|
||||
## Session settings
|
||||
|
||||
Asked for by Bryan on 2026-09-21, in one run while other work was in flight.
|
||||
The first two are one change; the rest can land separately.
|
||||
|
||||
- [ ] **A screen, not a modal.** The settings dialog has outgrown one: it
|
||||
scrolls inside itself and covers the session it is about.
|
||||
|
||||
- [ ] **Two tabs on that screen, the way the main screen has three.** One is
|
||||
the session's own settings; the other is *the same provider screen*
|
||||
reached from the machines tab (`ProviderScreen`), for this session's
|
||||
provider -- two ways in, one screen, no second copy of the truth.
|
||||
|
||||
- [ ] **Compact fields everywhere.** The label goes above the box rather than
|
||||
floating inside it, and the padding around the value comes down. The
|
||||
value's own text size does not change: what is costing a row its height
|
||||
is the framing, not the text.
|
||||
|
||||
- [ ] **A system prompt per session.** For llama.cpp it is a `system` message
|
||||
on each request, so it is one entry in `DriverKind::params` and no app
|
||||
change; whether the CLI drivers get one (`--append-system-prompt`) is a
|
||||
separate question.
|
||||
|
||||
- [ ] **Stop unloads the model where nothing else is using it.** Today
|
||||
`Driver::stop` deliberately leaves it in memory, because the server is
|
||||
the machine's and another session may be on the same model. The answer
|
||||
is a claim per live session on the router, and an unload when the last
|
||||
one goes -- not an unconditional unload.
|
||||
|
||||
- [ ] Autocompact belongs in session settings; empty disables it, which is the
|
||||
default. Iris chose "hand it to the driver" — only where a driver has
|
||||
auto-compaction of its own. **That option was offered on a false premise
|
||||
|
||||
Reference in new issue
Block a user