Give a llama session a thinking level, asked of the model

A `thinking` param on the llama driver: "auto", "off", or a level, applied as a
chat-template argument on the next request -- `reasoning_effort`, or
`enable_thinking: false` for off -- so unlike the server flags it costs no
reload. It lands in the session settings dialog beside the other model
settings, which is what declaring it in `DriverKind::params` buys.

Which levels exist is the model's answer rather than a constant, because the
vocabularies disagree: the 27B here takes low, medium and xhigh and **raises**
on high and max, so a fixed list is a turn that fails on send. The driver asks
the loaded server (`thinking_options`) -- `chat_template_caps.
supports_reasoning_effort` for whether levels mean anything at all, which is
the gate that stops the control silently doing nothing on a template that
ignores the argument, then `/apply-template` per level, one cheap render each
at load time. Off is a separate argument and a separate question: honoured when
turning it off renders a different prompt, and both renders have to have
worked, since a template that refuses it also renders differently.

A level the loaded model cannot take is dropped from the request and said in
the transcript, naming what it does take. What is *not* said is anything about
a model nobody has asked yet: the answer is `Option<Vec<String>>`, where None
is "no server has been up" and an empty list is the model that genuinely takes
none.

Verified against the 27B on the GPU: "low" thought for 697ms and 79 characters,
"off" produced no thinking block at all, and "high" answered `this model does
not take "high" -- it takes off, low, medium, xhigh.` The picker wraps to two
rows in the settings dialog and shows the session's current value.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
iris-aiandClaude Opus 5 committed 2026-09-19 16:14:35 -04:00
1 parent 369b8f7e52
commit 74cda485e5
4 files changed
+244

No files matched your search

+16
View File
@@ -357,6 +357,22 @@ deliberate and easy to undo by accident:
ephemeral one, because no portable way to ask a machine for a free port
avoids racing the bind anyway; a collision is not silent, since the server
fails to bind and the readiness poll reports what its log said.
- **A llama session's thinking level is the model's, asked of the model**
(2026-09-19). Thinking effort is a chat-template argument rather than a
server flag, so it rides on the next request and changes nothing about the
loaded model -- which is why it is a `params` entry (`thinking`) and not the
`effort` a coding CLI reads at launch. The levels templates use disagree:
the 27B here takes `low`, `medium` and `xhigh` and **raises** on `high` and
`max`, so a fixed list would be a turn that fails on send. The driver asks
the loaded server instead (`thinking_options`): `chat_template_caps.
supports_reasoning_effort` says whether levels mean anything at all -- the
gate that stops the control silently doing nothing on a template that
ignores the argument -- and `/apply-template` says which of them render, one
cheap round trip each at load time. `off` is a separate question and a
separate argument (`enable_thinking: false`), taken as supported when
turning it off renders a different prompt. A level the loaded model cannot
take is dropped from the request and said out loud, naming what it does
take.
- **The model file lives on the machine that serves it** (2026-09-04). Each
machine has its own models directory (`SshConfig::models_dir`, default
`~/.local/share/ai-app/models` expanded *there*), and a spawn resolves the