Give a llama session a thinking level, asked of the model
A `thinking` param on the llama driver: "auto", "off", or a level, applied as a chat-template argument on the next request -- `reasoning_effort`, or `enable_thinking: false` for off -- so unlike the server flags it costs no reload. It lands in the session settings dialog beside the other model settings, which is what declaring it in `DriverKind::params` buys. Which levels exist is the model's answer rather than a constant, because the vocabularies disagree: the 27B here takes low, medium and xhigh and **raises** on high and max, so a fixed list is a turn that fails on send. The driver asks the loaded server (`thinking_options`) -- `chat_template_caps. supports_reasoning_effort` for whether levels mean anything at all, which is the gate that stops the control silently doing nothing on a template that ignores the argument, then `/apply-template` per level, one cheap render each at load time. Off is a separate argument and a separate question: honoured when turning it off renders a different prompt, and both renders have to have worked, since a template that refuses it also renders differently. A level the loaded model cannot take is dropped from the request and said in the transcript, naming what it does take. What is *not* said is anything about a model nobody has asked yet: the answer is `Option<Vec<String>>`, where None is "no server has been up" and an empty list is the model that genuinely takes none. Verified against the 27B on the GPU: "low" thought for 697ms and 79 characters, "off" produced no thinking block at all, and "high" answered `this model does not take "high" -- it takes off, low, medium, xhigh.` The picker wraps to two rows in the settings dialog and shows the session's current value. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
369b8f7e52
commit
74cda485e5
4 files changed
+244
No files matched your search
@@ -80,6 +80,13 @@ Module-by-module intent is in PLAN.md's "Backend layout".
|
||||
14k), and as `running` it looked exactly like thinking. The phone draws
|
||||
both with the working spinner and its own words, "loading model" and
|
||||
"reading prompt".
|
||||
**Thinking effort is a param, and which levels exist is the model's answer**
|
||||
(2026-09-19): the `thinking` param rides on the request as a chat-template
|
||||
argument (`reasoning_effort`, or `enable_thinking: false` for `off`), so it
|
||||
needs no restart -- and the driver asks the loaded server which levels its
|
||||
template actually takes rather than trusting the offered list, because the
|
||||
27B raises on `high` and answers to `xhigh`. A level it cannot take is
|
||||
dropped and said in the transcript, naming the ones it can.
|
||||
**Every one of those is a default rather than a constant** (2026-09-19):
|
||||
`DriverKind::params` declares what a provider takes — key, label, shape,
|
||||
and whether a change waits for a restart — and the phone renders whatever
|
||||
|
||||
Reference in new issue
Block a user