Say how far a llama session's wait has got
Both of its waits are measured somewhere and neither reached the phone: a
model coming off disk, which the router publishes on its event stream and
nowhere else, and a prompt being read, which the generation stream will
report when asked. A session now answers GET /sessions/{id}/progress with
{of, fraction, stage?}, one thread per router keeping the load's fraction
per model, and the session screen asks twice a second while it is drawing
a wait that has one.
Asked for rather than emitted: a load reports five times a second, and an
event is a transcript line for ever. The sample says which status it
measures, so one that outlived its wait cannot be drawn under another
word. The phone puts the bar in the status row's free width and the
percentage where the context figure sits -- a row of its own would move
the transcript every time a turn started -- and names the stage where a
model loads more than one file, because the fraction starts again for
each. /loading and /reading in an echo session are the rig.
This commit is contained in:
1 parent
78f2fe3b79
commit
049780fda6
10 files changed
+549
-16
No files matched your search
@@ -477,6 +477,31 @@ deliberate and easy to undo by accident:
|
||||
only a turn whose model was changed under it. The conversation is read
|
||||
*before* the message is announced, which is what makes "everything before
|
||||
this message" true rather than a race against the pump.
|
||||
- **A wait that can be measured says how far along it is** (2026-09-21,
|
||||
`GET /sessions/{id}/progress`, `Driver::progress`). Both of this
|
||||
driver's waits have a real number behind them and neither used to reach
|
||||
the phone: the router publishes a model's load progress on its
|
||||
`/models/sse` event stream -- and *only* there, since `GET /models`
|
||||
reports that a model is loading and not how far -- while
|
||||
`"return_progress": true` puts `prompt_progress` chunks in the generation
|
||||
stream. One thread per router follows the event stream and keeps the
|
||||
fraction per model (`router::watch_loads`), because a load belongs to the
|
||||
machine: two sessions opening one model watch the same one arrive.
|
||||
**It is state to be asked for, never an event.** A load reports five
|
||||
times a second, and the transcript is the one thing every event a session
|
||||
produces is written into -- a record of where a load got to is of no
|
||||
interest to anybody once it has got there, and the phone is the only
|
||||
reader. So the session screen asks twice a second while it is showing a
|
||||
wait that has one, and not at all otherwise. The sample carries **which
|
||||
status it measures**, so one taken during a wait that has since ended
|
||||
cannot be drawn under a different word, and the phone shows it in the
|
||||
status row's free width with the percentage where the context figure sits
|
||||
-- a bar in a row of its own would move the transcript and the composer
|
||||
every time a turn started. The stage is named where a model loads more
|
||||
than one file (weights, projector, draft head), because the fraction
|
||||
starts again for each and a bar that only counted up would be lying.
|
||||
`/loading [seconds] [stages]` and `/reading [seconds]` in an echo session
|
||||
are the rig for the phone's half.
|
||||
- **Which tools a session offers is a filter here, not a flag there**
|
||||
(2026-09-19). The router is always started with `--tools all` and hosts one
|
||||
set of tools for the machine — one per session is not a thing a shared
|
||||
|
||||
Reference in new issue
Block a user