Don't draw a wait's bar at nothing
A fraction of zero is the absence of a sample, not a measurement of the work: a bar at nothing for two minutes says the load has not started, which of one 8 GB into a 12 GB file is false. Measured raw off the router's event stream today: llama-server reports a model's load as 0 and then 1 with nothing in between, on both the 0.6B and the 27B, and reports prompt processing once a batch. So the bar now draws for a long prompt -- the wait worth watching -- and a load that says nothing draws none, which is what not knowing should look like.
This commit is contained in:
1 parent
66b3403a71
commit
849c3b599f
4 files changed
+50
-15
No files matched your search
@@ -132,7 +132,12 @@ Module-by-module intent is in PLAN.md's "Backend layout".
|
|||||||
`"return_progress": true` adds to the generation stream. The sample says
|
`"return_progress": true` adds to the generation stream. The sample says
|
||||||
which status it measures so it cannot be drawn under another one, and
|
which status it measures so it cannot be drawn under another one, and
|
||||||
`/loading [seconds] [stages]` / `/reading [seconds]` in an echo session are
|
`/loading [seconds] [stages]` / `/reading [seconds]` in an echo session are
|
||||||
the rig for the phone's half.
|
the rig for the phone's half. **Zero is never reported**, being the absence
|
||||||
|
of a sample rather than a measurement -- which matters because
|
||||||
|
`llama-server` reports a model's load as 0 and then 1 with nothing between
|
||||||
|
(measured 2026-09-21 on both the 0.6B and the 27B), and reports prompt
|
||||||
|
processing once a batch; so the bar usually draws for a long prompt and not
|
||||||
|
for a load.
|
||||||
**A turn's wait has two halves and says which** (2026-09-19):
|
**A turn's wait has two halves and says which** (2026-09-19):
|
||||||
`SessionStatus::Loading` is the model coming off disk and
|
`SessionStatus::Loading` is the model coming off disk and
|
||||||
`SessionStatus::Reading` is `llama-server` processing the prompt -- emitted
|
`SessionStatus::Reading` is `llama-server` processing the prompt -- emitted
|
||||||
|
|||||||
@@ -502,6 +502,15 @@ deliberate and easy to undo by accident:
|
|||||||
starts again for each and a bar that only counted up would be lying.
|
starts again for each and a bar that only counted up would be lying.
|
||||||
`/loading [seconds] [stages]` and `/reading [seconds]` in an echo session
|
`/loading [seconds] [stages]` and `/reading [seconds]` in an echo session
|
||||||
are the rig for the phone's half.
|
are the rig for the phone's half.
|
||||||
|
**A fraction of zero is not reported at all**, because it is the absence of
|
||||||
|
a sample rather than a measurement: a bar sitting at nothing for two
|
||||||
|
minutes says "this has not started", which of a load 8 GB into a 12 GB file
|
||||||
|
is false. That is not theoretical -- measured 2026-09-21, raw off
|
||||||
|
`/models/sse`, `llama-server` reports a load as **0 and then 1** with
|
||||||
|
nothing in between (both the 0.6B and the 27B), and prompt processing
|
||||||
|
reports **once a batch**, so a prompt under `n_batch` is one step as well.
|
||||||
|
So what usually draws a moving bar is a long prompt, which is also the wait
|
||||||
|
worth watching -- tens of seconds at the context sizes measured above.
|
||||||
- **Which tools a session offers is a filter here, not a flag there**
|
- **Which tools a session offers is a filter here, not a flag there**
|
||||||
(2026-09-19). The router is always started with `--tools all` and hosts one
|
(2026-09-19). The router is always started with `--tools all` and hosts one
|
||||||
set of tools for the machine — one per session is not a thing a shared
|
set of tools for the machine — one per session is not a thing a shared
|
||||||
|
|||||||
@@ -1501,6 +1501,15 @@ impl Driver for LlamaDriver {
|
|||||||
/// Which of the two waits this session is in decides which measurement
|
/// Which of the two waits this session is in decides which measurement
|
||||||
/// answers, so a fraction from one can never be drawn under the other --
|
/// answers, so a fraction from one can never be drawn under the other --
|
||||||
/// and a session that is in neither says nothing rather than zero.
|
/// and a session that is in neither says nothing rather than zero.
|
||||||
|
///
|
||||||
|
/// **Nothing is reported at a fraction of zero**, because that is the
|
||||||
|
/// absence of a sample rather than a measurement of the work: a bar
|
||||||
|
/// sitting at nothing for two minutes says "this has not started", which
|
||||||
|
/// of a load 8 GB into a 12 GB file is simply false. It matters here
|
||||||
|
/// rather than in theory -- `llama-server` reports a model's load as 0
|
||||||
|
/// and then 1 with nothing in between (measured 2026-09-21 against both
|
||||||
|
/// the 0.6B and the 27B, raw off `/models/sse`), so what usually draws a
|
||||||
|
/// bar is prompt processing, which reports a batch at a time.
|
||||||
fn progress(&self) -> Option<Progress> {
|
fn progress(&self) -> Option<Progress> {
|
||||||
let coming = match &*self
|
let coming = match &*self
|
||||||
.shared
|
.shared
|
||||||
@@ -1513,20 +1522,19 @@ impl Driver for LlamaDriver {
|
|||||||
};
|
};
|
||||||
if let Some(model) = coming {
|
if let Some(model) = coming {
|
||||||
let coming = self.respawn.router.loading(&model)?;
|
let coming = self.respawn.router.loading(&model)?;
|
||||||
return Some(Progress {
|
return measured(SessionStatus::Loading, coming.fraction, coming.stage);
|
||||||
of: SessionStatus::Loading,
|
|
||||||
fraction: coming.fraction,
|
|
||||||
stage: coming.stage,
|
|
||||||
});
|
|
||||||
}
|
}
|
||||||
let (processed, total) = (*self.shared.reading.lock().unwrap())?;
|
let (processed, total) = (*self.shared.reading.lock().unwrap())?;
|
||||||
// A prompt of nothing is not a wait, and it is what a division would
|
// A prompt of nothing is not a wait, and it is what a division would
|
||||||
// fail on.
|
// fail on.
|
||||||
(total > 0).then(|| Progress {
|
if total == 0 {
|
||||||
of: SessionStatus::Reading,
|
return None;
|
||||||
fraction: processed as f32 / total as f32,
|
}
|
||||||
stage: None,
|
measured(
|
||||||
})
|
SessionStatus::Reading,
|
||||||
|
processed as f32 / total as f32,
|
||||||
|
None,
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
fn interrupt(&self) {
|
fn interrupt(&self) {
|
||||||
@@ -1767,6 +1775,16 @@ fn conversation(path: &Path, images: Option<&Path>) -> Vec<Message> {
|
|||||||
fold.messages
|
fold.messages
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A sample, or `None` where it says nothing -- see [`Driver::progress`] on
|
||||||
|
/// why zero is not a measurement.
|
||||||
|
fn measured(of: SessionStatus, fraction: f32, stage: Option<String>) -> Option<Progress> {
|
||||||
|
(fraction > 0.0).then_some(Progress {
|
||||||
|
of,
|
||||||
|
fraction,
|
||||||
|
stage,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
/// One message somebody sent, as the model is to read it.
|
/// One message somebody sent, as the model is to read it.
|
||||||
///
|
///
|
||||||
/// `images` is [`conversation`]'s: the session directory for a model with
|
/// `images` is [`conversation`]'s: the session directory for a model with
|
||||||
|
|||||||
@@ -344,15 +344,18 @@ pub struct TranscriptFile {
|
|||||||
pub path: String,
|
pub path: String,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Where a session directory keeps its transcript.
|
||||||
|
fn transcript_in(dir: &Path) -> PathBuf {
|
||||||
|
dir.join("transcript.jsonl")
|
||||||
|
}
|
||||||
|
|
||||||
/// Where `id`'s transcript is, said the way the explorer takes it.
|
/// Where `id`'s transcript is, said the way the explorer takes it.
|
||||||
fn transcript_file(config: &Config, data_dir: &Path, id: &str) -> Option<TranscriptFile> {
|
fn transcript_file(config: &Config, data_dir: &Path, id: &str) -> Option<TranscriptFile> {
|
||||||
let machine = config.machine(crate::config::LOCAL_MACHINE_ID)?;
|
let machine = config.machine(crate::config::LOCAL_MACHINE_ID)?;
|
||||||
Some(TranscriptFile {
|
Some(TranscriptFile {
|
||||||
machine: machine.id.clone(),
|
machine: machine.id.clone(),
|
||||||
machine_name: machine.name.clone(),
|
machine_name: machine.name.clone(),
|
||||||
path: data_dir
|
path: transcript_in(&data_dir.join(id))
|
||||||
.join(id)
|
|
||||||
.join("transcript.jsonl")
|
|
||||||
.to_string_lossy()
|
.to_string_lossy()
|
||||||
.into_owned(),
|
.into_owned(),
|
||||||
})
|
})
|
||||||
@@ -2629,7 +2632,7 @@ fn launch(
|
|||||||
) -> Result<Arc<LiveSession>> {
|
) -> Result<Arc<LiveSession>> {
|
||||||
let dir = env.data_dir.join(&meta.id);
|
let dir = env.data_dir.join(&meta.id);
|
||||||
wg_app_link::private::create_dir(&dir)?;
|
wg_app_link::private::create_dir(&dir)?;
|
||||||
let transcript_path = dir.join("transcript.jsonl");
|
let transcript_path = transcript_in(&dir);
|
||||||
let mut transcript = Transcript::open(&transcript_path)?;
|
let mut transcript = Transcript::open(&transcript_path)?;
|
||||||
let last_status = transcript.last_status().unwrap_or(SessionStatus::Idle);
|
let last_status = transcript.last_status().unwrap_or(SessionStatus::Idle);
|
||||||
// Before the driver starts, so the token is there when it looks and the
|
// Before the driver starts, so the token is there when it looks and the
|
||||||
|
|||||||
Reference in new issue
Block a user