Say how far a llama session's wait has got
Both of its waits are measured somewhere and neither reached the phone: a
model coming off disk, which the router publishes on its event stream and
nowhere else, and a prompt being read, which the generation stream will
report when asked. A session now answers GET /sessions/{id}/progress with
{of, fraction, stage?}, one thread per router keeping the load's fraction
per model, and the session screen asks twice a second while it is drawing
a wait that has one.
Asked for rather than emitted: a load reports five times a second, and an
event is a transcript line for ever. The sample says which status it
measures, so one that outlived its wait cannot be drawn under another
word. The phone puts the bar in the status row's free width and the
percentage where the context figure sits -- a row of its own would move
the transcript every time a turn started -- and names the stage where a
model loads more than one file, because the fraction starts again for
each. /loading and /reading in an echo session are the rig.
This commit is contained in:
1 parent
78f2fe3b79
commit
049780fda6
10 files changed
+549
-16
No files matched your search
@@ -75,8 +75,8 @@ use std::sync::{Arc, Mutex};
|
||||
use std::time::Duration;
|
||||
|
||||
use super::driver::{
|
||||
AttachmentRef, BackgroundTask, BackgroundTaskKind, Driver, Event, EventSink, QuestionOption,
|
||||
SessionStatus, Unqueued, patch_start,
|
||||
AttachmentRef, BackgroundTask, BackgroundTaskKind, Driver, Event, EventSink, Progress,
|
||||
QuestionOption, SessionStatus, Unqueued, patch_start,
|
||||
};
|
||||
use super::subagent::Subagents;
|
||||
|
||||
@@ -128,6 +128,12 @@ pub struct EchoDriver {
|
||||
/// Live background commands, for the same list a real provider reports.
|
||||
/// This is the deterministic UI/session-lifecycle rig for that state.
|
||||
background_tasks: Arc<Mutex<Vec<BackgroundTask>>>,
|
||||
/// What `/loading` and `/reading` are pretending is under way, which is
|
||||
/// the rig for the measured half of a wait -- `GET
|
||||
/// /sessions/{id}/progress` and the bar the status row draws from it.
|
||||
/// Real ones come off a model coming off disk and a prompt being read,
|
||||
/// neither of which an echo session has.
|
||||
progress: Arc<Mutex<Option<Progress>>>,
|
||||
/// This session's subagents -- see `SUBAGENTS.md`. `/subagent` is the
|
||||
/// test rig for the same registry the claude driver routes real Task
|
||||
/// calls into.
|
||||
@@ -381,6 +387,73 @@ impl EchoDriver {
|
||||
// imitated. The meter it should agree with is `/usage`'s fixture,
|
||||
// deliberately separate: the two disagreeing is a state worth being
|
||||
// able to produce, since it is what a stale reset time looks like.
|
||||
// A wait with a measurement under it: the session sits in `loading` or
|
||||
// `reading` for `seconds` while a fraction fills, which is what the
|
||||
// status row's bar is drawn from. Both real ones belong to llama.cpp
|
||||
// -- a model coming off disk, a prompt being read -- so this is the
|
||||
// only way to exercise the phone's half without a GPU.
|
||||
//
|
||||
// `stages` above one makes the fraction start again for each, the way
|
||||
// a model with a projector and a draft head loads: the case where a
|
||||
// bar that only counted up would be a lie.
|
||||
if let Some(rest) = text
|
||||
.strip_prefix("/loading")
|
||||
.or_else(|| text.strip_prefix("/reading"))
|
||||
{
|
||||
let of = if text.starts_with("/loading") {
|
||||
SessionStatus::Loading
|
||||
} else {
|
||||
SessionStatus::Reading
|
||||
};
|
||||
let mut words = rest.split_whitespace();
|
||||
let seconds = words
|
||||
.next()
|
||||
.and_then(|w| w.parse::<u64>().ok())
|
||||
.unwrap_or(10)
|
||||
.clamp(1, 600);
|
||||
let stages = words
|
||||
.next()
|
||||
.and_then(|w| w.parse::<usize>().ok())
|
||||
.unwrap_or(1)
|
||||
.clamp(1, 3);
|
||||
if announce {
|
||||
self.emit(Event::MessageTaken {
|
||||
id: None,
|
||||
text: text.clone(),
|
||||
attachments,
|
||||
});
|
||||
}
|
||||
self.emit(Event::Status { state: of });
|
||||
let sink = self.sink.clone();
|
||||
let progress = Arc::clone(&self.progress);
|
||||
let busy = Arc::clone(&self.busy);
|
||||
let queued = Arc::clone(&self.queued);
|
||||
busy.store(true, Ordering::SeqCst);
|
||||
tokio::spawn(async move {
|
||||
// Per stage, so three stages take three times as long -- the
|
||||
// same shape as three files coming off one disk.
|
||||
let steps = seconds * 5;
|
||||
for stage in 0..stages {
|
||||
for step in 0..=steps {
|
||||
*progress.lock().unwrap() = Some(Progress {
|
||||
of,
|
||||
fraction: step as f32 / steps as f32,
|
||||
stage: (stages > 1).then(|| {
|
||||
["text_model", "mmproj_model", "spec_model"][stage].to_string()
|
||||
}),
|
||||
});
|
||||
tokio::time::sleep(Duration::from_millis(200)).await;
|
||||
}
|
||||
}
|
||||
*progress.lock().unwrap() = None;
|
||||
let _ = sink.send(Event::AssistantText {
|
||||
delta: format!("done waiting ({stages} stage(s), {seconds}s each)"),
|
||||
});
|
||||
finish_turn(&sink, &queued, &busy);
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if let Some(rest) = text.strip_prefix("/limit") {
|
||||
if announce {
|
||||
self.emit(Event::MessageTaken {
|
||||
@@ -922,6 +995,7 @@ impl EchoDriver {
|
||||
pending_questions: Mutex::new(Vec::new()),
|
||||
context: Arc::new(AtomicU64::new(0)),
|
||||
background_tasks: Arc::new(Mutex::new(Vec::new())),
|
||||
progress: Arc::new(Mutex::new(None)),
|
||||
busy: Arc::new(AtomicBool::new(false)),
|
||||
queued: Arc::new(Mutex::new(Vec::new())),
|
||||
session_dir,
|
||||
@@ -1269,6 +1343,10 @@ impl Driver for EchoDriver {
|
||||
self.handle(text.to_string(), Vec::new(), false);
|
||||
}
|
||||
|
||||
fn progress(&self) -> Option<Progress> {
|
||||
self.progress.lock().unwrap().clone()
|
||||
}
|
||||
|
||||
fn answer_question(&self, id: &str, answers: &[String]) {
|
||||
let answer = answers.join(", ");
|
||||
let (answered, waiting) = {
|
||||
|
||||
Reference in new issue
Block a user