Let a session drop its context without ending
Adds `Event::Cleared`, `SessionCommand::Clear`, and `Driver::clear`, so
`POST /sessions/{id}/command {"text": "/clear"}` does for a session what
the CLI's own `/clear` does for a terminal.
The marker is a divider, not a truncation: everything above it stays in
the transcript, because that is the only copy of the conversation the
phone has and a person scrolling back is a different question from what
the model is given. It also makes clearing mean one thing across
drivers -- `claude` sends `/clear` and the CLI answers with a fresh
`init` whose new session_id the reader already persists as the resume
token, so the next launch resumes the cleared conversation with nothing
to keep in step; `llama` needs no state at all, since `conversation()`
already folds the transcript and now folds from the last marker; `echo`
emits the marker alone, so the phone's divider and scroll behaviour can
be exercised without spending a real session's context.
That fold is why `Cleared` is documented as load-bearing rather than
decorative. For any driver that rebuilds its conversation from the
transcript, this marker decides what the model sees, and treating it as
something only the phone draws would silently put the cleared
conversation back in front of the model at full price.
Clear rides the existing boundary pump like any other SessionCommand, so
one arriving mid-turn waits exactly as a compaction does, and nothing
grows a second way to wait.
Removes `--autocompact` in the same change, because clearing is the
cheaper answer to the problem it was added for and Bryan would rather
manage context that way. Keeping the measurement here, since it was the
reason for the constant and is worth more than the constant was:
context returned to 70-85k within ten calls of a compaction; a
compaction took 104,346 to 147,671 ms; compaction cost that session
2,655,508 tokens across six boundaries, of which the single automatic
one at the 1M ceiling was 1,696,870. Clearing costs nothing, because
nothing is sent.
70 tests, clippy clean, rustfmt clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VETa8afmpWaYezLCqJhDB8
This commit is contained in:
1 parent
797513bb86
commit
1e3eec8cf0
5 files changed
+122
-34
No files matched your search
@@ -19,7 +19,7 @@
|
|||||||
//! POST /sessions/{id}/interrupt
|
//! POST /sessions/{id}/interrupt
|
||||||
//! POST /sessions/{id}/title {title}
|
//! POST /sessions/{id}/title {title}
|
||||||
//! POST /sessions/{id}/model {model}
|
//! POST /sessions/{id}/model {model}
|
||||||
//! POST /sessions/{id}/command {text} -- /compact, /rename x, or the dialect's own
|
//! POST /sessions/{id}/command {text} -- /compact, /clear, /rename x, or the dialect's own
|
||||||
//! POST /sessions/{id}/compact
|
//! POST /sessions/{id}/compact
|
||||||
//! POST /sessions/{id}/attachments multipart image upload -> {id}, referenced by /message
|
//! POST /sessions/{id}/attachments multipart image upload -> {id}, referenced by /message
|
||||||
//! GET /sessions/{id}/files/{name} images the session produced or was sent
|
//! GET /sessions/{id}/files/{name} images the session produced or was sent
|
||||||
@@ -764,6 +764,7 @@ async fn command(
|
|||||||
};
|
};
|
||||||
match (name, rest) {
|
match (name, rest) {
|
||||||
("/compact", _) => lookup(&manager, &id)?.run_command(SessionCommand::Compact),
|
("/compact", _) => lookup(&manager, &id)?.run_command(SessionCommand::Compact),
|
||||||
|
("/clear", _) => lookup(&manager, &id)?.run_command(SessionCommand::Clear),
|
||||||
// Through the manager, not the session: a name is persisted and
|
// Through the manager, not the session: a name is persisted and
|
||||||
// listed as well as forwarded, and that is one operation.
|
// listed as well as forwarded, and that is one operation.
|
||||||
("/rename", "") => return Err(bad_request(anyhow::anyhow!("a session needs a name"))),
|
("/rename", "") => return Err(bad_request(anyhow::anyhow!("a session needs a name"))),
|
||||||
|
|||||||
@@ -176,36 +176,6 @@ const STDIN_FIFO: &str = "stdin.fifo";
|
|||||||
const STDOUT_LOG: &str = "stdout.log";
|
const STDOUT_LOG: &str = "stdout.log";
|
||||||
const STDERR_LOG: &str = "stderr.log";
|
const STDERR_LOG: &str = "stderr.log";
|
||||||
|
|
||||||
/// The context size the CLI compacts at, rather than letting it choose.
|
|
||||||
///
|
|
||||||
/// Left to `auto` a phone session drifts: these run for hours, nobody
|
|
||||||
/// closes them, and the transcript carries screenshots. Measured on one
|
|
||||||
/// here, the window the CLI chose was 1M and it compacted at the ceiling
|
|
||||||
/// -- its only automatic compaction fired at 1,000,184 tokens, and the
|
|
||||||
/// context peaked at 999,668. Every API call re-reads the whole context,
|
|
||||||
/// so near that ceiling a single tool call bills about 100k tokens
|
|
||||||
/// before it does anything.
|
|
||||||
///
|
|
||||||
/// What kept that session usable was the person in it running `/compact`
|
|
||||||
/// by hand, four times. That is the state this constant is really for:
|
|
||||||
/// even between those manual compactions, at the merely-large contexts
|
|
||||||
/// they left behind, one ordinary request ("can you make it so you can
|
|
||||||
/// rename a session?") cost 4.2 million tokens across its 130 calls.
|
|
||||||
///
|
|
||||||
/// 200k rather than the 100k floor, because the cheapest window on tokens
|
|
||||||
/// is not the best one to sit in front of. Measured on the same session:
|
|
||||||
/// context comes back to 70-85k within ten calls of a compaction, so a
|
|
||||||
/// 100k window compacts about every thirteen calls, and a compaction
|
|
||||||
/// takes roughly two minutes (`durationMs` 104,346 to 147,671 across the
|
|
||||||
/// six recorded). A 130-call request would spend some twenty minutes
|
|
||||||
/// compacting. 200k keeps most of the saving against the 1M ceiling and
|
|
||||||
/// halves the stalls.
|
|
||||||
///
|
|
||||||
/// Worth knowing before tuning this: the window decides how often the
|
|
||||||
/// context is thrown away, not how fast it fills. What fills it is ~7k
|
|
||||||
/// per call of tool output, and no value here touches that.
|
|
||||||
const AUTOCOMPACT_WINDOW: &str = "200k";
|
|
||||||
|
|
||||||
/// How often a reader with nothing to read looks again.
|
/// How often a reader with nothing to read looks again.
|
||||||
///
|
///
|
||||||
/// A poll rather than a watch: the alternative is an inotify dependency
|
/// A poll rather than a watch: the alternative is an inotify dependency
|
||||||
@@ -357,7 +327,6 @@ impl ClaudeDriver {
|
|||||||
};
|
};
|
||||||
push("--input-format", "stream-json");
|
push("--input-format", "stream-json");
|
||||||
push("--output-format", "stream-json");
|
push("--output-format", "stream-json");
|
||||||
push("--autocompact", AUTOCOMPACT_WINDOW);
|
|
||||||
// Hidden but load-bearing: without it the CLI resolves permissions
|
// Hidden but load-bearing: without it the CLI resolves permissions
|
||||||
// itself and nothing ever reaches the phone.
|
// itself and nothing ever reaches the phone.
|
||||||
push("--permission-prompt-tool", "stdio");
|
push("--permission-prompt-tool", "stdio");
|
||||||
@@ -600,6 +569,16 @@ impl Driver for ClaudeDriver {
|
|||||||
self.local_command("/compact".to_string());
|
self.local_command("/compact".to_string());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn clear(&self) {
|
||||||
|
// The CLI answers this with a fresh `init` carrying a new
|
||||||
|
// `session_id`, and the reader persists that as the resume token
|
||||||
|
// the moment it changes -- so the next launch of this session
|
||||||
|
// resumes the cleared conversation rather than the old one, with
|
||||||
|
// nothing here to keep in step.
|
||||||
|
let _ = self.sink.send(Event::Cleared);
|
||||||
|
self.local_command("/clear".to_string());
|
||||||
|
}
|
||||||
|
|
||||||
fn detach(&self) {
|
fn detach(&self) {
|
||||||
// Stop reading and leave everything else exactly as it is. The
|
// Stop reading and leave everything else exactly as it is. The
|
||||||
// process keeps its fifo (which it holds open itself), keeps
|
// process keeps its fifo (which it holds open itself), keeps
|
||||||
|
|||||||
@@ -244,6 +244,30 @@ pub enum Event {
|
|||||||
id: String,
|
id: String,
|
||||||
text: String,
|
text: String,
|
||||||
},
|
},
|
||||||
|
/// The conversation was cleared: everything above this is still in
|
||||||
|
/// the record but is no longer in the session's context.
|
||||||
|
///
|
||||||
|
/// Nothing is deleted. A transcript is the thing a person scrolls
|
||||||
|
/// back through, and a session that dropped its history from the
|
||||||
|
/// screen as well as from the model would lose the only copy the
|
||||||
|
/// phone has -- so this is a divider, not a truncation, and the
|
||||||
|
/// events before it stay exactly where they were.
|
||||||
|
///
|
||||||
|
/// It is also what makes clearing mean the same thing for every
|
||||||
|
/// driver, which is why the marker lives here rather than in one
|
||||||
|
/// dialect: `llama` folds its conversation out of the transcript and
|
||||||
|
/// simply folds from the last one of these, and `claude` starts a new
|
||||||
|
/// CLI conversation behind it.
|
||||||
|
///
|
||||||
|
/// **Load-bearing, not decorative.** For any driver that rebuilds its
|
||||||
|
/// conversation from the transcript, this marker decides what the
|
||||||
|
/// model is given -- dropping it, or treating it as something only
|
||||||
|
/// the phone draws, silently puts a cleared conversation back in
|
||||||
|
/// front of the model at full cost. Today `llama::conversation` is
|
||||||
|
/// the only fold that reads it, which is the reason to write this
|
||||||
|
/// down rather than leave it to be inferred from a second example
|
||||||
|
/// that does not exist yet.
|
||||||
|
Cleared,
|
||||||
Error {
|
Error {
|
||||||
message: String,
|
message: String,
|
||||||
},
|
},
|
||||||
@@ -260,6 +284,7 @@ pub enum Event {
|
|||||||
#[derive(Debug, Clone, PartialEq)]
|
#[derive(Debug, Clone, PartialEq)]
|
||||||
pub enum SessionCommand {
|
pub enum SessionCommand {
|
||||||
Compact,
|
Compact,
|
||||||
|
Clear,
|
||||||
SetTitle(String),
|
SetTitle(String),
|
||||||
Raw(String),
|
Raw(String),
|
||||||
}
|
}
|
||||||
@@ -270,6 +295,7 @@ impl SessionCommand {
|
|||||||
pub fn label(&self) -> String {
|
pub fn label(&self) -> String {
|
||||||
match self {
|
match self {
|
||||||
Self::Compact => "/compact".to_string(),
|
Self::Compact => "/compact".to_string(),
|
||||||
|
Self::Clear => "/clear".to_string(),
|
||||||
Self::SetTitle(title) => format!("/rename {title}"),
|
Self::SetTitle(title) => format!("/rename {title}"),
|
||||||
Self::Raw(text) => text.clone(),
|
Self::Raw(text) => text.clone(),
|
||||||
}
|
}
|
||||||
@@ -279,6 +305,7 @@ impl SessionCommand {
|
|||||||
pub fn apply(&self, driver: &dyn Driver) {
|
pub fn apply(&self, driver: &dyn Driver) {
|
||||||
match self {
|
match self {
|
||||||
Self::Compact => driver.compact(),
|
Self::Compact => driver.compact(),
|
||||||
|
Self::Clear => driver.clear(),
|
||||||
Self::SetTitle(title) => driver.set_title(title),
|
Self::SetTitle(title) => driver.set_title(title),
|
||||||
Self::Raw(text) => driver.run_command(text),
|
Self::Raw(text) => driver.run_command(text),
|
||||||
}
|
}
|
||||||
@@ -371,6 +398,20 @@ pub trait Driver: Send + Sync {
|
|||||||
fn run_command(&self, text: &str);
|
fn run_command(&self, text: &str);
|
||||||
/// pi: native compaction; claude: `/compact`.
|
/// pi: native compaction; claude: `/compact`.
|
||||||
fn compact(&self);
|
fn compact(&self);
|
||||||
|
|
||||||
|
/// Drops the conversation so far without ending the session.
|
||||||
|
///
|
||||||
|
/// The cheap half of managing a long session, and the reason it is a
|
||||||
|
/// driver operation rather than a manager one: compaction *reads* the
|
||||||
|
/// whole conversation in order to summarise it, so on a large context
|
||||||
|
/// it is itself one of the most expensive requests the session will
|
||||||
|
/// make -- measured at 1.7 million tokens for a single automatic
|
||||||
|
/// compaction on 2026-08-29. Clearing costs nothing, because nothing
|
||||||
|
/// is sent.
|
||||||
|
///
|
||||||
|
/// Every implementation emits [`Event::Cleared`] so the transcript
|
||||||
|
/// carries the divider whatever the dialect did behind it.
|
||||||
|
fn clear(&self);
|
||||||
/// Stop attending to the process but leave it running, because this
|
/// Stop attending to the process but leave it running, because this
|
||||||
/// server is going away and means to adopt it again when it comes
|
/// server is going away and means to adopt it again when it comes
|
||||||
/// back.
|
/// back.
|
||||||
|
|||||||
@@ -564,6 +564,14 @@ impl Driver for EchoDriver {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The same marker a real driver leaves, and nothing else -- there is
|
||||||
|
/// no context here to drop. It exists so the phone's divider, its
|
||||||
|
/// scroll behaviour and the transcript's shape can be exercised
|
||||||
|
/// without spending a real session's context to produce one.
|
||||||
|
fn clear(&self) {
|
||||||
|
let _ = self.sink.send(Event::Cleared);
|
||||||
|
}
|
||||||
|
|
||||||
/// Nothing to detach from and nothing to stop: the echo driver has no
|
/// Nothing to detach from and nothing to stop: the echo driver has no
|
||||||
/// process, so both halves of the way out are already done.
|
/// process, so both halves of the way out are already done.
|
||||||
fn detach(&self) {}
|
fn detach(&self) {}
|
||||||
|
|||||||
@@ -411,11 +411,19 @@ impl Driver for LlamaDriver {
|
|||||||
|
|
||||||
fn compact(&self) {
|
fn compact(&self) {
|
||||||
let _ = self.sink.send(Event::Error {
|
let _ = self.sink.send(Event::Error {
|
||||||
message: "llama.cpp has no compaction. When the context fills, start a new session."
|
message: "llama.cpp has no compaction. Clear the session instead, which costs nothing."
|
||||||
.to_string(),
|
.to_string(),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn clear(&self) {
|
||||||
|
// All of it. `conversation` folds from the last of these, so
|
||||||
|
// recording the marker *is* the reset -- there is no driver state
|
||||||
|
// to keep in step with it, which is the same property that makes
|
||||||
|
// a second device see the same conversation this one does.
|
||||||
|
let _ = self.sink.send(Event::Cleared);
|
||||||
|
}
|
||||||
|
|
||||||
/// Stops generating and leaves the server loaded.
|
/// Stops generating and leaves the server loaded.
|
||||||
///
|
///
|
||||||
/// Worth being deliberate about, because the cost is asymmetric and
|
/// Worth being deliberate about, because the cost is asymmetric and
|
||||||
@@ -457,7 +465,14 @@ fn conversation(path: &Path) -> Vec<Message> {
|
|||||||
};
|
};
|
||||||
let mut messages: Vec<Message> = Vec::new();
|
let mut messages: Vec<Message> = Vec::new();
|
||||||
let mut pending = String::new();
|
let mut pending = String::new();
|
||||||
for event in events {
|
// Everything before the last clear is still in the transcript and is
|
||||||
|
// deliberately not in the conversation. Folding from zero here would
|
||||||
|
// put it back, which is the whole of what clearing had to undo.
|
||||||
|
let events = match events.iter().rposition(|e| e.event == Event::Cleared) {
|
||||||
|
Some(at) => &events[at + 1..],
|
||||||
|
None => &events[..],
|
||||||
|
};
|
||||||
|
for event in events.iter().cloned() {
|
||||||
match event.event {
|
match event.event {
|
||||||
Event::UserMessage { text } => {
|
Event::UserMessage { text } => {
|
||||||
if !pending.is_empty() {
|
if !pending.is_empty() {
|
||||||
@@ -698,6 +713,50 @@ mod tests {
|
|||||||
assert_eq!(messages[1].content, "still here");
|
assert_eq!(messages[1].content, "still here");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
/// Clearing decides what the *model* is given, not just what the
|
||||||
|
/// phone draws. Everything above the marker stays in the transcript
|
||||||
|
/// -- a person can still scroll back to it -- and none of it is sent.
|
||||||
|
fn the_conversation_starts_after_the_last_clear() {
|
||||||
|
let (_dir, path) = transcript_with(&[
|
||||||
|
Event::UserMessage {
|
||||||
|
text: "the long expensive conversation".into(),
|
||||||
|
},
|
||||||
|
Event::AssistantText {
|
||||||
|
delta: "at length".into(),
|
||||||
|
},
|
||||||
|
Event::Cleared,
|
||||||
|
Event::UserMessage {
|
||||||
|
text: "a fresh start".into(),
|
||||||
|
},
|
||||||
|
Event::AssistantText {
|
||||||
|
delta: "cheaply".into(),
|
||||||
|
},
|
||||||
|
]);
|
||||||
|
let messages = conversation(&path);
|
||||||
|
assert_eq!(messages.len(), 2);
|
||||||
|
assert_eq!(messages[0].content, "a fresh start");
|
||||||
|
assert_eq!(messages[1].content, "cheaply");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
/// The *last* one, so clearing twice does not resurrect what the
|
||||||
|
/// first clear dropped.
|
||||||
|
fn only_the_newest_clear_counts() {
|
||||||
|
let (_dir, path) = transcript_with(&[
|
||||||
|
Event::UserMessage { text: "one".into() },
|
||||||
|
Event::Cleared,
|
||||||
|
Event::UserMessage { text: "two".into() },
|
||||||
|
Event::Cleared,
|
||||||
|
Event::UserMessage {
|
||||||
|
text: "three".into(),
|
||||||
|
},
|
||||||
|
]);
|
||||||
|
let messages = conversation(&path);
|
||||||
|
assert_eq!(messages.len(), 1);
|
||||||
|
assert_eq!(messages[0].content, "three");
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn a_model_key_cannot_climb_out_of_the_models_directory() {
|
fn a_model_key_cannot_climb_out_of_the_models_directory() {
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
|
|||||||
Reference in new issue
Block a user