iris-aiandClaude Opus 5
ac476ab0c9
A llama session was a chat box: no tools, a fixed model, no permission
mode, and a model name drawn as the path the file sits at. It now runs the
agent loop itself, which is what the pieces below all hang off.
Tools are `llama-server`'s own (`--tools all`), which that server both
publishes and runs -- `GET /tools` for the definitions, `POST /tools` to
call one. Web search is Exa's MCP server, reached from this backend rather
than from the machine serving the model: that is what llama.cpp's own web
UI does, and it puts the search on the machine with a route out instead of
the one with the GPU. `llama-server`'s `--mcp-servers-json` can only spawn
local commands, so using it would have meant a Node bridge on every
machine that serves a model.
Driving the loop is what makes the permission gate ours. Two modes,
`manual` and `bypassPermissions`, which is what the mechanism has: the web
UI asks before every call and remembers the tools you say "always" to. The
allowances fold back out of the transcript's own answers, so they survive
a restart and a model change without being stored anywhere else.
Also here, because tools made each of them matter:
- **Loading is a state.** A 12 GB model takes twenty seconds to reach
memory and refuses everything until it has; the session used to report
`running` for that whole time, and a message sent meanwhile came back as
an error. It is `loading` now, and the message waits.
- **The model can be changed.** A `llama-server` holds one model, so this
stops it and starts another. The conversation survives because it was
never in the server.
- **Models are named, not pathed.** `general.name` read out of the file
itself -- over ssh too, in the round trip the spawn was already making.
Where two models share a name the file name breaks the tie.
- **`-np 1`, and the MTP draft head where the file has one.** Measured on
the 27B here: 41.5 tok/s plain, 61.4 with `--spec-type draft-mtp` at one
slot, and 28 with it at four -- speculating against a split KV cache is
worse than not speculating. The flag is conditional because asking for a
head that is not there makes `llama-server` exit.
- **A refusal says what to do.** Tool results are thousands of tokens, so
an overrun context is now ordinary; it was "http status: 400" and is now
the server's own "exceeds the available context size, try increasing it".
`GET /machines/{id}/models` is gone: the provider models route answers the
same question, and two answers to one question is how a picker comes to
offer a model the spawn screen does not.
Verified end to end against real models: a tool call asked and allowed, an
Exa search, a shell command, a 27B loaded while a message waited on it, a
model switch mid-session, a second message queued behind a running turn,
and the whole of it again on a session running over ssh.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
295 lines
18 KiB
Markdown
295 lines
18 KiB
Markdown
---
|
||
name: ai-app-rigs
|
||
description: ai-app's test rigs, harness scripts and reference measurements - ui-sandbox.sh, debug-transcript.sh, transcript-bench.sh, stream-bench.sh, trace-draw.sh, the /usage fixture vocabulary, the fake CLI, the rule that no UI-driving script may tap a coordinate, how to test llama.cpp and ssh on this machine, how importing behaves, and the scroll/stream/explorer numbers not worth re-measuring. Read before running or writing a benchmark, driving the app's UI from a script, exercising the session lifecycle, testing a llama or remote session, or touching the import screen.
|
||
---
|
||
|
||
# ai-app: rigs, harnesses and measurements
|
||
|
||
Moved out of `AGENTS.md` on 2026-09-04 so it is read when it is relevant
|
||
rather than sent with every request in this repo -- it was 12 KB of the 35 KB
|
||
that file cost on every one. Unchanged in the move, and still the only copy.
|
||
|
||
## The rigs
|
||
|
||
Each exists because something was invisible without it.
|
||
|
||
- **`app/ui-sandbox.sh`** — a second `ai-server` with its own `$HOME`, config
|
||
and data directory, holding eight invented Claude Code transcripts and a
|
||
`claude` that is two lines of shell. **That isolation is the point**: the
|
||
import screen lists whatever is in `~/.claude/projects`, which in this VM is
|
||
real agent transcripts, so exercising *delete* against the ordinary server
|
||
deletes somebody's conversation and exercising *import* starts a real
|
||
`--resume` on the owner's account.
|
||
Its port and root derive from the checkout's name, so two checkouts'
|
||
sandboxes cannot reach each other, and its token is generated once into
|
||
`~/.config/ai-app/sandbox-token` and carried across restarts along with any
|
||
the enrolment flow appended — so the emulator app is enrolled **once** (the
|
||
start banner prints the command) and stays enrolled. It shares the real TLS
|
||
certificates, because the installed APK pins that CA.
|
||
Driving verbs, so none of this is re-derived per session:
|
||
`./ui-sandbox.sh spawn [title]` (an echo session, prints its id),
|
||
`./ui-sandbox.sh send SID text|@file`, and
|
||
`./ui-sandbox.sh api /path [curl args]`.
|
||
`./ui-sandbox.sh keep` restarts the server without wiping the sessions and
|
||
enrolment already there — for when the fixture under test was expensive to
|
||
build; plain `start` wipes them, which is right for the list-screen
|
||
fixtures and wrong for that.
|
||
It passes `--delay` by default, and `AI_SANDBOX_BIG_MB` puts one large
|
||
transcript among the small ones while `AI_SANDBOX_SPAWN_DELAY` makes the
|
||
fake CLI slow to start. Both exist because operations that finish in
|
||
milliseconds have states on the way that nothing can observe, and an
|
||
unobservable state is one where broken and working look identical.
|
||
It also builds a fixture tree at the sandbox home's `~/files` for the
|
||
explorer, holding the states otherwise only reachable by finding a real
|
||
machine in one: an empty directory, a name with a tab and one with an
|
||
apostrophe, a binary file, one over `FILE_LIMIT`, one `chmod 000`, a
|
||
symlink to a directory and a broken one, a source file per language, and
|
||
the three sizes the limits were measured against (`edit-32k.rs`,
|
||
`edit-128k.rs`, `big-source.rs`). Point a session at it with
|
||
`./ui-sandbox.sh api /sessions/<id>/cwd -X POST -H 'content-type: application/json' -d '{"cwd":"~/files"}'`.
|
||
The explorer's 409 is produced by editing the file on the machine
|
||
(`printf … > file`) between pressing the pencil and pressing save.
|
||
- **`app/debug-transcript.sh`** — a real conversation on the emulator. The
|
||
echo driver is the right rig for most things and the wrong one for anything
|
||
whose cost scales with what was actually written: a real reply is longer,
|
||
is real markdown, and carries tool calls whose input and output are
|
||
kilobytes. Two faults were invisible until a real transcript was loaded — a
|
||
page of history landing mid-fling threw the reader back to the newest end,
|
||
and parsing one real reply took 51ms against 4.6ms for a synthetic one.
|
||
`-b` takes the biggest conversation on the machine rather than the newest,
|
||
which is what a scrolling test wants; `--stop` takes it down.
|
||
It copies the transcript into `/tmp` and gives the server a `HOME` of its
|
||
own, so the import can only see the copy — importing spawns `claude
|
||
--resume`, and against the real file that is a second CLI writing to a
|
||
conversation somebody may still be in. **A transcript never goes in this
|
||
repository**: they hold whatever was said, read and written in that
|
||
session, and `~/repos` is shared with the host besides.
|
||
- **`/usage` in an echo session puts up an invented meter**, which is how the
|
||
rate-limit screens' states are reached without spending quota: `/usage 42`,
|
||
`/usage 95 20` (minutes left), `/usage 42 never` (the between-blocks window
|
||
with no reset time), `/usage 42 unreadable`, `/usage notloggedin`,
|
||
`/usage unreachable`, `/usage failed`, `/usage off`. The vocabulary is
|
||
`usage::Fixture`'s, since those are its states. With none set an echo
|
||
session meters nothing, which is the ordinary case and draws no bar.
|
||
- **A fake CLI exercises the process lifecycle without a token.** Point a
|
||
`claude_cli` provider's `command` at a script that ordinarily runs
|
||
`cat > /dev/null` and it behaves the way the lifecycle code cares about:
|
||
it holds the fifo open, records a real pid, writes nothing, and dies on a
|
||
signal. So adopt, stop, restart and start are all drivable without a real
|
||
`--resume` and without spending a turn on somebody's account. Reach for
|
||
this when what is under test is *whether a process is running*, and for
|
||
`debug-transcript.sh` when it is *what the transcript draws*. The sandbox's
|
||
version also handles `auth login`: it prints an inert Anthropic-shaped URL,
|
||
rejects any code except `sandbox-code`, and exits successfully for that one.
|
||
- **`app/transcript-bench.sh`** is the standard scroll measurement: it opens
|
||
the first session (or `-k` keeps the current screen), scrolls a fixed
|
||
gesture loop, and prints the app's render report — the same one the in-app
|
||
copy button produces, whose `on screen:` line names what the viewport was
|
||
holding. Compare two runs with the same gestures; the emulator's absolute
|
||
frame times transfer nothing, the report's accounting does. Run it either
|
||
side of any change under `Markdown*.kt`, `Transcript*.kt` or
|
||
`SessionScreen.kt`'s list, and put the report in the commit. The numbers
|
||
that move first are the worst `record: one block`, the reparse mean while
|
||
streaming, and the draw phase's accounting line.
|
||
- **`app/stream-bench.sh [-k] FILE`** is that measurement for a reply still
|
||
arriving. It taps "Jump to latest" so the list is pinned to the newest end,
|
||
resets the report, sends FILE, waits for the transcript to stop growing,
|
||
and prints. Both of those are corrections to a first version that measured
|
||
nothing: a transcript parked further back never redraws while a reply
|
||
streams into it, and a session is idle at *both* ends of a turn, so polling
|
||
for idle answers before the turn has started.
|
||
- **`app/trace-draw.sh`** names what a scrolling frame spends inside the
|
||
framework, from `atrace` text output with no trace processor needed. It is
|
||
how the cost of a layout node per link was attributed to the framework
|
||
rather than guessed at.
|
||
|
||
### Driving the UI
|
||
|
||
**No script that drives this app's UI presses a coordinate.** Every control
|
||
is found by the name it already carries for assistive technology —
|
||
`ui-trace record --do "tap 'Session settings'"` — which resolves the label
|
||
against the screen at the moment of the gesture and fails the whole run when
|
||
it is not there. `app/bench-lib.sh` is what the bench scripts share for it. A
|
||
coordinate is a position measured once by hand, and anything that moves the
|
||
control makes the tap land on whatever now sits there — the bench then
|
||
reports a number that was never measured, which reads exactly like a result.
|
||
Both bench scripts pressed the render report at `tap 723 205` until that
|
||
button moved into the session settings dialog on 2026-09-03. The check that
|
||
none has crept back:
|
||
|
||
grep -n "tap [0-9]" app/*.sh
|
||
|
||
Swipes are still coordinates, deliberately: a gesture across a scrolling area
|
||
is a distance rather than a control.
|
||
|
||
**Two traps in the emulator bench loop**, each of which cost a run.
|
||
`adb shell pm clear` removes the enrolment and the notification permission
|
||
along with the saved anchors, so the next run measures a permission dialog —
|
||
re-enrol with the command `ui-sandbox.sh` prints, and
|
||
`pm grant … POST_NOTIFICATIONS`. And a saved scroll anchor is per session id,
|
||
so the only way two builds start a scroll from the same place is a *fresh
|
||
session for each*.
|
||
|
||
**The emulator is `~/repos/emulator-tools`' business, not this repo's.**
|
||
`emu up` creates and boots the AVD named after this checkout — whatever `emu
|
||
name` prints, never a name typed out here, since this file is the same in
|
||
every clone. `run-android.sh` is that plus a build and an install. The `adb`
|
||
on `PATH` after sourcing `android-env.sh` is that repo's wrapper, which fills
|
||
in `-s` from the same rule. Gradle does not go through it, so a Gradle init
|
||
script from `emulator-tools` runs `emu check` before `installDebug`,
|
||
`uninstallDebug` and `connectedAndroidTest` and fails rather than fanning out
|
||
to every attached device; when it refuses, say which device you mean at the
|
||
moment you use it — `ANDROID_SERIAL=$(emu serial) ./gradlew …`.
|
||
|
||
### Testing llama.cpp and ssh here
|
||
|
||
**Both are set up here** and need nothing typed. The prebuilt llama.cpp lives
|
||
outside the repo at `~/.local/opt/llama.cpp-vk` — a **Vulkan** build as of
|
||
2026-09-19, replacing the CPU one that was there before — and is symlinked as
|
||
both `~/.local/bin/llama-server` and `/usr/local/bin/llama-server`. The second
|
||
is what makes **discovery find it over ssh**: `~/.local/bin` is not on the
|
||
PATH a non-interactive ssh session gets. It resolves its own libraries through
|
||
`$ORIGIN`, so no `LD_LIBRARY_PATH` is needed.
|
||
|
||
Two models are downloaded under `~/.local/share/ai-app/models`:
|
||
|
||
- `unsloth/Qwen3-0.6B-GGUF/Qwen3-0.6B-Q8_0.gguf`, 639 MB, loads in ~4s. It
|
||
calls tools correctly and is the right rig for the driver's shape. Do not
|
||
judge *answers* by it — asked for the second line of a file it read from
|
||
line 2 and then named the third.
|
||
- `ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf`,
|
||
12 GB, ~20s to load, and the only one here with a multi-token-prediction
|
||
head. It is the rig for anything about `loading` being a state of its own,
|
||
since 20s is long enough to send into.
|
||
|
||
**Do not test with a 2-bit quant**: the IQ2_XXS of the 0.6B produces fluent
|
||
nonsense, which reads exactly like a broken driver — `llama-cli` produces the
|
||
same from the file directly, which is how to tell the two apart in a hurry.
|
||
|
||
**The GPU is shared and llama-server dies loudly when it runs out.** A second
|
||
server loading a model while the 27B holds VRAM fails with `radv/amdgpu:
|
||
Failed to allocate a buffer` / `MESA: error: buffer allocation failed` and
|
||
exits mid-request. `-ngl 0` runs it on the 8 cores instead, which is the way
|
||
to test the driver while something else holds the card.
|
||
|
||
**Testing tools and MCP without the app**: `llama-server --tools all` publishes
|
||
its built-in tools at `GET /tools` and runs one at `POST /tools` with
|
||
`{"tool": …, "params": …}` and an `x-tool-cwd` header — so a whole agent loop
|
||
is drivable with `curl` and no model at all. The Exa MCP server at
|
||
`https://mcp.exa.ai/mcp` answers **without an API key** and needs a
|
||
`User-Agent` header (Cloudflare answers 403 without one, which reads as a
|
||
refusal rather than a missing header).
|
||
|
||
There is no second machine, so **ssh this VM to itself**. That is set up
|
||
too: the key is `~/.config/ai-app/ssh-self` (its public half is in
|
||
`~/.ssh/authorized_keys`, labelled removable), and the real config carries a
|
||
machine called **"this vm over ssh"** — `bob@127.0.0.1` with that
|
||
`identityFile` plus
|
||
`options: ["StrictHostKeyChecking=no", "UserKnownHostsFile=/tmp/ai-app-known-hosts"]`
|
||
so it touches nothing real — offering `claude-cli` and `llama-cpp`. It is the
|
||
whole rig for "does a remote llama session work", since the far machine is
|
||
this one and the model file is the same file. For a throwaway machine of your
|
||
own, point a provider's `command` at something harmless like `/bin/echo`
|
||
rather than at `claude`: the transport is what is under test, the process
|
||
exiting immediately is the signal, and it costs no tokens. The remote login
|
||
shell here is **fish**; the
|
||
remote script and `ssh.rs`'s POSIX quoting happen to mean the same thing in
|
||
both, but that is luck rather than design, and a shell that is neither is the
|
||
thing to suspect first if a remote spawn ever mangles an argument.
|
||
|
||
## Importing
|
||
|
||
The import list reports each session's **size as well as its line count**,
|
||
because the two disagree in the way that matters: these transcripts embed
|
||
screenshots as base64, so one line can be a megabyte. On this machine a 69 MB
|
||
session has 3,427 lines and a 44 MB one has 6,792 — nothing about a line
|
||
count tells you what continuing a session will cost. Shown, not warned about;
|
||
importing a large session is a choice somebody is entitled to make.
|
||
|
||
**Never import a Claude Code session that is open in a terminal.** The app
|
||
refuses it — see PLAN.md for the incident that made that a refusal rather
|
||
than a warning.
|
||
|
||
**One Claude Code session id can name two files, and the listing offers it
|
||
once.** Resuming from a different working directory makes the CLI write a
|
||
second transcript with the same id under that directory's project folder — an
|
||
ordinary state of a machine, not corruption. Everything downstream addresses
|
||
a session by id, and the phone keyed its list on it, so two rows sharing one
|
||
**closed the app** on a Compose duplicate-key throw. `parse_listing` keeps
|
||
the copy with the most lines, because the other is usually a few-hundred-byte
|
||
stub and is often the *newer* of the two, so recency is the wrong key.
|
||
Deleting removes every copy rather than the first, or the row came back after
|
||
a delete that reported success. The phone's half is `uniqueItems`, which
|
||
every list keyed on a server-chosen id goes through: a repeat there must
|
||
never be able to close the app, whatever produced it.
|
||
|
||
**Deleting a session offers to take the machine's own transcript with it** —
|
||
`DELETE /sessions/{id}?deleteForeign=true`, behind a switch in the
|
||
confirmation, and only where the driver keeps a record of its own
|
||
(`keepsOwnTranscript`, currently Claude Code or Codex). Off by default,
|
||
because leaving that copy is what makes an ordinary delete recoverable — and
|
||
the dialog's paragraph is rewritten when it is on rather than appended to,
|
||
since the sentence promising the conversation "should still be there to
|
||
import again" is exactly the one the switch makes false. The server deletes
|
||
the machine's copy *first*, so a machine it cannot reach leaves the session
|
||
where it was instead of half-deleted.
|
||
|
||
## Measurements worth not re-taking
|
||
|
||
- **`-np 1` is what makes the MTP draft head pay.** Taken 2026-09-19 on the
|
||
27B above, decode speed for a 300-token reply, from `llama-server`'s own
|
||
timings rather than the clock:
|
||
|
||
| flags | tok/s |
|
||
| --- | --- |
|
||
| plain, any `-np` | 41.5 |
|
||
| `--spec-type draft-mtp -np 1` | 61.4 |
|
||
| `--spec-type draft-mtp -np 2` (n-max 2) | 65.9 |
|
||
| `--spec-type draft-mtp`, default `-np` (4 slots) | 28 |
|
||
|
||
Draft acceptance is 0.53–0.73 in every case, so the head is working in all
|
||
of them: what changes is that speculating against a KV cache split four ways
|
||
is slower than not speculating. The driver passes `-np 1` always, so this is
|
||
recorded for whoever next sees MTP look broken. `--spec-draft-n-max 2` was
|
||
worth another 7% in a single sample and is deliberately *not* passed — one
|
||
sample on a virtualised GPU is not a number to hardcode.
|
||
|
||
- **Asking for the head when the file has none is fatal**, not ignored:
|
||
`context type MTP requested but model doesn't contain MTP layers` and the
|
||
server exits. Without the flag the same file logs `unused tensor
|
||
blk.N.nextn.* — ignoring` and runs normally, which is the state to look for
|
||
when MTP is silently not happening.
|
||
|
||
- **What the transcript screen costs to scroll.** Taken 2026-08-30 on the GPU
|
||
emulator against a real imported transcript with the server at
|
||
`--delay 120`. Settled and flinging fast, both into fresh history and back
|
||
through rows already drawn: **5.2–5.9% janky frames, 99th percentile
|
||
29–32ms, 0–2 slow UI-thread frames.** The stock Settings app on the same
|
||
device is 3.3% and 38ms, so this is at the platform floor. The number that
|
||
is *not* at the floor is the first few seconds after opening a session,
|
||
where every row on the way is being composed for the first time; that is
|
||
inherent to a lazy list and it is why a measurement taken before the screen
|
||
settles reads three times worse. **Settle first, then reset `gfxinfo`.**
|
||
- **The reset path is not reachable by reopening a session.** Measured
|
||
2026-09-04 against a session streaming at 20 events a second: reopening one
|
||
with an anchor 1,800 events back connects **87–119 events behind**, well
|
||
under `CATCH_UP_LIMIT`'s 200, because the restore is two requests — the
|
||
opening page, then one span covering the whole distance. To exercise the
|
||
reset at all you have to lower `CATCH_UP_LIMIT` in a throwaway build; at 5
|
||
the app takes the reset on a live connection, clears, refills and carries
|
||
on without reconnecting.
|
||
- **The session screen's stream survives backgrounding here** — 20 seconds at
|
||
the launcher while 415 events were produced brought no reconnect at all,
|
||
which is not what the comment above that loop expects, and is most likely
|
||
this emulator being headless rather than the phone's behaviour.
|
||
- **Reopening a cached session costs one request for one event** (the probe),
|
||
and scrolling the whole conversation back costs nothing more; a cold open
|
||
of the same 500-event session is two pages, 100 events. Measured
|
||
2026-09-04 on the emulator against the sandbox.
|
||
- **Reading is cheap and editing is not.** The viewer handles a 1 MiB,
|
||
28,000-line file because it draws one row per line; the editor is one
|
||
`BasicTextField`, which costs two seconds a frame at 128 kB and stops the
|
||
app at 1 MiB, so `EDIT_LIMIT` caps it at 32 kB with the reason said on
|
||
screen. If you make the editor faster, that number is what to move.
|
||
EXPLORER.md's "What the measurements said" has the rest.
|