irisandClaude Opus 5
3deeffd1e7
The second half of the llama.cpp work: a session can now name a downloaded model and talk to it. `llama-server` is spawned through the same transport as any other driver, polled until the model is loaded, then driven over its OpenAI-compatible streaming endpoint and translated into the same events the Claude driver emits -- so the transcript, the SSE stream and the phone need to know nothing new. **The conversation is rebuilt from the transcript, not held in the driver.** llama-server is stateless between requests, so the whole history goes with every one, and the obvious place to keep it is a Vec in the driver. That fails the requirement: memory in a driver is invisible to a second device and gone on restart, and this app is meant to work across devices. Reading it back also means the model is prompted with exactly what the phone was shown -- including a reply that was interrupted half way, which is in the transcript because the deltas were already emitted. That leaves the Claude driver as the odd one out rather than this one: the CLI's memory of a conversation is a cache in front of the same transcript, not a second truth. Said so at the top of llama.rs, because it is the sort of inconsistency that gets "fixed" in the wrong direction. Session settings arrive as a driver-interpreted `params` map rather than new typed fields, so the shared schema does not grow one dialect's vocabulary. Context size, gpu layers and threads become server flags; temperature and the rest ride on each request, so changing them need not reload a model. **Also fixes an orphan this feature would have created.** Drivers set kill_on_drop, which covers a session being deleted -- but nothing drops on the way out of a SIGTERM, so signalling the server left its children running. For the Claude CLI that is untidy; for a llama-server holding a model it is gigabytes belonging to nobody. The server now stops its sessions on SIGTERM and SIGINT. Found by killing a test server and noticing two 600 MB processes still resident. Remote llama sessions are refused rather than half-working: the model is reached over HTTP, and forwarding that port to an ssh host is the "reach this port" operation the transport does not have yet. Verified end to end against a real model: downloaded Qwen3-0.6B Q8_0 through the app's own download route, spawned a session on it, and held a two-turn conversation -- "my favourite colour is teal" then "what is my favourite colour?", answered "teal", which is the transcript replay doing its job. Token counts arrive. An earlier attempt with the IQ2_XXS quant produced fluent nonsense, which turned out to be the quantisation rather than the pipeline: llama-cli produces the same from that file directly. Four unit tests cover the fold and the path guard. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017xn8nHw1tw1R6PtiY1eEtw
59 lines
2.5 KiB
TOML
59 lines
2.5 KiB
TOML
[package]
|
|
name = "ai-server"
|
|
version = "0.1.0"
|
|
edition = "2024"
|
|
|
|
[[bin]]
|
|
name = "ai-server"
|
|
path = "src/main.rs"
|
|
|
|
[dependencies]
|
|
axum = { version = "0.8", features = ["json", "multipart"] }
|
|
axum-server = { version = "0.8", features = ["tls-rustls"] }
|
|
tokio = { version = "1", features = ["rt-multi-thread", "macros", "net", "sync", "time", "process", "io-util", "signal"] }
|
|
tokio-stream = "0.1"
|
|
tracing = "0.1"
|
|
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
|
serde = { version = "1", features = ["derive"] }
|
|
serde_json = "1"
|
|
# The config file's format. Not JSON, because this file is written and read
|
|
# by hand and RON says a sum type as syntax -- the same choice, and the same
|
|
# house rules, as the sibling dev-updater project's config.
|
|
ron = "0.12.2"
|
|
clap = { version = "4", features = ["derive"] }
|
|
anyhow = "1"
|
|
thiserror = "2"
|
|
# Token auth: hash for storage, constant-time compare for verification,
|
|
# CSPRNG-backed generation, base64url for the enrollment string.
|
|
sha2 = "0.11"
|
|
subtle = "2"
|
|
rand = "0.10"
|
|
base64 = "0.23"
|
|
# Renders the enrollment QR straight to the terminal; no image output needed.
|
|
qrcode = { version = "0.14", default-features = false }
|
|
# The wg0-bound listener needs the interface's address; the stdlib has no
|
|
# getifaddrs. This is the smallest crate that wraps just that.
|
|
if-addrs = "0.15"
|
|
# Generates this server's TLS certificates on first start, replacing a
|
|
# setup script that shelled out to whatever openssl happened to be
|
|
# installed. In process means one place decides the extensions, the file
|
|
# modes, and which addresses the leaf covers. x509-parser so the issuer is
|
|
# read back from the CA actually on disk: reconstructing it from the same
|
|
# parameters would work only as long as nothing ever changed them, and a
|
|
# mismatched issuer name yields a chain that fails to validate rather than
|
|
# anything that looks wrong at generation time.
|
|
rcgen = { version = "0.14", features = ["pem", "x509-parser"] }
|
|
# Outbound HTTPS for the usage endpoint. A small blocking client fits an
|
|
# every-few-minutes poll better than pulling in reqwest's tower stack;
|
|
# rustls-backed like the rest of the TLS here.
|
|
ureq = { version = "3", features = ["json"] }
|
|
# Direct dependency only to pick the process-level CryptoProvider in main:
|
|
# ureq pulls rustls-with-ring, axum-server rustls-with-aws-lc-rs, and with
|
|
# both in the graph rustls refuses to auto-select one.
|
|
rustls = "0.23"
|
|
|
|
[dev-dependencies]
|
|
tempfile = "3"
|
|
# ServiceExt::oneshot, to drive the auth middleware without a socket.
|
|
tower = { version = "0.5", features = ["util"] }
|