iris/ai-app
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Attach any file, take shares from other apps, and survive a backwards highlight

Browse Source 
Attachments were images only. Now any file can be attached: from the file
chooser behind the "+" menu, or from Android's share sheet, which the app
is now in. An image still goes to the model as a picture; anything else is
stored under its own name (`<hex>-<name>`, cleaned by `safe_file_name`)
and the Claude driver ends the message with `Attached file: /abs/path`,
since the CLI reads files by path and a model cannot be shown a trace. The
user-message field is renamed `images` -> `attachments` on both sides,
with a serde alias reading the rows written before. A share arrives before
anyone has said which session it is for, so it is held in AppRoot with a
banner on the list until a session takes it; an open session takes it at
once. Unreadable shares are reported beside the composer, not thrown.

The tool card crashed the app when opened on a command holding a quoted
glob such as `-path '*/.git/*'`: highlights 1.1.0's shell lexer answers
`x '*/a/*'` with a span whose end is before its start, and AnnotatedString
refuses the range. Such spans are dropped; the library is the place for
the fix. The echo driver gains `/bash <command>` so a card with a given
command can be produced on the emulator.

ui-sandbox.sh's token salvage read the tokens block's close only at a line
start, ran past the compact `),],` the server writes, and copied `setups`
into the new config twice, which the server then refused.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
irisandClaude Fable 5.1 committed 2026-09-03 08:15:10 -04:00
1 parent 4bc69e8f9c
commit 6180663f14
25 files changed
+653 -146
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+13
View File
@@ -65,6 +65,19 @@ repo is in PLAN.md's "Backend layout" section.
refuse to resume onto a partial from a different revision, and are
checked against HuggingFace's published sha256 before the file gets its
real name.
- **Attachments** are one list on a user message (`attachments`, the
ref the files route serves), in two shapes. An image is `<hex>.<ext>`
and goes to the model as an image block. Anything else is
`<hex>-<name>` -- the name it was shared or picked under, cleaned by
`safe_file_name` -- and the Claude driver appends `Attached file:
/abs/path` to the message text, since the CLI reads files by path and
a model cannot be shown a trace. `media::media_type_for` on the server
and `isImageRef` on the phone tell the two apart; keep those lists
level. The phone attaches from the photo picker, the file chooser and
Android's share sheet (`Share.kt`; the manifest's SEND filter), all
through one `attach` path in `SessionScreen`. Files exist only on the
server's machine -- see PLAN.md's "Transport" for what that means for
remote sessions.
- `server/` — Rust backend (`ai-server`). `main.rs` bootstraps (TLS, the
auth layer, token/QR enrollment, wg0 binding), `routes.rs` has the HTTP
table in its module doc comment, `auth.rs` the bearer-token middleware,
PLAN.md
+13 -4
View File
@@ -713,11 +713,18 @@ host) and **hosts**. The manager runs at most one llama-server per
forwarded port (`ssh -L`) as well as a spawned process. A transport is
therefore "run this" plus "reach this port", and the second operation is
a no-op locally.
- Attachments need no file transfer, contrary to what this section said
- Images need no file transfer, contrary to what this section said
before: `attachment_block` base64s an uploaded image into the
stream-json message itself, and produced images come back the same way
for the translator to write out locally. Nothing has to exist on the
remote filesystem, so there is no `scp` step to get wrong.
- **Any other file is told to the session by path** (2026-09-03: a trace,
a log, a zip -- things a model cannot be shown and the CLI can read).
The upload stays under the session's `attachments/` and the message
ends with `Attached file: /abs/path`. That directory exists only on the
machine running this server, so a file attached to a remote (ssh)
session names a path that is not there. Shipping it is not built; the
one host in use runs its sessions locally. Images are unaffected.
### Usage limits (Claude)
@@ -1094,9 +1101,11 @@ window just fills.
with port forward, attachment shipping. *Host config and remote spawn
done 2026-08-25* (any session of any provider can name a host; the
command is the identical one wrapped in `ssh -T`, with every argument
shell-quoted). Attachment shipping turned out to be unnecessary for the
Claude driver — images ride the stdio JSONL as base64 in both
directions, so nothing needs `scp`. Still outstanding: remote
shell-quoted). Attachment shipping turned out to be unnecessary for
images — they ride the stdio JSONL as base64 in both directions, so
nothing needs `scp` — and became necessary again on 2026-09-03 for
files, which are attached by path (see "Transport" above). Still
outstanding: file shipping for remote sessions, and remote
llama-server with its port forward, which comes with phase 4.
Two things learned doing it: a remote session inherits ssh's non-login
PATH, which is narrower than an interactive shell's (point `command` at
app/androidApp/src/main/AndroidManifest.xml
+10
View File
@@ -61,6 +61,16 @@
<category android:name="android.intent.category.BROWSABLE" />
<data android:scheme="aiapp" android:host="enroll" />
</intent-filter>
<!-- The share sheet: a file, a photo or some text from another app
lands here and is attached to a session (see Share.kt). Any
type, because what a session can be handed is the server's
decision rather than the sheet's. -->
<intent-filter>
<action android:name="android.intent.action.SEND" />
<action android:name="android.intent.action.SEND_MULTIPLE" />
<category android:name="android.intent.category.DEFAULT" />
<data android:mimeType="*/*" />
</intent-filter>
</activity>
<!-- specialUse rather than dataSync, which is the type this looks
app/androidApp/src/main/kotlin/com/example/aiapp/Api.kt
+9 -2
View File
@@ -539,17 +539,24 @@ fun setSessionCwd(settings: ServerSettings, sessionId: String, cwd: String) {
) {}
}
/** Uploads one picked image; the returned id goes into [sendMessage]. */
/**
* Uploads one attachment; the returned id goes into [sendMessage]. [name] is what the server keeps
* a file under and tells the session; for an image it is ignored, since the model is shown the
* picture rather than told its name.
*/
fun uploadAttachment(
settings: ServerSettings,
sessionId: String,
bytes: ByteArray,
mime: String,
name: String,
): String {
val boundary = "----aiapp-${System.currentTimeMillis()}"
// The header is a line: a quote or a line break in the name would end it early.
val safeName = name.replace(Regex("[\"\r\n]"), "_")
val head =
("--$boundary\r\n" +
"Content-Disposition: form-data; name=\"file\"; filename=\"image\"\r\n" +
"Content-Disposition: form-data; name=\"file\"; filename=\"$safeName\"\r\n" +
"Content-Type: $mime\r\n\r\n")
.encodeToByteArray()
val tail = "\r\n--$boundary--\r\n".encodeToByteArray()
app/androidApp/src/main/kotlin/com/example/aiapp/AppRoot.kt
+27 -2
View File
@@ -62,9 +62,16 @@ private data class FailedOpen(val request: SessionOpenRequest, val message: Stri
* re-reading the stored settings -- a plain `remember` would keep serving the pre-enrollment null.
*
* [openRequest] is the session a notification tap asked for, likewise from MainActivity.
*
* [shareRequest] is what another app shared in, likewise. It is held here until a session takes it,
* because the share arrives before anyone has said which session it is for.
*/
@Composable
fun AppRoot(settingsVersion: Int, openRequest: SessionOpenRequest?) {
fun AppRoot(
settingsVersion: Int,
openRequest: SessionOpenRequest?,
shareRequest: ShareRequest? = null,
) {
val context = LocalContext.current
val scope = rememberCoroutineScope()
var settings by remember(settingsVersion) { mutableStateOf(loadServerSettings(context)) }
@@ -75,6 +82,17 @@ fun AppRoot(settingsVersion: Int, openRequest: SessionOpenRequest?) {
// Bumped whenever another screen changes something the list shows, so
// returning to it refetches instead of showing a stale list.
var reloadToken by remember { mutableIntStateOf(0) }
// Cleared by the session screen that attached it, not when a newer request arrives: a share
// must be attached exactly once, and only the screen that did it knows that it has.
var share by remember { mutableStateOf<ShareRequest?>(null) }
LaunchedEffect(shareRequest) {
if (shareRequest != null) {
share = shareRequest
// A session already open takes it. Otherwise the list is where the choice is made,
// whatever screen was showing: Spawn and Settings have nowhere to put a file.
if (screen !is Screen.Session) screen = Screen.Main
}
}
// A standing condition rather than a per-request failure, so it is
// stated once here instead of appended to every error that might be
@@ -162,6 +180,7 @@ fun AppRoot(settingsVersion: Int, openRequest: SessionOpenRequest?) {
MainScreen(
settings = current,
reloadToken = reloadToken,
share = share,
onOpen = { screen = Screen.Session(it) },
onSpawn = { screen = Screen.Spawn },
onImported = { imported ->
@@ -179,7 +198,13 @@ fun AppRoot(settingsVersion: Int, openRequest: SessionOpenRequest?) {
// row key. Only reachable since a notification can move straight from one session to
// another; every other way here passes through [Screen.Main], which disposes it anyway.
key(here.summary.id) {
SessionScreen(settings = current, summary = here.summary, onBack = goToMain)
SessionScreen(
settings = current,
summary = here.summary,
onBack = goToMain,
share = share,
onShareTaken = { share = null },
)
}
is Screen.Spawn ->
Box(Modifier.imePadding()) {
app/androidApp/src/main/kotlin/com/example/aiapp/Attachment.kt
+66
View File
@@ -0,0 +1,66 @@
package com.example.aiapp
import androidx.compose.foundation.background
import androidx.compose.foundation.layout.padding
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.ui.Modifier
import androidx.compose.ui.draw.clip
import androidx.compose.ui.text.font.FontFamily
import androidx.compose.ui.text.style.TextOverflow
import androidx.compose.ui.unit.dp
/**
* Whether [ref] names an image the server stored as one -- `<hex>.<extension>`, with an extension
* from the list it writes -- rather than a file kept under its own name. Mirrors the server's
* `media` table, which is the one other place the list lives.
*/
fun isImageRef(ref: String): Boolean = ref.substringAfterLast('.', "") in IMAGE_EXTENSIONS
private val IMAGE_EXTENSIONS = setOf("png", "jpg", "gif", "webp")
/**
* The name a file was attached under: the ref less the hex the server put before it. The hex has no
* dash in it, so the first one is the boundary however many the name has.
*/
fun attachmentName(ref: String): String = ref.substringAfter('-', ref)
/**
* One attachment on a sent message, drawn as what it is: an image inline, a file as its name. A
* file is not fetched -- there is nothing on this phone to open a trace or a log with -- so the
* name is the whole of it.
*/
@Composable
fun Attachment(
settings: ServerSettings,
sessionId: String,
ref: String,
onOpenImage: (String) -> Unit,
) {
if (isImageRef(ref)) SessionImage(settings, sessionId, ref, onOpenImage)
else
FileName(
attachmentName(ref),
Modifier.clip(MaterialTheme.shapes.extraSmall)
.background(rawSurface)
.padding(horizontal = 8.dp, vertical = 4.dp),
)
}
/**
* A file's name, one line, in the face names are read in. Overlong names lose their middle: a name
* is identified by both ends -- what it is at the front, what kind at the back -- and either
* ellipsis alone takes away one of them.
*/
@Composable
fun FileName(name: String, modifier: Modifier = Modifier) {
Text(
name,
modifier = modifier,
style = MaterialTheme.typography.bodySmall,
fontFamily = FontFamily.Monospace,
maxLines = 1,
overflow = TextOverflow.MiddleEllipsis,
)
}
app/androidApp/src/main/kotlin/com/example/aiapp/Attachments.kt
+63 -4
View File
@@ -1,10 +1,12 @@
package com.example.aiapp
import android.content.ContentResolver
import android.content.Context
import android.graphics.Bitmap
import android.graphics.BitmapFactory
import android.graphics.Matrix
import android.net.Uri
import android.provider.OpenableColumns
import androidx.exifinterface.media.ExifInterface
import java.io.ByteArrayOutputStream
import kotlin.math.max
@@ -31,7 +33,66 @@ suspend fun uploadPickedImage(
maxEdge: Int?,
): String {
val (bytes, mime) = readForUpload(context, uri, maxEdge)
return uploadAttachment(settings, sessionId, bytes, mime)
return uploadAttachment(settings, sessionId, bytes, mime, "image")
}
/**
* Uploads whatever [uri] names, the way its kind needs. An image goes through [uploadPickedImage]
* and is shrunk; anything else goes whole, under the name the other app or the file chooser gave
* it, because the session is told that name rather than shown the bytes.
*/
suspend fun uploadPicked(
context: Context,
settings: ServerSettings,
sessionId: String,
uri: Uri,
maxEdge: Int?,
): String {
val resolver = context.contentResolver
val mime = resolver.getType(uri)
if (mime != null && mime.startsWith("image/")) {
return uploadPickedImage(context, settings, sessionId, uri, maxEdge)
}
val bytes = readAll(resolver, uri)
return uploadAttachment(
settings,
sessionId,
bytes,
mime ?: "application/octet-stream",
displayName(resolver, uri),
)
}
/**
* Everything at [uri], or the failure as the kind the composer reports beside the message.
*
* A share arrives with whatever access the other app granted, and a provider that refuses says so
* with a `SecurityException`; a file gone between the pick and the read is an `IOException`. Both
* are things the reader can act on, so neither is left to end the process.
*/
private fun readAll(resolver: ContentResolver, uri: Uri): ByteArray =
try {
resolver.openInputStream(uri)?.use { it.readBytes() }
?: throw ApiException("couldn't read ${uri.lastPathSegment ?: uri}: nothing there")
} catch (e: SecurityException) {
throw ApiException("couldn't read ${uri.lastPathSegment ?: uri}: no access to it")
} catch (e: java.io.IOException) {
throw ApiException("couldn't read ${uri.lastPathSegment ?: uri}: ${e.message}")
}
/**
* The name a document provider shows for [uri]. The last path segment is the fallback because a
* provider's own id for a file is usually a number, which says nothing to the session.
*/
private fun displayName(resolver: ContentResolver, uri: Uri): String {
resolver.query(uri, arrayOf(OpenableColumns.DISPLAY_NAME), null, null, null)?.use { cursor ->
val column = cursor.getColumnIndex(OpenableColumns.DISPLAY_NAME)
if (column >= 0 && cursor.moveToFirst())
cursor.getString(column)?.let {
return it
}
}
return uri.lastPathSegment ?: "file"
}
/**
@@ -44,9 +105,7 @@ suspend fun uploadPickedImage(
private fun readForUpload(context: Context, uri: Uri, maxEdge: Int?): Pair<ByteArray, String> {
val resolver = context.contentResolver
val mime = resolver.getType(uri) ?: "image/jpeg"
val original =
resolver.openInputStream(uri)?.use { it.readBytes() }
?: throw ApiException("couldn't read the picked image")
val original = readAll(resolver, uri)
if (maxEdge == null) return original to mime
val bounds = BitmapFactory.Options().apply { inJustDecodeBounds = true }
app/androidApp/src/main/kotlin/com/example/aiapp/Events.kt
+6 -5
View File
@@ -30,14 +30,15 @@ sealed class SessionEvent {
*/
val id: String?,
/**
* What was attached to it, by the ref the files route serves.
* What was attached to it, by the ref the files route serves: images, and since 2026-09-03
* any file, told apart by [isImageRef].
*
* On the message rather than beside it: these arrived as separate image events until
* 2026-08-30, which drew somebody's screenshot as a row floating above the bubble that sent
* it, and left this app deciding from adjacency alone which message an image went with --
* something the sender knew and could simply have said.
*/
val images: List<String>,
val attachments: List<String>,
) : SessionEvent()
/**
@@ -50,7 +51,7 @@ sealed class SessionEvent {
* Resolved by the [UserMessage] carrying the same id, exactly as [CommandQueued] is resolved by
* [CommandSent].
*/
data class MessageQueued(val id: String, val text: String, val images: List<String>) :
data class MessageQueued(val id: String, val text: String, val attachments: List<String>) :
SessionEvent()
/**
@@ -197,13 +198,13 @@ fun parseSeqEvent(json: String): SeqEvent {
SessionEvent.UserMessage(
body.getString("text"),
body.optString("id").ifEmpty { null },
body.stringList("images"),
body.stringList("attachments"),
)
"messageQueued" ->
SessionEvent.MessageQueued(
body.getString("id"),
body.getString("text"),
body.stringList("images"),
body.stringList("attachments"),
)
"messageDropped" -> SessionEvent.MessageDropped(body.getString("id"))
"assistantText" -> SessionEvent.AssistantText(body.getString("delta"))
app/androidApp/src/main/kotlin/com/example/aiapp/MainActivity.kt
+17 -6
View File
@@ -37,6 +37,10 @@ class MainActivity : ComponentActivity() {
private var openRequest by mutableStateOf<SessionOpenRequest?>(null)
private var opens = 0
// What another app shared into this one, for the same reason and with the same serial.
private var shareRequest by mutableStateOf<ShareRequest?>(null)
private var shares = 0
// Registered up front since permission launchers must be registered
// before the activity reaches STARTED.
private val requestLocalNetworkPermission =
@@ -139,7 +143,7 @@ class MainActivity : ComponentActivity() {
// scoped to what actually moves.
.navigationBarsPadding()
) {
AppRoot(settingsVersion, openRequest)
AppRoot(settingsVersion, openRequest, shareRequest)
}
}
}
@@ -155,14 +159,21 @@ class MainActivity : ComponentActivity() {
}
/**
* The one place an incoming `aiapp://` URI is sorted into what it means.
* The one place an incoming intent is sorted into what it means.
*
* Two things arrive this way -- an enrollment code and a notification naming a session -- and
* they are told apart by the URI's host rather than by two entry points, so a third kind is a
* branch here rather than another intent to remember to handle.
* Three things arrive this way -- a share from another app, and an `aiapp://` URI that is
* either an enrollment code or a notification naming a session. The URIs are told apart by host
* rather than by two entry points, so a further kind is a branch here rather than another
* intent to remember to handle.
*/
private fun handleIntent(intent: Intent?) {
val uri = intent?.data ?: return
intent ?: return
sharedContent(intent, shares + 1)?.let { shared ->
shares = shared.serial
shareRequest = shared
return
}
val uri = intent.data ?: return
val sessionId = notifiedSessionId(uri)
if (sessionId != null) {
opens++
app/androidApp/src/main/kotlin/com/example/aiapp/MainScreen.kt
+21
View File
@@ -1,6 +1,7 @@
package com.example.aiapp
import androidx.activity.compose.BackHandler
import androidx.compose.foundation.background
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.fillMaxSize
@@ -46,6 +47,8 @@ private enum class MainTab(val label: String) {
fun MainScreen(
settings: ServerSettings,
reloadToken: Int,
/** What another app shared in and no session has taken yet; see [ShareRequest]. */
share: ShareRequest? = null,
onOpen: (SessionSummary) -> Unit,
onSpawn: () -> Unit,
onImported: (SessionSummary) -> Unit,
@@ -106,6 +109,24 @@ fun MainScreen(
GlyphButton(SETTINGS_GLYPH, "Settings", onSettings)
}
}
// What is waiting to be attached, and what to do about it. Said here because the list
// below is where the choice is made, and a share that arrived with nothing on screen
// saying so would read as a tap that did nothing.
share?.let {
Text(
it.summary() + " -- open the session it belongs in.",
style = MaterialTheme.typography.bodyMedium,
color = MaterialTheme.colorScheme.onPrimaryContainer,
modifier =
Modifier.fillMaxWidth()
.padding(horizontal = 16.dp, vertical = 8.dp)
.background(
MaterialTheme.colorScheme.primaryContainer,
MaterialTheme.shapes.small,
)
.padding(12.dp),
)
}
// Primary rather than the plain TabRow, which is deprecated in favour of the two that
// say where they sit: these are the app's top-level destinations.
PrimaryTabRow(selectedTabIndex = tab.ordinal) {
app/androidApp/src/main/kotlin/com/example/aiapp/PendingAttachments.kt
+38 -2
View File
@@ -8,8 +8,12 @@ import androidx.compose.foundation.horizontalScroll
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.height
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.layout.width
import androidx.compose.foundation.layout.widthIn
import androidx.compose.foundation.rememberScrollState
import androidx.compose.foundation.shape.CircleShape
import androidx.compose.foundation.shape.RoundedCornerShape
@@ -34,7 +38,8 @@ import androidx.compose.ui.unit.sp
* anywhere else on the screen.
*
* Scrolls sideways rather than wrapping or shrinking: the row keeps one thumbnail size whatever is
* in it, so four attachments look like four of the same thing rather than four smaller ones.
* in it, so four attachments look like four of the same thing rather than four smaller ones. A file
* is a tile of the same height carrying its name, since a name is all there is to show of it.
*/
@Composable
fun PendingAttachments(
@@ -49,7 +54,10 @@ fun PendingAttachments(
modifier = modifier.horizontalScroll(rememberScrollState()).padding(bottom = 8.dp),
horizontalArrangement = Arrangement.spacedBy(8.dp),
) {
refs.forEach { ref -> PendingThumbnail(settings, sessionId, ref) { onRemove(ref) } }
refs.forEach { ref ->
if (isImageRef(ref)) PendingThumbnail(settings, sessionId, ref) { onRemove(ref) }
else PendingFile(ref) { onRemove(ref) }
}
}
}
@@ -122,4 +130,32 @@ private fun PendingThumbnail(
}
}
/**
* One attached file: its name, tap to take it back off. The same height and removal as a thumbnail,
* so a row of mixed attachments is one row; the cross sits after the name because a tile this wide
* has no corner the eye goes to.
*/
@Composable
private fun PendingFile(ref: String, onRemove: () -> Unit) {
val name = attachmentName(ref)
val shape = RoundedCornerShape(8.dp)
Row(
Modifier.height(THUMBNAIL)
.clip(shape)
.border(1.dp, MaterialTheme.colorScheme.outlineVariant, shape)
.background(MaterialTheme.colorScheme.surfaceVariant)
.clickable(onClick = onRemove)
.semantics { contentDescription = "Attached file $name, tap to remove" }
.padding(horizontal = 8.dp),
verticalAlignment = Alignment.CenterVertically,
) {
FileName(name, Modifier.widthIn(max = FILE_TILE_WIDTH))
Spacer(Modifier.width(6.dp))
Glyph(CLOSE_GLYPH, colour = MaterialTheme.colorScheme.onSurface, size = 12.sp)
}
}
private val THUMBNAIL = 64.dp
/** Wide enough for most names whole; longer ones lose their middle, keeping both ends. */
private val FILE_TILE_WIDTH = 200.dp
app/androidApp/src/main/kotlin/com/example/aiapp/SessionScreen.kt
+70 -29
View File
@@ -2,6 +2,7 @@ package com.example.aiapp
import android.content.Context
import android.content.pm.ApplicationInfo
import android.net.Uri
import android.os.Build
import android.os.SystemClock
import android.util.Log
@@ -228,7 +229,15 @@ private fun Modifier.holdTopEdge(key: Any, held: TopEdgeHold, hold: (Int) -> Uni
// self-correction needs it.
@OptIn(ExperimentalLayoutApi::class)
@Composable
fun SessionScreen(settings: ServerSettings, summary: SessionSummary, onBack: () -> Unit) {
fun SessionScreen(
settings: ServerSettings,
summary: SessionSummary,
onBack: () -> Unit,
/** What another app shared in while this session is the one open; see [ShareRequest]. */
share: ShareRequest? = null,
/** Said once [share] has been attached here, so it is not attached again. */
onShareTaken: () -> Unit = {},
) {
DebugStats.count("session screen recomposed")
val scope = rememberCoroutineScope()
val topEdgeHeld = remember { TopEdgeHold() }
@@ -410,7 +419,7 @@ fun SessionScreen(settings: ServerSettings, summary: SessionSummary, onBack: ()
// Waiting, then read. Matched by id: the same message sent twice is two
// bubbles, and clearing by text would take away whichever matched first.
if (event is SessionEvent.MessageQueued) {
queued = queued + QueuedMessage(event.id, event.text, event.images)
queued = queued + QueuedMessage(event.id, event.text, event.attachments)
}
if (event is SessionEvent.UserMessage) {
queued = queued.filterNot { it.id == event.id }
@@ -1076,25 +1085,18 @@ fun SessionScreen(settings: ServerSettings, summary: SessionSummary, onBack: ()
act { sendMessage(settings, summary.id, text, attachments) }
}
// The system photo picker; the image uploads as soon as it's chosen,
// so Send only has ids to reference.
val pickImage =
rememberLauncherForActivityResult(ActivityResultContracts.PickVisualMedia()) { uri ->
if (uri != null) {
// One path for everything attached, however it arrived: the photo picker, the file chooser
// or another app's share sheet. It uploads as soon as it is chosen, so Send only has ids to
// reference.
fun attach(uri: Uri) {
scope.launch {
try {
val id =
withContext(Dispatchers.IO) {
// Shrunk to what this session's provider takes before it is
// uploaded, so a twelve-megapixel photo does not cross the tunnel
// to be rejected at the far end -- see `uploadPickedImage`.
uploadPickedImage(
context,
settings,
summary.id,
uri,
summary.maxImageEdge,
)
// An image is shrunk to what this session's provider takes before it is
// uploaded, so a twelve-megapixel photo does not cross the tunnel to be
// rejected at the far end; a file goes whole -- see `uploadPicked`.
uploadPicked(context, settings, summary.id, uri, summary.maxImageEdge)
}
pendingAttachments = pendingAttachments + id
actionError = null
@@ -1103,6 +1105,24 @@ fun SessionScreen(settings: ServerSettings, summary: SessionSummary, onBack: ()
}
}
}
val pickImage =
rememberLauncherForActivityResult(ActivityResultContracts.PickVisualMedia()) { uri ->
uri?.let(::attach)
}
val pickFile =
rememberLauncherForActivityResult(ActivityResultContracts.OpenDocument()) { uri ->
uri?.let(::attach)
}
// What another app shared in, attached the moment this screen has it. Taken off the request
// first, so a recomposition or a return to this screen cannot attach it a second time.
LaunchedEffect(share) {
val incoming = share ?: return@LaunchedEffect
onShareTaken()
incoming.uris.forEach(::attach)
incoming.text?.let { shared ->
input = if (input.isBlank()) shared else input + "\n" + shared
saveDraft(context, summary.id, input)
}
}
// One poll for this machine's limits, read by the two things that show them: the bar under
@@ -1333,7 +1353,7 @@ fun SessionScreen(settings: ServerSettings, summary: SessionSummary, onBack: ()
settings = settings,
sessionId = summary.id,
text = waiting.text,
images = waiting.images,
attachments = waiting.attachments,
onOpenImage = ::openImage,
pending = true,
refusal = waiting.refusal,
@@ -1470,7 +1490,7 @@ fun SessionScreen(settings: ServerSettings, summary: SessionSummary, onBack: ()
settings = settings,
sessionId = summary.id,
text = item.text,
images = item.images,
attachments = item.attachments,
onOpenImage = ::openImage,
)
is TranscriptItem.AssistantMsg ->
@@ -1727,17 +1747,38 @@ fun SessionScreen(settings: ServerSettings, summary: SessionSummary, onBack: ()
verticalAlignment = Alignment.CenterVertically,
modifier = Modifier.fillMaxWidth(),
) {
TextButton(
// Photo or file, asked here rather than by two buttons: the row is full,
// and attaching is one action whichever picker answers it.
var attaching by remember { mutableStateOf(false) }
Box {
// Just "+". The count it used to carry was standing in for showing them.
TextButton(onClick = { attaching = true }) { Text("+") }
DropdownMenu(
expanded = attaching,
onDismissRequest = { attaching = false },
// See PickerButton: without this the menu opens a status bar's
// height away from the button in an edge-to-edge activity.
properties = PopupProperties(clippingEnabled = false),
) {
DropdownMenuItem(
text = { Text("Photo") },
onClick = {
attaching = false
pickImage.launch(
PickVisualMediaRequest(
ActivityResultContracts.PickVisualMedia.ImageOnly
)
)
},
)
DropdownMenuItem(
text = { Text("File") },
onClick = {
attaching = false
pickFile.launch(arrayOf("*/*"))
},
)
}
) {
// Just "+" now. The count was standing in for showing them.
Text("+")
}
// The settings share what is left after the actions have
// taken what they need. A Row hands out intrinsic widths in
@@ -1929,9 +1970,9 @@ private fun UserChunkRow(
Text(unit.text, color = MaterialTheme.colorScheme.onPrimaryContainer)
// The same arrangement [UserBubble] gives them: under the words, on the last slice
// because that is the bubble's bottom.
unit.images.forEachIndexed { index, ref ->
unit.attachments.forEachIndexed { index, ref ->
if (index > 0 || unit.text.isNotEmpty()) Spacer(Modifier.height(4.dp))
SessionImage(settings, sessionId, ref, onOpenImage)
Attachment(settings, sessionId, ref, onOpenImage)
}
}
}
@@ -1957,7 +1998,7 @@ private fun UserBubble(
settings: ServerSettings,
sessionId: String,
text: String,
images: List<String> = emptyList(),
attachments: List<String> = emptyList(),
onOpenImage: (String) -> Unit,
pending: Boolean = false,
refusal: String? = null,
@@ -2003,9 +2044,9 @@ private fun UserBubble(
// Under the words: what somebody wrote is what the bubble is, and the picture is
// what they attached to it. It also keeps the first line of every bubble at the
// same place down the transcript, whether or not there is an image in it.
images.forEachIndexed { index, ref ->
attachments.forEachIndexed { index, ref ->
if (index > 0 || text.isNotEmpty()) Spacer(Modifier.height(4.dp))
SessionImage(settings, sessionId, ref, onOpenImage)
Attachment(settings, sessionId, ref, onOpenImage)
}
refusal?.let {
Spacer(Modifier.height(6.dp))
@@ -2030,7 +2071,7 @@ private fun UserBubble(
private data class QueuedMessage(
val id: String,
val text: String,
val images: List<String>,
val attachments: List<String>,
val refusal: String? = null,
)
app/androidApp/src/main/kotlin/com/example/aiapp/Share.kt
+44
View File
@@ -0,0 +1,44 @@
package com.example.aiapp
import android.content.Intent
import android.net.Uri
import androidx.core.content.IntentCompat
/**
* What another app handed this one through the share sheet, waiting to be attached to a session.
*
* Held as the URIs rather than uploaded on arrival, because an upload belongs to a session and the
* share arrives before anyone has said which. [serial] makes two shares of the same thing two
* requests, for the reason [SessionOpenRequest] carries one: equal values would not recompose.
*/
data class ShareRequest(val uris: List<Uri>, val text: String?, val serial: Int)
/** The share in [intent], or null when it is some other intent. */
fun sharedContent(intent: Intent, serial: Int): ShareRequest? {
val uris =
when (intent.action) {
Intent.ACTION_SEND ->
listOfNotNull(
IntentCompat.getParcelableExtra(intent, Intent.EXTRA_STREAM, Uri::class.java)
)
Intent.ACTION_SEND_MULTIPLE ->
IntentCompat.getParcelableArrayListExtra(
intent,
Intent.EXTRA_STREAM,
Uri::class.java,
)
.orEmpty()
else -> return null
}
val text = intent.getStringExtra(Intent.EXTRA_TEXT)?.takeIf { it.isNotBlank() }
if (uris.isEmpty() && text == null) return null
return ShareRequest(uris, text, serial)
}
/** What is waiting, for the banner that says so. */
fun ShareRequest.summary(): String =
when {
uris.size == 1 -> "1 file to attach"
uris.isNotEmpty() -> "${uris.size} files to attach"
else -> "Text to attach"
}
app/androidApp/src/main/kotlin/com/example/aiapp/ToolInput.kt
+9
View File
@@ -160,6 +160,15 @@ private fun highlighted(code: String, language: SyntaxLanguage?): AnnotatedStrin
Highlights.Builder(code = code, language = language, theme = theme)
.build()
.getHighlights()
// highlights 1.1.0's shell lexer answers a quoted glob that looks like a
// comment -- `x '*/a/*'` is the smallest input -- with a span whose end is
// before its start, and AnnotatedString refuses such a range. That crashed the
// app the moment a card holding `-path '*/.git/*'` was opened. Dropped rather
// than clamped: a span the lexer got backwards is not one it knows the colour
// of. Delete when snipme/highlights fixes it.
.filter {
it.location.start in 0..it.location.end && it.location.end <= code.length
}
buildAnnotatedString {
append(code)
marks.forEach { mark ->
app/androidApp/src/main/kotlin/com/example/aiapp/TranscriptItems.kt
+2 -2
View File
@@ -42,7 +42,7 @@ sealed class TranscriptItem {
override val seq: Long,
val text: String,
/** Refs of what was attached, drawn inside the bubble. */
val images: List<String> = emptyList(),
val attachments: List<String> = emptyList(),
) : TranscriptItem()
data class AssistantMsg(
@@ -375,7 +375,7 @@ private fun splitRun(tail: List<TranscriptItem>, behind: String?): List<Transcri
fun foldEvent(items: List<TranscriptItem>, entry: SeqEvent): List<TranscriptItem> =
when (val event = entry.event) {
is SessionEvent.UserMessage ->
items + TranscriptItem.UserMsg(entry.seq, event.text, event.images)
items + TranscriptItem.UserMsg(entry.seq, event.text, event.attachments)
is SessionEvent.AssistantText -> {
// Deltas accumulate into the message they're streaming, which keeps the seq of the
// first of them: a row whose identity changed with every delta would be a new row on
app/androidApp/src/main/kotlin/com/example/aiapp/TranscriptUnits.kt
+2 -2
View File
@@ -129,7 +129,7 @@ sealed class TranscriptUnit {
val first: Boolean,
val last: Boolean,
/** The message's attachments, drawn under the words -- so only the last slice has any. */
val images: List<String>,
val attachments: List<String>,
override val gap: Dp,
) : TranscriptUnit() {
override val key: Any
@@ -211,7 +211,7 @@ fun transcriptUnits(
chunk,
first = at == 0,
last = at == chunks.lastIndex,
images = if (at == chunks.lastIndex) item.images else emptyList(),
attachments = if (at == chunks.lastIndex) item.attachments else emptyList(),
gap = if (at == 0) rowGap else 0.dp,
)
}
app/ui-sandbox.sh
+12 -1
View File
@@ -148,7 +148,18 @@ salvaged=""
if [ -f "$ROOT/config.ron" ]; then
salvaged=$(awk '
/^tokens: \[/ { in_tokens = 1; next }
in_tokens && /^\],/ { exit }
# The server writes the list back compactly, with the last entry
# and the close on one line: " ),],". Reading the close only
# at a line start ran past it into `setups`, and the salvage then
# carried a second copy of that block into the new config.
in_tokens && /\],/ {
sub(/\],.*/, "")
if ($0 ~ /\),/) {
entry = entry $0 "\n"
if (entry !~ h) entries = entries entry
}
exit
}
in_tokens {
entry = entry $0 "\n"
if ($0 ~ /\),/) {
server/src/routes.rs
+36 -15
View File
@@ -30,8 +30,8 @@
//! POST /sessions/{id}/command {text} -- /compact, /clear, /rename x, or the dialect's own
//! (starts the process first if it has exited)
//! POST /sessions/{id}/compact
//! POST /sessions/{id}/attachments multipart image upload -> {id}, referenced by /message
//! GET /sessions/{id}/files/{name} images the session produced or was sent
//! POST /sessions/{id}/attachments multipart upload, image or any file -> {id}, referenced by /message
//! GET /sessions/{id}/files/{name} images the session produced, and what it was sent
//! DELETE /sessions/{id} kill process, delete transcript + files
//! (?deleteForeign=true removes the machine's own copy too)
//! POST /sessions/{id}/notify {notify} -- announce this one or not
@@ -110,7 +110,13 @@ pub fn router(manager: Arc<SessionManager>) -> Router {
.route("/notifications", get(notifications))
.route("/sessions/{id}/compact", post(compact))
.route("/sessions/{id}/command", post(command))
.route("/sessions/{id}/attachments", post(upload_attachment))
.route(
"/sessions/{id}/attachments",
// A trace or a log is bigger than a photo; the cap below is
// for everything else, and the innermost limit is the one
// axum applies.
post(upload_attachment).layer(axum::extract::DefaultBodyLimit::max(ATTACHMENT_LIMIT)),
)
.route("/sessions/{id}/files/{name}", get(serve_file))
// Phone photos overflow axum's 2 MB default body cap.
.layer(axum::extract::DefaultBodyLimit::max(32 * 1024 * 1024))
@@ -1273,8 +1279,14 @@ async fn compact(
Ok(StatusCode::NO_CONTENT)
}
/// Accepts one image (any multipart field) and stores it under the
/// The most one attachment may be. A day of `perfetto` is under a
/// gigabyte; a phone photo is a few megabytes; this is the room between.
const ATTACHMENT_LIMIT: usize = 1024 * 1024 * 1024;
/// Accepts one file (any multipart field) and stores it under the
/// session; the returned id goes into a later `/message`'s attachmentIds.
/// An image is later shown to the model, anything else is named to it by
/// path -- see `ClaudeDriver::send_user_message`.
async fn upload_attachment(
State(manager): State<Arc<SessionManager>>,
UrlPath(id): UrlPath<String>,
@@ -1286,27 +1298,36 @@ async fn upload_attachment(
.await
.map_err(|err| ApiError::BadRequest(format!("bad upload: {err}")))?
.ok_or_else(|| ApiError::BadRequest("no file in the upload".to_string()))?;
let content_type = field.content_type().unwrap_or("image/jpeg").to_string();
let content_type = field
.content_type()
.unwrap_or("application/octet-stream")
.to_string();
let file_name = field.file_name().map(str::to_string);
let bytes = field
.bytes()
.await
.map_err(|err| ApiError::BadRequest(format!("upload read failed: {err}")))?;
let name = session
.save_attachment(&bytes, &content_type)
.save_attachment(&bytes, &content_type, file_name.as_deref())
.map_err(bad_request)?;
Ok(axum::Json(serde_json::json!({ "id": name })))
}
/// Serves a session's stored images -- both `files/` (produced by tools)
/// and `attachments/` (uploaded from the phone), by the id events and
/// uploads reference.
/// Serves a session's stored files -- both `files/` (images produced by
/// tools) and `attachments/` (uploaded from the phone), by the id events
/// and uploads reference.
async fn serve_file(
State(manager): State<Arc<SessionManager>>,
UrlPath((id, name)): UrlPath<(String, String)>,
) -> Result<Response, ApiError> {
// Ids are server-generated hex + extension; anything else (and any
// path separator in particular) is refused, not resolved.
if !name.chars().all(|c| c.is_ascii_alphanumeric() || c == '.') || name.contains("..") {
// Ids are server-generated -- hex and an extension, or hex and a
// cleaned file name (`safe_file_name`); anything else (and any path
// separator in particular) is refused, not resolved.
if !name
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-' || c == '_')
|| name.contains("..")
{
return Err(ApiError::BadRequest("invalid file id".to_string()));
}
let session = lookup(&manager, &id)?;
@@ -1324,9 +1345,9 @@ async fn serve_file(
let bytes = std::fs::read(path)
.with_context(|| format!("read {}", path.display()))
.map_err(ApiError::Internal)?;
// Names are server-generated, so an unrecognized extension can only
// mean a file this server didn't write.
let content_type = crate::media::media_type_for(&name).unwrap_or("image/jpeg");
// Every image this server writes has an extension it knows; the rest
// are files attached by name, served as the bytes they are.
let content_type = crate::media::media_type_for(&name).unwrap_or("application/octet-stream");
Ok(([(axum::http::header::CONTENT_TYPE, content_type)], bytes).into_response())
}
server/src/session/claude.rs
+50 -20
View File
@@ -57,7 +57,7 @@ use serde_json::{Value, json};
use tokio::io::AsyncWriteExt;
use tokio::sync::mpsc;
use super::driver::{Driver, Event, EventSink, ImageRef, SessionStatus, Unqueued};
use super::driver::{AttachmentRef, Driver, Event, EventSink, SessionStatus, Unqueued};
use super::process;
use super::transport::{Launch, Streams, Transport};
use crate::config::{ProviderConfig, SessionConfig};
@@ -133,7 +133,7 @@ struct Queue {
/// Written, not yet announced, oldest first, each with the id of the
/// `MessageQueued` that told the phone it was waiting -- so the
/// announcement can name which bubble it resolves.
awaiting: VecDeque<(String, String, Vec<ImageRef>)>,
awaiting: VecDeque<(String, String, Vec<AttachmentRef>)>,
/// The process is gone, so nothing can be taken up any more.
///
/// Needed because every other way out of a turn is an `Idle` this
@@ -550,20 +550,35 @@ impl ClaudeDriver {
}
impl Driver for ClaudeDriver {
fn send_user_message(&self, text: String, images: Vec<ImageRef>) {
fn send_user_message(&self, text: String, attachments: Vec<AttachmentRef>) {
let mut content = Vec::new();
for id in &images {
match attachment_block(&self.session_dir, id) {
Ok(block) => content.push(block),
Err(err) => {
// An image goes into the message itself; the model looks at it. Any
// other file stays where the upload put it and the message says
// where, because the CLI can read a file by path and a model cannot
// be handed a trace, a log or a zip any other way. Named after the
// text, so the words come first, the way they were typed.
let mut files = Vec::new();
for id in &attachments {
let sent = if crate::media::media_type_for(id).is_some() {
attachment_block(&self.session_dir, id).map(|block| content.push(block))
} else {
attachment_path(&self.session_dir, id).map(|path| files.push(path))
};
if let Err(err) = sent {
let _ = self.sink.send(Event::Error {
message: format!("attachment {id} couldn't be sent: {err:#}"),
});
}
}
let mut body = text.clone();
for path in files {
if !body.is_empty() {
body.push_str("\n\n");
}
if !text.is_empty() {
content.push(json!({"type": "text", "text": text}));
body.push_str(&format!("Attached file: {}", path.display()));
}
if !body.is_empty() {
content.push(json!({"type": "text", "text": body}));
}
let line =
json!({"type": "user", "message": {"role": "user", "content": content}}).to_string();
@@ -591,9 +606,13 @@ impl Driver for ClaudeDriver {
let id = super::random_hex();
queue
.awaiting
.push_back((id.clone(), text.clone(), images.clone()));
.push_back((id.clone(), text.clone(), attachments.clone()));
drop(queue);
let _ = self.sink.send(Event::MessageQueued { id, text, images });
let _ = self.sink.send(Event::MessageQueued {
id,
text,
attachments,
});
self.send_line(line);
return;
}
@@ -605,7 +624,7 @@ impl Driver for ClaudeDriver {
let _ = self.sink.send(Event::MessageTaken {
id: None,
text,
images,
attachments,
});
let _ = self.sink.send(Event::Status {
state: SessionStatus::Running,
@@ -1087,16 +1106,16 @@ fn proves_a_turn(event: &Event) -> bool {
/// [`translate_line`], and the pair is the whole of the rule -- a steer
/// announced anywhere else lands above output that predates it.
fn announce_steers(queue: &Arc<Mutex<Queue>>, sink: &EventSink) -> bool {
let taken: Vec<(String, String, Vec<ImageRef>)> = {
let taken: Vec<(String, String, Vec<AttachmentRef>)> = {
let mut queue = queue.lock().unwrap();
queue.awaiting.drain(..).collect()
};
for (id, text, images) in taken {
for (id, text, attachments) in taken {
if sink
.send(Event::MessageTaken {
id: Some(id),
text,
images,
attachments,
})
.is_err()
{
@@ -1185,17 +1204,28 @@ pub(super) fn write_resume_token(session_dir: &Path, session_id: &str) {
}
}
/// Reads an uploaded attachment into an API image content block.
fn attachment_block(session_dir: &Path, id: &str) -> Result<Value> {
// Ids are server-generated hex (see routes::upload_attachment); the
// check keeps a crafted "id" from naming an arbitrary file.
/// Where an uploaded attachment is, as a path the CLI can be told.
///
/// Absolute, because the CLI's working directory is the session's and the
/// attachments are not in it. Refused rather than resolved when the id is
/// not one this server would have written -- see
/// `SessionManager::save_attachment` -- so a crafted id cannot name a file
/// outside the session.
fn attachment_path(session_dir: &Path, id: &str) -> Result<PathBuf> {
if !id
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-')
.all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-' || c == '_')
|| id.contains("..")
{
anyhow::bail!("invalid attachment id");
}
let path = session_dir.join("attachments").join(id);
std::fs::canonicalize(&path).with_context(|| format!("find {}", path.display()))
}
/// Reads an uploaded image into an API image content block.
fn attachment_block(session_dir: &Path, id: &str) -> Result<Value> {
let path = attachment_path(session_dir, id)?;
let bytes = std::fs::read(&path).with_context(|| format!("read {}", path.display()))?;
use base64::Engine;
Ok(json!({
server/src/session/driver.rs
+18 -7
View File
@@ -15,6 +15,14 @@ use tokio::sync::mpsc;
/// directions use the one id so the transcript renders them identically.
pub type ImageRef = String;
/// The name an upload from the phone is stored and served under: an image
/// is `<hex>.<extension>` and is an [`ImageRef`] like any other; any other
/// file keeps its own name after the hex, `<hex>-<name>`, because the name
/// is what the reader attached and what the session is told. The two are
/// told apart by `crate::media::media_type_for`, which knows every image
/// extension this server writes.
pub type AttachmentRef = String;
/// One choice offered in answer to a [`Event::Question`].
///
/// More than a label because the reader is deciding, not confirming: what
@@ -89,8 +97,11 @@ pub enum Event {
/// sent it -- and left the phone to decide, from nothing but
/// adjacency, which message an image belonged to. Belonging is not
/// something to infer when the sender knew.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
images: Vec<ImageRef>,
///
/// `images` on disk until 2026-09-03, when files joined them;
/// the alias reads the rows written before that.
#[serde(default, alias = "images", skip_serializing_if = "Vec::is_empty")]
attachments: Vec<AttachmentRef>,
},
/// A message accepted from the phone that the session cannot read yet.
///
@@ -111,8 +122,8 @@ pub enum Event {
/// Carried for the same reason [`Event::UserMessage`] carries it,
/// and it matters more here: a waiting message is on screen for as
/// long as the turn runs, so its attachment has nowhere else to be.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
images: Vec<ImageRef>,
#[serde(default, alias = "images", skip_serializing_if = "Vec::is_empty")]
attachments: Vec<AttachmentRef>,
},
/// A message taken out of the queue before the session read it, by
/// somebody tapping the bubble that was waiting for it.
@@ -146,8 +157,8 @@ pub enum Event {
id: Option<String>,
text: String,
/// Carried through onto the `UserMessage` with everything else.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
images: Vec<ImageRef>,
#[serde(default, alias = "images", skip_serializing_if = "Vec::is_empty")]
attachments: Vec<AttachmentRef>,
},
/// Streaming assistant text; the phone renders the concatenation as
/// markdown.
@@ -515,7 +526,7 @@ pub trait Driver: Send + Sync {
/// moment it actually starts reading it: that event is what puts the
/// message in the transcript, so a driver that never sends it drops
/// the message from the conversation entirely.
fn send_user_message(&self, text: String, images: Vec<ImageRef>);
fn send_user_message(&self, text: String, attachments: Vec<AttachmentRef>);
/// Takes back a message that is still waiting, named by the id its
/// [`Event::MessageQueued`] carried.
///
server/src/session/echo.rs
+44 -16
View File
@@ -7,6 +7,8 @@
//! A leading word asks for something more specific:
//!
//! - `/tool [input]` -- a full tool run, start through end.
//! - `/bash [command]` -- a Bash call carrying that command, for what the
//! phone's shell highlighting does to a particular line.
//! - `/tools [n] [gap]` -- n calls back to back, for what a run of them
//! looks like when a screen groups them. `gap` is seconds between one
//! call and the next, default none: it is what makes a run *grow* while
@@ -37,7 +39,7 @@
//! shape a real model's reply arrives in, and the one where the row a
//! reader is anchored to is the row that keeps changing height.
//! - `/mixed N` -- N beats of an interleaved transcript: paragraphs of
//! different lengths, single tool calls, runs of adjacent ones, images
//! different lengths, single tool calls, runs of adjacent ones, attachments
//! and a peer message. Rows of every shape and height the app draws, in
//! one session, which is what a scrolling problem needs in order to be
//! reproduced twice the same way.
@@ -55,7 +57,9 @@ use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use super::driver::{Driver, Event, EventSink, ImageRef, QuestionOption, SessionStatus, Unqueued};
use super::driver::{
AttachmentRef, Driver, Event, EventSink, QuestionOption, SessionStatus, Unqueued,
};
/// Delay between streamed deltas -- long enough that streaming is visibly
/// streaming in the UI, short enough that tests waiting on a full turn
@@ -94,7 +98,7 @@ pub struct EchoDriver {
/// Held messages with the id of the `MessageQueued` each one announced,
/// so the announcement can say which waiting bubble it resolves.
queued: Arc<Mutex<Vec<Held>>>,
/// Where `/mixed` writes the images it references, which is the same
/// Where `/mixed` writes the attachments it references, which is the same
/// directory the files route serves them from.
session_dir: PathBuf,
/// Ids of the questions awaiting an answer, in the order they were
@@ -249,7 +253,7 @@ impl EchoDriver {
/// transcript, and a command is not -- the manager has already
/// recorded that one was sent, and saying so twice drew the same
/// line in both colours.
fn handle(&self, text: String, images: Vec<ImageRef>, announce: bool) {
fn handle(&self, text: String, attachments: Vec<AttachmentRef>, announce: bool) {
let sink = self.sink.clone();
// Mid-turn messages are held rather than answered, the way a real
@@ -265,9 +269,13 @@ impl EchoDriver {
self.queued
.lock()
.unwrap()
.push((id.clone(), text.clone(), images.clone()));
.push((id.clone(), text.clone(), attachments.clone()));
if announce {
self.emit(Event::MessageQueued { id, text, images });
self.emit(Event::MessageQueued {
id,
text,
attachments,
});
}
return;
}
@@ -288,7 +296,7 @@ impl EchoDriver {
self.emit(Event::MessageTaken {
id: None,
text: text.clone(),
images: images.clone(),
attachments: attachments.clone(),
});
}
self.emit(Event::Status {
@@ -319,7 +327,7 @@ impl EchoDriver {
self.emit(Event::MessageTaken {
id: None,
text: text.clone(),
images: images.clone(),
attachments: attachments.clone(),
});
}
self.emit(Event::PeerMessage {
@@ -345,7 +353,7 @@ impl EchoDriver {
self.emit(Event::MessageTaken {
id: None,
text,
images,
attachments,
});
}
self.compact();
@@ -357,7 +365,7 @@ impl EchoDriver {
self.emit(Event::MessageTaken {
id: None,
text,
images,
attachments,
});
}
self.ask_user_question();
@@ -427,6 +435,9 @@ impl EchoDriver {
text.strip_prefix("/tool")
.map(|rest| rest.trim().to_string())
};
let run_bash = text
.strip_prefix("/bash")
.map(|rest| rest.trim().to_string());
// Seconds to stay running before answering, default 30. Clamped
// rather than trusted: this is a test affordance, and a session
// pinned running for an hour by a typo is a worse outcome than a
@@ -469,7 +480,7 @@ impl EchoDriver {
send(Event::MessageTaken {
id: None,
text: text.clone(),
images: images.clone(),
attachments: attachments.clone(),
});
}
send(Event::Status {
@@ -585,6 +596,23 @@ impl EchoDriver {
return;
}
if let Some(command) = run_bash {
let id = format!("b-{}", super::random_hex());
send(Event::ToolStart {
id: id.clone(),
tool: "Bash".to_string(),
input: serde_json::json!({
"command": command,
"description": "Run what /bash was given",
}),
});
tokio::time::sleep(DELTA_DELAY).await;
send(Event::ToolEnd {
id,
output: format!("ran: {command}"),
});
}
if let Some(input) = run_tool {
let id = format!("t-{}", super::random_hex());
send(Event::ToolStart {
@@ -762,7 +790,7 @@ async fn write_beat(sink: &EventSink, session_dir: &Path, beat: usize) {
/// it. All three, because all three are what the `MessageTaken` at the other
/// end owes -- named rather than written out at each of the four places that
/// mention it.
type Held = (String, String, Vec<ImageRef>);
type Held = (String, String, Vec<AttachmentRef>);
/// A markdown table [columns] wide, with cells too long for one line.
///
@@ -832,7 +860,7 @@ fn markdown_table(columns: usize) -> String {
/// of them owes the same answer.
fn finish_turn(sink: &EventSink, queued: &Mutex<Vec<Held>>, busy: &AtomicBool) {
let held = std::mem::take(&mut *queued.lock().unwrap());
for (id, text, images) in held {
for (id, text, attachments) in held {
// Announced before it is answered, in that order: a phone showing
// the message as pending needs the signal that it has been read,
// and the answer is meaningless above a message still drawn as
@@ -840,7 +868,7 @@ fn finish_turn(sink: &EventSink, queued: &Mutex<Vec<Held>>, busy: &AtomicBool) {
let _ = sink.send(Event::MessageTaken {
id: Some(id),
text: text.clone(),
images,
attachments,
});
let _ = sink.send(Event::AssistantText {
delta: format!("\n(taken from the queue) You said: {text}"),
@@ -874,14 +902,14 @@ impl Driver for EchoDriver {
Unqueued::Dropped
}
fn send_user_message(&self, text: String, images: Vec<ImageRef>) {
fn send_user_message(&self, text: String, attachments: Vec<AttachmentRef>) {
// Announced, because this is a message: every driver owes exactly
// one `MessageTaken` per message, and one that quietly vanishes
// from the transcript is the thing echo must not model. A command
// owes none -- the manager has already recorded that it was sent,
// and announcing it again drew the same line twice, once in each
// colour.
self.handle(text, images, true);
self.handle(text, attachments, true);
}
/// Echo's commands *are* its messages -- `/tool`, `/slow`, `/ask` --
server/src/session/import.rs
+1 -1
View File
@@ -628,7 +628,7 @@ fn push_user(events: &mut Vec<Event>, content: &Value, session_dir: &std::path::
events.push(Event::UserMessage {
id: None,
text,
images: Vec::new(),
attachments: Vec::new(),
});
}
}
server/src/session/llama.rs
+16 -16
View File
@@ -33,7 +33,7 @@ use anyhow::{Context, Result, bail};
use serde::{Deserialize, Serialize};
use serde_json::json;
use super::driver::{Driver, Event, EventSink, ImageRef, SessionStatus};
use super::driver::{AttachmentRef, Driver, Event, EventSink, SessionStatus};
use super::process;
use super::transport::{Launch, Streams, Transport};
use crate::config::{ProviderConfig, SessionConfig};
@@ -322,10 +322,10 @@ fn watch(session_dir: PathBuf, sink: EventSink) {
}
impl Driver for LlamaDriver {
fn send_user_message(&self, text: String, images: Vec<ImageRef>) {
if !images.is_empty() {
fn send_user_message(&self, text: String, attachments: Vec<AttachmentRef>) {
if !attachments.is_empty() {
let _ = self.sink.send(Event::Error {
message: "this model can't be sent images".to_string(),
message: "this model can't be sent attachments or files".to_string(),
});
}
let sink = self.sink.clone();
@@ -344,9 +344,9 @@ impl Driver for LlamaDriver {
let _ = sink.send(Event::MessageTaken {
id: None,
text: text.clone(),
// Never any: this driver refuses images above, and saying
// so is what the refusal above is for.
images: Vec::new(),
// Never any: this driver refuses attachments above, and
// saying so is what the refusal above is for.
attachments: Vec::new(),
});
let _ = sink.send(Event::Status {
state: SessionStatus::Running,
@@ -649,7 +649,7 @@ mod tests {
Event::UserMessage {
id: None,
text: "hello".into(),
images: Vec::new(),
attachments: Vec::new(),
},
Event::AssistantText {
delta: "hi ".into(),
@@ -663,7 +663,7 @@ mod tests {
Event::UserMessage {
id: None,
text: "again".into(),
images: Vec::new(),
attachments: Vec::new(),
},
Event::AssistantText {
delta: "yes".into(),
@@ -695,7 +695,7 @@ mod tests {
Event::UserMessage {
id: None,
text: "count".into(),
images: Vec::new(),
attachments: Vec::new(),
},
Event::AssistantText {
delta: "one two".into(),
@@ -721,7 +721,7 @@ mod tests {
Event::UserMessage {
id: None,
text: "hello".into(),
images: Vec::new(),
attachments: Vec::new(),
},
Event::Error {
message: "something went wrong".into(),
@@ -749,7 +749,7 @@ mod tests {
Event::UserMessage {
id: None,
text: "the long expensive conversation".into(),
images: Vec::new(),
attachments: Vec::new(),
},
Event::AssistantText {
delta: "at length".into(),
@@ -758,7 +758,7 @@ mod tests {
Event::UserMessage {
id: None,
text: "a fresh start".into(),
images: Vec::new(),
attachments: Vec::new(),
},
Event::AssistantText {
delta: "cheaply".into(),
@@ -778,19 +778,19 @@ mod tests {
Event::UserMessage {
id: None,
text: "one".into(),
images: Vec::new(),
attachments: Vec::new(),
},
Event::Cleared,
Event::UserMessage {
id: None,
text: "two".into(),
images: Vec::new(),
attachments: Vec::new(),
},
Event::Cleared,
Event::UserMessage {
id: None,
text: "three".into(),
images: Vec::new(),
attachments: Vec::new(),
},
]);
let messages = conversation(&path);
server/src/session/mod.rs
+65 -11
View File
@@ -33,7 +33,7 @@ use crate::config::{
};
use claude::ClaudeDriver;
use driver::{
Driver, Event, EventSink, ImageRef, SessionCommand, SessionStatus, Unqueued, context_after,
AttachmentRef, Driver, Event, EventSink, SessionCommand, SessionStatus, Unqueued, context_after,
};
use echo::EchoDriver;
use llama::LlamaDriver;
@@ -417,14 +417,14 @@ impl LiveSession {
/// The message is deliberately not recorded here. Sent into a running
/// turn it waits, and writing it down on the way past would put it
/// above output that happened before the session ever saw it.
pub fn send_message(&self, text: String, images: Vec<ImageRef>) {
pub fn send_message(&self, text: String, attachments: Vec<AttachmentRef>) {
// The attachments ride *on* the message rather than as `Image`
// events emitted just before it. They used to be the latter, which
// drew a person's screenshot as a row floating above the bubble
// that sent it, and left the phone inferring from adjacency which
// message an image went with -- a thing the sender already knew.
self.ask("take a message", |driver| {
driver.send_user_message(text, images)
driver.send_user_message(text, attachments)
});
}
@@ -488,11 +488,24 @@ impl LiveSession {
/// Stores one uploaded attachment, returning the id `POST /message`
/// references it by. Removed with the session directory on delete --
/// the same path out as everything else in it.
pub fn save_attachment(&self, bytes: &[u8], content_type: &str) -> Result<String> {
// An unrecognized type is almost always a phone photo whose
// content type the picker didn't set; jpg is the useful guess.
let extension = crate::media::extension_for(content_type).unwrap_or("jpg");
let name = format!("{}.{extension}", random_hex());
///
/// An image is named `<hex>.<extension>` and nothing else, since the
/// model is shown the picture rather than told its name. Anything else
/// keeps the name it arrived with after the hex: the session is told
/// the path, and a trace called `trace-komodo-….perfetto-trace` says
/// more to it than `3f9a…` would. The name is cleaned to characters a
/// path and a URL both take unquoted, and the hex keeps two uploads of
/// the same name apart. `AttachmentRef` documents the two shapes.
pub fn save_attachment(
&self,
bytes: &[u8],
content_type: &str,
file_name: Option<&str>,
) -> Result<AttachmentRef> {
let name = match crate::media::extension_for(content_type) {
Some(extension) => format!("{}.{extension}", random_hex()),
None => format!("{}-{}", random_hex(), safe_file_name(file_name)),
};
let dir = self.dir().join("attachments");
wg_app_link::private::create_dir(&dir)?;
std::fs::write(dir.join(&name), bytes)
@@ -1442,14 +1455,19 @@ impl SessionManager {
/// Started before the message rather than after, because starting
/// replaces the driver and the driver that takes the message has to be
/// the one with a process behind it.
pub fn send_message(&self, id: &str, text: String, images: Vec<ImageRef>) -> Result<()> {
pub fn send_message(
&self,
id: &str,
text: String,
attachments: Vec<AttachmentRef>,
) -> Result<()> {
// Only `Exited` starts anything -- see `start_if_exited`. A session
// this cannot say has exited keeps the behaviour it always had: the
// message goes to the driver, which answers for it.
self.start_if_exited(id)?;
self.session(id)
.with_context(|| format!("no session {id}"))?
.send_message(text, images);
.send_message(text, attachments);
Ok(())
}
@@ -1767,6 +1785,34 @@ fn names<'a>(all: impl Iterator<Item = &'a str>) -> String {
/// 8 random bytes, hex -- short enough for a URL, unique enough forever at
/// this scale.
/// A file name reduced to what an attachment id may hold: letters, digits,
/// `.`, `-` and `_`, no run of dots that could read as a parent directory,
/// at most [`FILE_NAME_LIMIT`] characters keeping the tail (the extension
/// is what identifies a file), and `file` when nothing usable is left.
fn safe_file_name(name: Option<&str>) -> String {
let cleaned: String = name
.unwrap_or_default()
.chars()
.map(|c| {
if c.is_ascii_alphanumeric() || c == '.' || c == '-' || c == '_' {
c
} else {
'_'
}
})
.collect();
let cleaned = cleaned.replace("..", "_").trim_matches('.').to_string();
if cleaned.is_empty() {
return "file".to_string();
}
let excess = cleaned.chars().count().saturating_sub(FILE_NAME_LIMIT);
cleaned.chars().skip(excess).collect()
}
/// Longer than any name a person types, shorter than what a filesystem
/// refuses once the hex and a dash are in front of it.
const FILE_NAME_LIMIT: usize = 120;
pub fn random_hex() -> String {
use rand::Rng;
let mut bytes = [0u8; 8];
@@ -2232,7 +2278,15 @@ async fn pump(
// the position is the only way a reader can put it back where it
// happened -- see `Event::PeerMessage::turn_start`.
let event = match event {
Event::MessageTaken { id, text, images } => Event::UserMessage { id, text, images },
Event::MessageTaken {
id,
text,
attachments,
} => Event::UserMessage {
id,
text,
attachments,
},
Event::PeerMessage { from, text, .. } => Event::PeerMessage {
from,
text,
server/src/session/transcript.rs
+1 -1
View File
@@ -636,7 +636,7 @@ mod tests {
Event::UserMessage {
id: None,
text: "hi".into(),
images: Vec::new(),
attachments: Vec::new(),
},
text("hello"),
Event::ToolStart {