iris/ai-app
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Shrink a photo to what the provider takes, and put it in its own bubble

Browse Source 
Sending an image was broken in the way that is hardest to see from the
phone: a camera photo is twelve megapixels and several megabytes, the Claude
API resizes anything past 1568px on its long edge before looking at it and
refuses far larger outright, so the picture was uploaded whole over the
tunnel to be thrown away or rejected at the other end.

Shrunk on the phone, to a limit the server states. Which number it is comes
from the provider's *kind* -- `DriverKind::max_image_edge`, reported on the
session row -- because that is where a provider's requirements are known,
and a phone carrying its own copy of them would be a second place to update
when one changes. `None` where nothing cares, rather than a large number:
"no limit" and "a limit that happens to be big" are different answers and
only one of them stays true. Doing it before the upload rather than after is
the point -- the expensive part on a phone is the tunnel, not the decode --
and an image already inside the limit is uploaded byte for byte rather than
being round-tripped through JPEG for nothing.

EXIF orientation is applied while scaling. The camera writes which way up
the picture is into a tag rather than into the pixels, and re-encoding drops
it, so a portrait photo would have arrived at the model on its side with
nothing anywhere saying so.

**What is attached is now visible before it is sent**, in a row directly
above the box it will be sent from: the count on the "+" button said how
many and never which, so the only way to find out what you had picked was to
send it. It scrolls sideways rather than shrinking, and tapping one takes it
back off -- an image picked by mistake could otherwise only be dealt with by
sending it. The tile is outlined as well as filled, because most of what
gets attached here is a screenshot of a dark app and a cropped one is
near-black: without an edge the only thing on screen saying an image was
attached was the cross drawn on top of nothing.

**And the picture is inside the bubble that sent it.** Attachments used to
be their own `Image` events emitted just before the message, which drew
somebody's screenshot as a row floating above the bubble and left the phone
deciding from adjacency alone which message an image belonged to -- a thing
the sender knew and could simply say. `UserMessage`, `MessageQueued` and
`MessageTaken` carry the refs now, so a waiting message keeps its picture
for as long as the turn runs, and a replay puts it back in the same place.

Verified on a real claude-cli session rather than an echo one, since the
limit only exists for that kind: a 3000x4000 image arrived as 1176x1568
JPEG -- long edge exactly the limit, aspect ratio intact -- and haiku
answered "AI Sessions displays idle Photo", which is what the picture was.
No error, and the transcript records the message with `images` on it.
This commit is contained in:
iris committed 2026-08-30 00:47:16 -04:00
1 parent 5d47a1ec89
commit b0629f77ca
16 files changed
+529 -70
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

app/androidApp/src/main/kotlin/com/example/aiapp/SessionScreen.kt
+78 -25
View File
@@ -86,7 +86,12 @@ sealed class TranscriptItem {
*/
abstract val seq: Long
data class UserMsg(override val seq: Long, val text: String) : TranscriptItem()
data class UserMsg(
override val seq: Long,
val text: String,
/** Refs of what was attached, drawn inside the bubble. */
val images: List<String> = emptyList(),
) : TranscriptItem()
data class AssistantMsg(override val seq: Long, val text: String) : TranscriptItem()
@@ -267,7 +272,8 @@ private fun adoptRun(
fun foldEvent(items: List<TranscriptItem>, entry: SeqEvent): List<TranscriptItem> =
when (val event = entry.event) {
is SessionEvent.UserMessage -> items + TranscriptItem.UserMsg(entry.seq, event.text)
is SessionEvent.UserMessage ->
items + TranscriptItem.UserMsg(entry.seq, event.text, event.images)
is SessionEvent.AssistantText -> {
// Deltas accumulate into the message they're streaming, which keeps the seq of the
// first of them: a row whose identity changed with every delta would be a new row on
@@ -473,7 +479,7 @@ fun SessionScreen(
// them. From the event stream rather than from what this screen sent, so they are still here
// after leaving the session or restarting the app -- and so a message sent from another device
// is drawn waiting on this one too.
var queued by remember { mutableStateOf(listOf<Pair<String, String>>()) }
var queued by remember { mutableStateOf(listOf<QueuedMessage>()) }
// Commands the session has been asked to run and cannot yet, by the id that will resolve
// them. From the server rather than from this screen, so a rename sent from the settings
// screen -- or from another device -- is drawn waiting here too.
@@ -535,10 +541,10 @@ fun SessionScreen(
// Waiting, then read. Matched by id: the same message sent twice is two
// bubbles, and clearing by text would take away whichever matched first.
if (event is SessionEvent.MessageQueued) {
queued = queued + (event.id to event.text)
queued = queued + QueuedMessage(event.id, event.text, event.images)
}
if (event is SessionEvent.UserMessage) {
queued = queued.filterNot { it.first == event.id }
queued = queued.filterNot { it.id == event.id }
}
// Waiting, then gone: a command leaves this list when the session takes it,
// and the row it becomes is added by `foldEvent` in the same pass.
@@ -818,12 +824,16 @@ fun SessionScreen(
try {
val id =
withContext(Dispatchers.IO) {
val bytes =
context.contentResolver.openInputStream(uri)?.use {
it.readBytes()
} ?: throw ApiException("couldn't read the picked image")
val mime = context.contentResolver.getType(uri) ?: "image/jpeg"
uploadAttachment(settings, summary.id, bytes, mime)
// Shrunk to what this session's provider takes before it is
// uploaded, so a twelve-megapixel photo does not cross the tunnel
// to be rejected at the far end -- see `uploadPickedImage`.
uploadPickedImage(
context,
settings,
summary.id,
uri,
summary.maxImageEdge,
)
}
pendingAttachments = pendingAttachments + id
actionError = null
@@ -936,7 +946,15 @@ fun SessionScreen(
waitingCommands.forEach { (_, text) ->
CommandBubble(text, waiting = true)
}
queued.forEach { (_, text) -> UserBubble(text, pending = true) }
queued.forEach { waiting ->
UserBubble(
settings = settings,
sessionId = summary.id,
text = waiting.text,
images = waiting.images,
pending = true,
)
}
}
}
}
@@ -978,7 +996,13 @@ fun SessionScreen(
)
is TranscriptRow.Single ->
when (val item = row.item) {
is TranscriptItem.UserMsg -> UserBubble(item.text)
is TranscriptItem.UserMsg ->
UserBubble(
settings = settings,
sessionId = summary.id,
text = item.text,
images = item.images,
)
is TranscriptItem.AssistantMsg -> AssistantMessage(item.text)
is TranscriptItem.ToolRun ->
ToolCard(
@@ -1099,6 +1123,14 @@ fun SessionScreen(
// put the full width behind three controls, so the thing being
// typed into was the narrowest thing on the row.
Column(Modifier.fillMaxWidth().padding(8.dp)) {
// Directly above the box they will be sent from, so what is attached is visible
// rather than counted: the "+2" on the button below said how many and never which.
PendingAttachments(
settings = settings,
sessionId = summary.id,
refs = pendingAttachments,
onRemove = { pendingAttachments = pendingAttachments - it },
)
OutlinedTextField(
value = input,
onValueChange = {
@@ -1106,9 +1138,9 @@ fun SessionScreen(
saveDraft(context, summary.id, it)
},
modifier = Modifier.fillMaxWidth(),
placeholder = {
Text(if (pendingAttachments.isEmpty()) "Message" else "Message (+image)")
},
// No longer "(+image)": the images are on screen above this, and a placeholder
// saying so said it in words beside the thing itself.
placeholder = { Text("Message") },
maxLines = 4,
)
Row(
@@ -1124,7 +1156,8 @@ fun SessionScreen(
)
}
) {
Text(if (pendingAttachments.isEmpty()) "+" else "+${pendingAttachments.size}")
// Just "+" now. The count was standing in for showing them.
Text("+")
}
// The settings share what is left after the actions have
// taken what they need. A Row hands out intrinsic widths in
@@ -1212,7 +1245,13 @@ private fun sendLabel(running: Boolean) = if (running) "Queue" else "Send"
* bitmap is remembered per ref, so scrolling doesn't refetch.
*/
@Composable
private fun UserBubble(text: String, pending: Boolean = false) {
private fun UserBubble(
settings: ServerSettings,
sessionId: String,
text: String,
images: List<String> = emptyList(),
pending: Boolean = false,
) {
Box(Modifier.fillMaxWidth()) {
Card(
// A message the session has not read yet is drawn quieter than
@@ -1227,17 +1266,31 @@ private fun UserBubble(text: String, pending: Boolean = false) {
),
modifier = Modifier.align(Alignment.CenterEnd).padding(start = 48.dp),
) {
Text(
text,
modifier = Modifier.padding(12.dp),
color =
if (pending) MaterialTheme.colorScheme.onSurfaceVariant
else MaterialTheme.colorScheme.onPrimaryContainer,
)
Column(Modifier.padding(12.dp)) {
// Above the words, in the order they were composed: the picture was attached
// before the sentence about it was typed, and it is what the sentence refers to.
images.forEach { ref ->
SessionImage(settings, sessionId, ref)
Spacer(Modifier.height(4.dp))
}
// A message can be nothing but an attachment, and an empty line under a picture
// is a bubble with a gap in it for a sentence nobody wrote.
if (text.isNotEmpty()) {
Text(
text,
color =
if (pending) MaterialTheme.colorScheme.onSurfaceVariant
else MaterialTheme.colorScheme.onPrimaryContainer,
)
}
}
}
}
}
/** A message the server has accepted and the session has not read yet. */
private data class QueuedMessage(val id: String, val text: String, val images: List<String>)
/**
* Collapsed by default: name plus a spinner while running, expandable to the input and output. The
* spinner-while-unfinished is exactly "ToolStart with no matching ToolEnd yet".