Colour code with a scanner of our own instead of the library

dev.snipme:highlights 1.1.0 found comments before it knew the language and
paired /* with */ by ordinal, so `//` in any URL commented out the rest of
its line, every Rust `#[derive(...)]` greyed out as a comment, a `#` inside
a Kotlin string swallowed the line, and `x '*/a/*'` in shell produced a span
whose end preceded its start -- the one that crashed a card holding
`-path '*/.git/*'`. None of that could be post-processed away, because
comments won over strings before the language was known.

Highlighter.kt is one left-to-right scanner: at each position it is in a
line comment, a block comment, a string, or ordinary code, and every span is
emitted by advancing an index, so spans cannot overlap, arrive out of order
or run backwards. Languages.kt is a `Rules` row per language -- comment
tokens, block comment and whether it nests, the string forms, what opens an
attribute, and the keyword set -- so a new language is a table entry. The
keyword lists came from the library's SyntaxTokens.kt (Apache-2.0, noted at
the table) so nothing that is coloured today turns plain, and RON, TOML,
fish and JSON are coloured for the first time.

HighlighterTest.kt is a new JVM unit test source set -- 24 cases, the
library's mistakes kept as regressions, plus a sweep asserting no span
escapes the code for any language on unterminated and empty input.
AGENTS.md's app line now runs :androidApp:testDebugUnitTest.

Measured on the ai-app emulator, debug build, a ~200-line Kotlin fence sent
into a sandbox session:

  before  code highlighted: 1, 101.9ms total, 101.9ms mean, 101.9ms worst
  after   code highlighted: 1,  15.0ms total,  15.0ms mean,  15.0ms worst

and a second fence in the same run took 13.9ms, so that is the steady cost
rather than class loading. stream-bench.sh after the change:

  code highlighted: 1, 12.1ms total, 12.1ms mean, 12.1ms worst
  markdown reparsed while streaming: 1329, 2130.7ms total, 1.6ms mean, 8.7ms worst
  record: one block: 131, 11.4ms total, 0.1ms mean, 0.4ms worst
  draw phase 1.21ms per frame, the transcript 0.23ms of it

transcript-bench.sh after: draw phase 1.10ms per frame, the transcript
0.49ms (place 0.48), worst place 4.3ms -- unchanged within run-to-run noise,
as expected, since the scan happens in `warm` and not while drawing.

Looked at on the emulator: a URL inside a Kotlin string, a Rust attribute
with a lifetime and a raw string, a shell line with globs and `$#`, a RON
fence and a TOML fence all colour correctly; a Bash tool card still colours
its command; a plain Python fence -- which this change had no reason to
touch -- looks as it did; an unknown language stays plain; and a fence is
plain while it streams and colours when it freezes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
irisandClaude Opus 5 committed 2026-09-03 20:23:59 -04:00
1 parent 3d90e0947c
commit a2b11d516f
13 files changed
+1136 -495

No files matched your search

+14 -5
View File
@@ -27,10 +27,6 @@ zxing-embedded = "4.3.0"
# https://repo1.maven.org/maven2/com/mikepenz/multiplatform-markdown-renderer-m3/maven-metadata.xml
# Latest stable there, checked 2026-08-29.
markdown-renderer = "0.45.0"
# Syntax highlighting for a tool call's input. Same reasoning as the markdown
# renderer: a language's lexical rules are somebody else's specification.
# Latest stable, checked 2026-08-31 against Maven Central.
highlights = "1.1.0"
# The support ExifInterface rather than android.media's, which lint warns off:
# the framework one is missing formats and the fixes for parsing hostile
# images, and images here arrive from outside the phone. Latest stable,
@@ -46,6 +42,12 @@ androidx-lifecycle = "2.11.0"
# itself is Kotlin-org owned and has almost nothing to configure, which is
# the point; this is the Gradle wrapper for it. Checked 2026-08-28.
ktfmt-gradle = "0.27.0"
# The test framework for the app's JVM unit tests (Highlighter.kt's scanner).
# JUnit 6 is the current line -- Jupiter and the Platform ship on one version
# now -- and the tests themselves are written against `kotlin.test`, so the
# framework is a build-file choice rather than something the source names.
# Latest stable, checked 2026-09-03 against Maven Central.
junit = "6.1.3"
# Backports java.time (and more) to API 24, which UsageScreen needs: its
# reset countdown is OffsetDateTime/Duration, both API 26. Checked
# 2026-08-28 against Google Maven.
@@ -64,8 +66,15 @@ desugar-jdk-libs = { module = "com.android.tools:desugar_jdk_libs", version.ref
# The -m3 flavour: it takes its colours and type from the ambient Material 3
# theme, so the app's Catppuccin scheme is what it draws with.
markdown-renderer = { module = "com.mikepenz:multiplatform-markdown-renderer-m3", version.ref = "markdown-renderer" }
highlights = { module = "dev.snipme:highlights", version.ref = "highlights" }
androidx-exifinterface = { module = "androidx.exifinterface:exifinterface", version.ref = "androidx-exifinterface" }
# The framework-neutral assertions the tests are written against. The `-junit5`
# artifact rather than plain `kotlin-test`: AGP 9 compiles Kotlin itself rather
# than through the Kotlin Android plugin, so nothing here resolves the variant
# of `kotlin-test` that carries `kotlin.test.Test` -- naming the artifact is
# what makes the annotation exist.
kotlin-test-junit5 = { module = "org.jetbrains.kotlin:kotlin-test-junit5", version.ref = "kotlin" }
junit-jupiter = { module = "org.junit.jupiter:junit-jupiter", version.ref = "junit" }
junit-platform-launcher = { module = "org.junit.platform:junit-platform-launcher", version.ref = "junit" }
# Declared directly rather than through the plugin's `compose.*` accessors,
# which are deprecated as of CMP 1.11.
compose-runtime = { module = "org.jetbrains.compose.runtime:runtime", version.ref = "compose-multiplatform" }