Colour code with a scanner of our own instead of the library

dev.snipme:highlights 1.1.0 found comments before it knew the language and
paired /* with */ by ordinal, so `//` in any URL commented out the rest of
its line, every Rust `#[derive(...)]` greyed out as a comment, a `#` inside
a Kotlin string swallowed the line, and `x '*/a/*'` in shell produced a span
whose end preceded its start -- the one that crashed a card holding
`-path '*/.git/*'`. None of that could be post-processed away, because
comments won over strings before the language was known.

Highlighter.kt is one left-to-right scanner: at each position it is in a
line comment, a block comment, a string, or ordinary code, and every span is
emitted by advancing an index, so spans cannot overlap, arrive out of order
or run backwards. Languages.kt is a `Rules` row per language -- comment
tokens, block comment and whether it nests, the string forms, what opens an
attribute, and the keyword set -- so a new language is a table entry. The
keyword lists came from the library's SyntaxTokens.kt (Apache-2.0, noted at
the table) so nothing that is coloured today turns plain, and RON, TOML,
fish and JSON are coloured for the first time.

HighlighterTest.kt is a new JVM unit test source set -- 24 cases, the
library's mistakes kept as regressions, plus a sweep asserting no span
escapes the code for any language on unterminated and empty input.
AGENTS.md's app line now runs :androidApp:testDebugUnitTest.

Measured on the ai-app emulator, debug build, a ~200-line Kotlin fence sent
into a sandbox session:

  before  code highlighted: 1, 101.9ms total, 101.9ms mean, 101.9ms worst
  after   code highlighted: 1,  15.0ms total,  15.0ms mean,  15.0ms worst

and a second fence in the same run took 13.9ms, so that is the steady cost
rather than class loading. stream-bench.sh after the change:

  code highlighted: 1, 12.1ms total, 12.1ms mean, 12.1ms worst
  markdown reparsed while streaming: 1329, 2130.7ms total, 1.6ms mean, 8.7ms worst
  record: one block: 131, 11.4ms total, 0.1ms mean, 0.4ms worst
  draw phase 1.21ms per frame, the transcript 0.23ms of it

transcript-bench.sh after: draw phase 1.10ms per frame, the transcript
0.49ms (place 0.48), worst place 4.3ms -- unchanged within run-to-run noise,
as expected, since the scan happens in `warm` and not while drawing.

Looked at on the emulator: a URL inside a Kotlin string, a Rust attribute
with a lifetime and a raw string, a shell line with globs and `$#`, a RON
fence and a TOML fence all colour correctly; a Bash tool card still colours
its command; a plain Python fence -- which this change had no reason to
touch -- looks as it did; an unknown language stays plain; and a fence is
plain while it streams and colours when it freezes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRANSCRIPT_RENDERING.md

@@ -107,17 +107,18 @@ time went from 2412ms of reparsing to 674ms, and `record: one block` from
 1.8ms worst to 0.7ms.
 
 **Fences are highlighted off the drawing thread, and a fence still being
-written is drawn plain.** `CodeFence.kt` holds `highlight` (shared with a
-tool call's input, so the same code is the same colours wherever it
-appears), the `fenceLanguage` alias table, and `fenceContent`. A word not in
-the table stays plain, because a fence coloured by the wrong language's
-rules looks highlighted and is wrong in a way the reader cannot see.
-Highlighting is warmed and cached exactly as parsing is
-(`ParsedReplies.highlighted`, filled by `warm` from `fences(parse)`), and
-`highlight` takes no colour from the theme, which is what lets it run off
-the drawing thread: a two-hundred-line Kotlin fence costs 174ms to lex, and
-a `remember` inside the fence was charged that again every time the block
-scrolled back into composition. Because the warming has to ask for the same
+written is drawn plain.** `Highlighter.kt` holds `highlight` and the scanner
+behind it (shared with a tool call's input, so the same code is the same
+colours wherever it appears); `CodeFence.kt` holds the `fenceLanguage` alias
+table and `fenceContent`. A word not in the table stays plain, because a
+fence coloured by the wrong language's rules looks highlighted and is wrong
+in a way the reader cannot see. Highlighting is warmed and cached exactly as
+parsing is (`ParsedReplies.highlighted`, filled by `warm` from
+`fences(parse)`), and `highlight` takes no colour from the theme, which is
+what lets it run off the drawing thread: a two-hundred-line Kotlin fence
+costs 15ms to scan on the emulator's debug build -- it cost 102ms through
+the library that used to do this -- and a `remember` inside the fence was
+charged that again every time the block scrolled back into composition. Because the warming has to ask for the same
 string the drawing does, `fenceContent` extracts the code and the language
 word itself -- two extractions would be two keys, and the warmed answer
 would be missed at every fence with nothing saying so. A fence still
@@ -209,25 +210,29 @@ and are the reason several tempting simplifications were rejected.
 - **Compose `DropdownMenu` in an edge-to-edge activity** needs
   `PopupProperties(clippingEnabled = false)` or it opens a status bar's
   height away from its anchor (`~/.claude/TOOLCHAIN.md`).
-- **highlights 1.1.0 finds comments before it knows the language, and pairs
-  `/*` with `*/` by ordinal.** Measured against the library directly with
-  `app/highlights-repro.sh`, which asks it for a piece of code outside the
-  app and prints every span with the text under it -- the mistakes are
-  invisible on a phone, where a line greyed out as a comment looks like a
-  comment. In `MultilineCommentLocator` and `CommentLocator`: it collects every `/*` and
-  every `*/` in the code, zips the two lists by position and never checks
-  that the end follows the start, so `x '*/a/*'` yields `start=6, end=5` --
-  a range `AnnotatedString` rejects, which crashed a card holding
-  `-path '*/.git/*'` until `highlight` started dropping such spans. The same
-  delimiters are used for every language, so a shell glob is read as a
-  comment opener and, worse, `//` in any URL comments out the rest of its
-  line: in `curl https://example.com/x && echo done` the comment runs to the
-  end and takes `echo` with it, and in Kotlin `val url = "https://..."` the
-  string span disappears inside it. Comments are located before strings and
-  win over them. What we can reach is `getCodeStructure()`, which is public,
-  plus a nine-line reconstruction of the library's private
-  `constructHighlights`; the locators themselves are `internal`. 1.1.0 is
-  the newest release, so there is nothing to upgrade to.
+- **The syntax highlighter is ours: `Highlighter.kt` and `Languages.kt`.**
+  One left-to-right scanner with a small state -- in a line comment, in a
+  block comment, in a string, or in ordinary code -- and a `Rules` row per
+  language, so a new language is a table entry rather than code. Every span
+  is emitted by advancing an index, so spans cannot overlap, arrive out of
+  order or run backwards, and an unterminated string or comment simply runs
+  to the end of the code. `HighlighterTest.kt` is the JVM unit test
+  (`./gradlew :androidApp:testDebugUnitTest`); the cases in it are the
+  library's mistakes, kept as regressions.
+  It replaced dev.snipme:highlights 1.1.0 on 2026-09-03, which found
+  comments before it knew the language and paired `/*` with `*/` by
+  ordinal. That library used one set of delimiters for every language, so
+  `//` in any URL commented out the rest of its line (in `curl
+  https://example.com/x && echo done` the comment ran to the end and took
+  `echo` with it, and in Kotlin `val url = "https://..."` the string
+  disappeared inside it), every Rust `#[derive(...)]` greyed out as a
+  comment, a `#` inside a Kotlin string swallowed the line, and `x '*/a/*'`
+  in shell yielded `start=6, end=5` -- a range `AnnotatedString` rejects,
+  which crashed a card holding `-path '*/.git/*'`. Comments were located
+  before strings and won over them, so post-processing could not recover
+  what a wrong comment range had already suppressed. The scanner is also
+  about seven times faster on the same fixture, and it colours RON, TOML,
+  fish and JSON, which the library did not know at all.
 
 ## Rejected, and why
 
@@ -254,19 +259,7 @@ and are the reason several tempting simplifications were rejected.
    against. The restore's one-event-per-request bug was part of what made it
    so visible and has been fixed; whether this survives that fix is the
    first thing to find out.
-2. **Decide what to do about the highlighting bug above.** Two options, both
-   ours: post-process `getCodeStructure()` -- pair `/*` with the next `*/`
-   after it, and only for languages that have them, ignore `#` and `//`
-   where the language does not use them -- then build the annotated string
-   from the corrected structure, which costs no extra lexing but cannot
-   recover keywords and strings the wrong comment range already suppressed;
-   or fork the twenty-file library and fix the locators. Filing it upstream
-   needs Bryan or a token, since there is no `gh` and no GitHub credential
-   in this VM. One-line repro: lexing `x '*/a/*'` as `SyntaxLanguage.SHELL`
-   in highlights 1.1.0 returns a highlight whose `location.end` precedes its
-   `location.start`. Whichever is chosen, `app/highlights-repro.sh` is what
-   checks it.
-3. **Regression runs.** `transcript-bench.sh` and `stream-bench.sh` before
+2. **Regression runs.** `transcript-bench.sh` and `stream-bench.sh` before
    and after any change to the files above, with the report in the commit.
    The numbers to watch are the worst `record: one block`, the reparse mean
    while streaming, and the draw phase's accounting line.