ai-app/HIGHLIGHTER_PLAN.md

Replacing the code highlighter with a scanner of our own

Written 2026-09-03 for whichever session does the work. Bryan decided on this after the assessment below; TRANSCRIPT_RENDERING.md records the measurements that led here and stays the record of the transcript work, and PLAN.md stays the design source of truth for the app as a whole. Nothing here contradicts either. Work goes on main, committed and pushed when it is verified, as usual.

Why

The app colours code in two places -- a fenced block in a reply and the command in a tool call's card -- through one function, highlight in app/androidApp/src/main/kotlin/com/example/aiapp/CodeFence.kt, backed by the library dev.snipme:highlights 1.1.0. That library finds comments before it knows the language, using //, # and /* */ for every language, then suppresses keywords and strings under whatever it found; it pairs quotes by ordinal per quote character; and it pairs /* with */ by ordinal too. Measured against the library directly on 2026-09-03:

• Any language: // inside a URL comments out the rest of the line, and a string holding a URL loses its colour inside the comment.
• Rust: every attribute line, #[derive(Debug)], is greyed as a comment.
• Kotlin: a # inside a string ("#FF0000") comments out the rest of the line; an apostrophe inside a double-quoted string ("don't") opens a bogus single-quoted string that runs to the next apostrophe in the block.
• Shell: x '*/a/*' produced a span whose end preceded its start, which crashed the app until highlight started dropping such spans.

Upstream merged a one-line fix for the last of these in August 2026 and has not released it; nothing else moved. Post-processing the library's output cannot restore what it already suppressed, and forking it means rewriting its analyzer, which is the part that is wrong. No maintained pure-Kotlin alternative exists on Maven Central; tree-sitter would cost the NDK on every machine that builds the APK and one to five megabytes per language per ABI. So the highlighter becomes ours: one sequential scanner and a table of languages. The rule in prefer-libraries-for-specs still holds in general; this is the case it allows, where the only library is a worse hand-rolled scanner than the one we would write.

Counting fence words across the Claude Code transcripts on the dev VM says which languages matter: shell in its variants 181, kotlin 119, rust 98, ron 67, then toml, python, java, js, groovy, json and diff at three to eight each. RON, TOML and fish are not known to the library at all today, so those fences are drawn plain; the scanner colours them from day one.

Shape

Two new files under app/androidApp/src/main/kotlin/com/example/aiapp/, and the library removed.

Highlighter.kt holds the scanner, the palette type, and highlight. The signature stays fun highlight(code: String, language: Language?): AnnotatedString, returning plain text for a null language, so the two callers and the cache do not change shape. It is not a composable and takes no colour from the theme, for the reason CodeFence.kt already states: ParsedReplies.warm runs it off the drawing thread. The palette is a plain SyntaxPalette value class of our own -- keyword, string, literal, comment, metadata, punctuation, mark, as Color -- returned by catppuccinSyntax() in Theme.kt, which currently builds the library's SyntaxTheme from ARGB ints. The library's separate multiline-comment colour collapses into comment; the palette already gave both Overlay0.

The scanner is one pass over the code with a small state: at each position it is in a line comment, a block comment (with a nesting depth where the language nests them), a string (with its closing delimiter and whether backslash escapes apply), or ordinary code. In ordinary code it recognises, in this order, a comment opener, a string opener, an attribute, a number, a word, and finally single punctuation or mark characters. Every span it emits is produced by advancing an index forward, so spans are non-overlapping and ordered by construction and the reversed-range filter goes with the library. Spans map onto the palette as: keyword, string, literal (numbers), comment, metadata (attributes and annotations), punctuation (, . : ;) and mark (( ) = { } < > - + [ ] | &). Those two character sets are the library's, kept so nothing the reader is used to changes colour except where it was wrong; punctuation and marks are only coloured in ordinary code, never inside a string or a comment, which is the one visible difference on correct input (a . in a URL inside a string used to take the punctuation colour).

A word is [A-Za-z_][A-Za-z0-9_]* and is a keyword if the language's set contains it. A number is a word starting with a digit and continuing through letters, digits, _ and ., which covers 0xFF, 1_000, 1u32 and 3.14 without a grammar. A date in TOML comes out as digits with marks between; leave that, it is untidy rather than misleading.

Languages.kt holds enum class Language and a Rules value per member. Keep the vocabulary the code already uses: the enum replaces the library's SyntaxLanguage under the name Language, and fenceLanguage in CodeFence.kt keeps its name and its alias table, retargeted. Rules is data, not code, and is what makes a new language a table row:

• lineComments: the tokens that open a comment to end of line, and whether they count only at the start of a word. Shell and fish need the word-start rule, since $#, ${#x} and a#b are not comments; a # counts there when it is preceded by nothing, whitespace, or one of ;, |, &, (. Python, Ruby, Perl, TOML: # anywhere. C-family, Kotlin, Rust, RON, Go, Swift, Dart, JavaScript, TypeScript, Java, C#, PHP: //; PHP also #.
• blockComment: the pair, or none, and whether it nests. /* */ for the C-family and friends; Rust, Kotlin and RON nest; C, Java, JavaScript, Go do not. Python, Ruby, shell, fish, TOML: none.
• quotes: each string form as opener, closer, and whether backslash escapes the closer. Double quotes with escapes nearly everywhere; single quotes with escapes in the C-family (a character literal reads fine as a string); single quotes with no escapes in shell and TOML; fish single quotes escape only \' and \\; triple quotes in Kotlin (""", no escapes), Python (""" and ''') and TOML (""" and '''). Longest opener wins, so """ is tried before ".
• Two string forms need a line of code rather than a row, and belong in the scanner behind a flag on Rules. Rust and RON raw strings: optional b, then r, then n hashes, then ", closing at " followed by n hashes, no escapes. Rust lifetimes: a ' followed by a backslash or by one character and a ' is a character literal; otherwise it is a lifetime or label and no string starts. Without that rule 'a opens a string that runs to the next apostrophe.
• attributes: what opens a metadata span. @ followed by a word for Kotlin, Java, Dart, Swift, TypeScript, PHP, Python (decorators); #[ or #![ through the matching ] for Rust and RON; a # at the start of a line through end of line for C and C++ (preprocessor directives, which the library greyed out as comments); a [ at the start of a line through ] for TOML (table headers). Nothing for the rest.
• keywords: the set. Bring the lists over from the library's SyntaxTokens.kt (Apache-2.0; say so in a comment at the table), for every language the alias table names today, so no fence that is coloured now turns plain. Drop entries that are not plain words -- Kotlin's as?, !in, !is; Swift's #if family; Ruby's defined?; CoffeeScript's = and -> -- since the word scanner cannot reach them and the library only matched them by luck. Add rows for RON (true false Some None inf NaN), TOML (true false inf nan), fish (and begin break builtin case command continue else end exec for function if in not or return switch while set echo test string math read source) and JSON (true false null). Add ron, toml, fish and json to the alias table in CodeFence.kt.

The streaming arrangement stays as it is: a fence still being written is drawn plain and coloured when it freezes. That decision was made against a lexer costing 174ms for two hundred lines; a linear scanner may make live colouring affordable, but that is a follow-up to measure with app/stream-bench.sh, not something to assume here.

Files

• Highlighter.kt, new: scanner, SyntaxPalette, highlight. The DebugStats.timed("code highlighted") wrapper stays, as the number the before-and-after is read from.
• Languages.kt, new: Language, Rules, the rows and keyword sets.
• CodeFence.kt: remove highlight and the library imports; retarget fenceLanguage and FENCE_LANGUAGES; add the four aliases.
• ToolInput.kt: SyntaxLanguage becomes Language in ToolInput and SUBJECTS. ToolInputView keeps calling highlight uncached.
• Markdown.kt: the import at line 58 and the signature of ParsedReplies.highlighted. The cache key "$language\n$code" works unchanged with an enum.
• Theme.kt: catppuccinSyntax() returns SyntaxPalette of Colors rather than the library's theme of ints.
• app/gradle/libs.versions.toml and app/androidApp/build.gradle.kts: remove highlights and its comment.
• app/highlights-repro.sh: delete. It exists only to interrogate the library, and its cases move into the test below; this is the harness being replaced by a better one, not thrown away.
• app/androidApp/src/test/kotlin/com/example/aiapp/HighlighterTest.kt, new: see next section.
• Docs, in the same commit: in TRANSCRIPT_RENDERING.md rewrite the fences paragraph's mention of the library, replace the "highlights 1.1.0 finds comments before it knows the language" bullet with a short description of the scanner and where its rows live, and drop item 2 of "What is next". In AGENTS.md, add :androidApp:testDebugUnitTest to the app line under "Checking your work". Prune rather than append.

Tests

The app has no unit test source set today. The scanner is pure logic with no Android imports, which is exactly the case CODE_RULES names for adding tests, and the cost of the harness is one line, testImplementation(kotlin("test")), run by ./gradlew :androidApp:testDebugUnitTest. Keep Highlighter.kt and Languages.kt free of Android and Compose imports other than AnnotatedString, SpanStyle and Color (which are plain JVM classes), so the test runs on the JVM without Robolectric. Better still, have the scanner return a list of (range, kind) spans and let highlight be the thin wrapper that turns them into an AnnotatedString: the test then asserts on the list, which prints legibly on failure, and the test does not depend on Compose loading at all.

Cases, each asserting the spans against the code so a failure prints what was coloured:

• The four inputs from the deleted script: x '*/a/*' (shell), the find . -path '*/.git/*' line, curl https://example.com/x && echo done, and val url = "https://example.com"\nfun f() = 1.
• #[derive(Debug)] then a struct in Rust: metadata, then keywords.
• "#FF0000" and "don't" in Kotlin: two strings, no comment.
• 'a as a Rust lifetime beside 'x' as a character: one string.
• r#"a "quoted" b"# in Rust: one string.
• Nested /* a /* b */ c */ in Rust and Kotlin: one comment to the end; the same in C: comment ends at the first */.
• ${#x} $# a#b # real in shell: one comment, starting at the last #.
• """a "b" c""" in Kotlin: one string.
• A TOML table header and a key with a # in a quoted value.
• A language the alias table does not know: highlight returns the plain string.
• Every emitted span is inside the code and starts before it ends, over a handful of nasty inputs (unterminated string, unterminated block comment, a lone ', empty code). The scanner must never throw; an unterminated thing runs to the end of the code.

Verification

1. From app/: . ./android-env.sh && ./gradlew :androidApp:ktfmtFormat :androidApp:compileDebugKotlin :androidApp:lintDebug :androidApp:testDebugUnitTest. Lint must stay clean; an unused dependency or import it reports is a finding.
2. Timing. TRANSCRIPT_RENDERING.md gives the library at 174ms for a two-hundred-line Kotlin fence on the emulator. Regenerate /tmp/longfence.md from that description and read code highlighted from the render report after app/transcript-bench.sh; expect single milliseconds. Run app/transcript-bench.sh and app/stream-bench.sh before and after, and put the report lines named in that doc's item 3 into the commit message.
3. Look at it. app/run-android.sh, send /tmp/fixture.md through app/ui-sandbox.sh (both described in TRANSCRIPT_RENDERING.md), and screenshot a fence holding a URL in a string, a Rust attribute, a shell line with a glob and a $#, a RON fence, and a TOML fence. Check a tool card's Bash command still colours, since that is the other caller. Check a fence in an unknown language is plain, and that a fence still streaming is plain and colours when the next block starts. Per the code rule about trying a fix on the half that cannot fail, look at a fence the change had no reason to touch -- a plain Python function -- and confirm it looks as it did.
4. Commit on main with git add -A, and push.

Traps

• fenceContent is shared by the drawing and the warming and must stay the single extraction; nothing here should touch it.
• The Color constructor in highlight currently ORs in an alpha byte because the library's ints had none. With Color values in the palette that goes away; do not carry the mask over.
• Keyword lists are per language in the library's DEFAULT case merged into one; there is no DEFAULT here, since a null language is plain.
• The two callers pass different things: fences pass what the alias table answered for the info word, tool cards pass SHELL for Bash and null for everything else. Both must keep compiling without a cast.
• ktfmt will reflow the keyword tables; let it.