Merge branch 'main' of git.arirex.me:iris/ai-app
This commit is contained in:
iris committed
commit
a4d512af26
1 file changed
+246
@@ -0,0 +1,246 @@
|
||||
# Replacing the code highlighter with a scanner of our own
|
||||
|
||||
Written 2026-09-03 for whichever session does the work. Bryan decided on
|
||||
this after the assessment below; `TRANSCRIPT_RENDERING.md` records the
|
||||
measurements that led here and stays the record of the transcript work,
|
||||
and `PLAN.md` stays the design source of truth for the app as a whole.
|
||||
Nothing here contradicts either. Work goes on `main`, committed and pushed
|
||||
when it is verified, as usual.
|
||||
|
||||
## Why
|
||||
|
||||
The app colours code in two places -- a fenced block in a reply and the
|
||||
command in a tool call's card -- through one function, `highlight` in
|
||||
`app/androidApp/src/main/kotlin/com/example/aiapp/CodeFence.kt`, backed by
|
||||
the library `dev.snipme:highlights` 1.1.0. That library finds comments
|
||||
before it knows the language, using `//`, `#` and `/* */` for every
|
||||
language, then suppresses keywords and strings under whatever it found;
|
||||
it pairs quotes by ordinal per quote character; and it pairs `/*` with
|
||||
`*/` by ordinal too. Measured against the library directly on 2026-09-03:
|
||||
|
||||
- Any language: `//` inside a URL comments out the rest of the line, and a
|
||||
string holding a URL loses its colour inside the comment.
|
||||
- Rust: every attribute line, `#[derive(Debug)]`, is greyed as a comment.
|
||||
- Kotlin: a `#` inside a string (`"#FF0000"`) comments out the rest of the
|
||||
line; an apostrophe inside a double-quoted string (`"don't"`) opens a
|
||||
bogus single-quoted string that runs to the next apostrophe in the block.
|
||||
- Shell: `x '*/a/*'` produced a span whose end preceded its start, which
|
||||
crashed the app until `highlight` started dropping such spans.
|
||||
|
||||
Upstream merged a one-line fix for the last of these in August 2026 and has
|
||||
not released it; nothing else moved. Post-processing the library's output
|
||||
cannot restore what it already suppressed, and forking it means rewriting
|
||||
its analyzer, which is the part that is wrong. No maintained pure-Kotlin
|
||||
alternative exists on Maven Central; tree-sitter would cost the NDK on
|
||||
every machine that builds the APK and one to five megabytes per language
|
||||
per ABI. So the highlighter becomes ours: one sequential scanner and a
|
||||
table of languages. The rule in `prefer-libraries-for-specs` still holds
|
||||
in general; this is the case it allows, where the only library is a worse
|
||||
hand-rolled scanner than the one we would write.
|
||||
|
||||
Counting fence words across the Claude Code transcripts on the dev VM
|
||||
says which languages matter: shell in its variants 181, kotlin 119, rust
|
||||
98, ron 67, then toml, python, java, js, groovy, json and diff at three to
|
||||
eight each. RON, TOML and fish are not known to the library at all today,
|
||||
so those fences are drawn plain; the scanner colours them from day one.
|
||||
|
||||
## Shape
|
||||
|
||||
Two new files under `app/androidApp/src/main/kotlin/com/example/aiapp/`,
|
||||
and the library removed.
|
||||
|
||||
**`Highlighter.kt`** holds the scanner, the palette type, and `highlight`.
|
||||
The signature stays `fun highlight(code: String, language: Language?):
|
||||
AnnotatedString`, returning plain text for a null language, so the two
|
||||
callers and the cache do not change shape. It is not a composable and
|
||||
takes no colour from the theme, for the reason `CodeFence.kt` already
|
||||
states: `ParsedReplies.warm` runs it off the drawing thread. The palette
|
||||
is a plain `SyntaxPalette` value class of our own -- keyword, string,
|
||||
literal, comment, metadata, punctuation, mark, as `Color` -- returned by
|
||||
`catppuccinSyntax()` in `Theme.kt`, which currently builds the library's
|
||||
`SyntaxTheme` from ARGB ints. The library's separate multiline-comment
|
||||
colour collapses into comment; the palette already gave both `Overlay0`.
|
||||
|
||||
The scanner is one pass over the code with a small state: at each
|
||||
position it is in a line comment, a block comment (with a nesting depth
|
||||
where the language nests them), a string (with its closing delimiter and
|
||||
whether backslash escapes apply), or ordinary code. In ordinary code it
|
||||
recognises, in this order, a comment opener, a string opener, an
|
||||
attribute, a number, a word, and finally single punctuation or mark
|
||||
characters. Every span it emits is produced by advancing an index
|
||||
forward, so spans are non-overlapping and ordered by construction and the
|
||||
reversed-range filter goes with the library. Spans map onto the palette
|
||||
as: keyword, string, literal (numbers), comment, metadata (attributes and
|
||||
annotations), punctuation (`, . : ;`) and mark (`( ) = { } < > - + [ ] |
|
||||
&`). Those two character sets are the library's, kept so nothing the
|
||||
reader is used to changes colour except where it was wrong; punctuation
|
||||
and marks are only coloured in ordinary code, never inside a string or a
|
||||
comment, which is the one visible difference on correct input (a `.` in a
|
||||
URL inside a string used to take the punctuation colour).
|
||||
|
||||
A word is `[A-Za-z_][A-Za-z0-9_]*` and is a keyword if the language's set
|
||||
contains it. A number is a word starting with a digit and continuing
|
||||
through letters, digits, `_` and `.`, which covers `0xFF`, `1_000`,
|
||||
`1u32` and `3.14` without a grammar. A date in TOML comes out as digits
|
||||
with marks between; leave that, it is untidy rather than misleading.
|
||||
|
||||
**`Languages.kt`** holds `enum class Language` and a `Rules` value per
|
||||
member. Keep the vocabulary the code already uses: the enum replaces the
|
||||
library's `SyntaxLanguage` under the name `Language`, and `fenceLanguage`
|
||||
in `CodeFence.kt` keeps its name and its alias table, retargeted. `Rules`
|
||||
is data, not code, and is what makes a new language a table row:
|
||||
|
||||
- `lineComments`: the tokens that open a comment to end of line, and
|
||||
whether they count only at the start of a word. Shell and fish need
|
||||
the word-start rule, since `$#`, `${#x}` and `a#b` are not comments; a
|
||||
`#` counts there when it is preceded by nothing, whitespace, or one of
|
||||
`;`, `|`, `&`, `(`. Python, Ruby, Perl, TOML: `#` anywhere. C-family,
|
||||
Kotlin, Rust, RON, Go, Swift, Dart, JavaScript, TypeScript, Java,
|
||||
C#, PHP: `//`; PHP also `#`.
|
||||
- `blockComment`: the pair, or none, and whether it nests. `/* */` for
|
||||
the C-family and friends; Rust, Kotlin and RON nest; C, Java,
|
||||
JavaScript, Go do not. Python, Ruby, shell, fish, TOML: none.
|
||||
- `quotes`: each string form as opener, closer, and whether backslash
|
||||
escapes the closer. Double quotes with escapes nearly everywhere;
|
||||
single quotes with escapes in the C-family (a character literal reads
|
||||
fine as a string); single quotes with *no* escapes in shell and TOML;
|
||||
fish single quotes escape only `\'` and `\\`; triple quotes in Kotlin
|
||||
(`"""`, no escapes), Python (`"""` and `'''`) and TOML (`"""` and
|
||||
`'''`). Longest opener wins, so `"""` is tried before `"`.
|
||||
- Two string forms need a line of code rather than a row, and belong in
|
||||
the scanner behind a flag on `Rules`. Rust and RON raw strings:
|
||||
optional `b`, then `r`, then `n` hashes, then `"`, closing at `"`
|
||||
followed by `n` hashes, no escapes. Rust lifetimes: a `'` followed by
|
||||
a backslash or by one character and a `'` is a character literal;
|
||||
otherwise it is a lifetime or label and no string starts. Without that
|
||||
rule `'a` opens a string that runs to the next apostrophe.
|
||||
- `attributes`: what opens a metadata span. `@` followed by a word for
|
||||
Kotlin, Java, Dart, Swift, TypeScript, PHP, Python (decorators);
|
||||
`#[` or `#![` through the matching `]` for Rust and RON; a `#` at the
|
||||
start of a line through end of line for C and C++ (preprocessor
|
||||
directives, which the library greyed out as comments); a `[` at the
|
||||
start of a line through `]` for TOML (table headers). Nothing for the
|
||||
rest.
|
||||
- `keywords`: the set. Bring the lists over from the library's
|
||||
`SyntaxTokens.kt` (Apache-2.0; say so in a comment at the table), for
|
||||
every language the alias table names today, so no fence that is
|
||||
coloured now turns plain. Drop entries that are not plain words --
|
||||
Kotlin's `as?`, `!in`, `!is`; Swift's `#if` family; Ruby's `defined?`;
|
||||
CoffeeScript's `=` and `->` -- since the word scanner cannot reach
|
||||
them and the library only matched them by luck. Add rows for RON
|
||||
(`true false Some None inf NaN`), TOML (`true false inf nan`), fish
|
||||
(`and begin break builtin case command continue else end exec for
|
||||
function if in not or return switch while set echo test string math
|
||||
read source`) and JSON (`true false null`). Add `ron`, `toml`, `fish`
|
||||
and `json` to the alias table in `CodeFence.kt`.
|
||||
|
||||
The `streaming` arrangement stays as it is: a fence still being written
|
||||
is drawn plain and coloured when it freezes. That decision was made
|
||||
against a lexer costing 174ms for two hundred lines; a linear scanner may
|
||||
make live colouring affordable, but that is a follow-up to measure with
|
||||
`app/stream-bench.sh`, not something to assume here.
|
||||
|
||||
## Files
|
||||
|
||||
- `Highlighter.kt`, new: scanner, `SyntaxPalette`, `highlight`. The
|
||||
`DebugStats.timed("code highlighted")` wrapper stays, as the number the
|
||||
before-and-after is read from.
|
||||
- `Languages.kt`, new: `Language`, `Rules`, the rows and keyword sets.
|
||||
- `CodeFence.kt`: remove `highlight` and the library imports; retarget
|
||||
`fenceLanguage` and `FENCE_LANGUAGES`; add the four aliases.
|
||||
- `ToolInput.kt`: `SyntaxLanguage` becomes `Language` in `ToolInput` and
|
||||
`SUBJECTS`. `ToolInputView` keeps calling `highlight` uncached.
|
||||
- `Markdown.kt`: the import at line 58 and the signature of
|
||||
`ParsedReplies.highlighted`. The cache key `"$language\n$code"` works
|
||||
unchanged with an enum.
|
||||
- `Theme.kt`: `catppuccinSyntax()` returns `SyntaxPalette` of `Color`s
|
||||
rather than the library's theme of ints.
|
||||
- `app/gradle/libs.versions.toml` and `app/androidApp/build.gradle.kts`:
|
||||
remove `highlights` and its comment.
|
||||
- `app/highlights-repro.sh`: delete. It exists only to interrogate the
|
||||
library, and its cases move into the test below; this is the harness
|
||||
being replaced by a better one, not thrown away.
|
||||
- `app/androidApp/src/test/kotlin/com/example/aiapp/HighlighterTest.kt`,
|
||||
new: see next section.
|
||||
- Docs, in the same commit: in `TRANSCRIPT_RENDERING.md` rewrite the
|
||||
fences paragraph's mention of the library, replace the "highlights
|
||||
1.1.0 finds comments before it knows the language" bullet with a short
|
||||
description of the scanner and where its rows live, and drop item 2 of
|
||||
"What is next". In `AGENTS.md`, add `:androidApp:testDebugUnitTest` to
|
||||
the app line under "Checking your work". Prune rather than append.
|
||||
|
||||
## Tests
|
||||
|
||||
The app has no unit test source set today. The scanner is pure logic
|
||||
with no Android imports, which is exactly the case CODE_RULES names for
|
||||
adding tests, and the cost of the harness is one line,
|
||||
`testImplementation(kotlin("test"))`, run by
|
||||
`./gradlew :androidApp:testDebugUnitTest`. Keep `Highlighter.kt` and
|
||||
`Languages.kt` free of Android and Compose imports other than
|
||||
`AnnotatedString`, `SpanStyle` and `Color` (which are plain JVM classes),
|
||||
so the test runs on the JVM without Robolectric. Better still, have the
|
||||
scanner return a list of (range, kind) spans and let `highlight` be the
|
||||
thin wrapper that turns them into an `AnnotatedString`: the test then
|
||||
asserts on the list, which prints legibly on failure, and the test does
|
||||
not depend on Compose loading at all.
|
||||
|
||||
Cases, each asserting the spans against the code so a failure prints
|
||||
what was coloured:
|
||||
|
||||
- The four inputs from the deleted script: `x '*/a/*'` (shell), the
|
||||
`find . -path '*/.git/*'` line, `curl https://example.com/x && echo
|
||||
done`, and `val url = "https://example.com"\nfun f() = 1`.
|
||||
- `#[derive(Debug)]` then a struct in Rust: metadata, then keywords.
|
||||
- `"#FF0000"` and `"don't"` in Kotlin: two strings, no comment.
|
||||
- `'a` as a Rust lifetime beside `'x'` as a character: one string.
|
||||
- `r#"a "quoted" b"#` in Rust: one string.
|
||||
- Nested `/* a /* b */ c */` in Rust and Kotlin: one comment to the end;
|
||||
the same in C: comment ends at the first `*/`.
|
||||
- `${#x} $# a#b # real` in shell: one comment, starting at the last `#`.
|
||||
- `"""a "b" c"""` in Kotlin: one string.
|
||||
- A TOML table header and a key with a `#` in a quoted value.
|
||||
- A language the alias table does not know: `highlight` returns the
|
||||
plain string.
|
||||
- Every emitted span is inside the code and starts before it ends, over
|
||||
a handful of nasty inputs (unterminated string, unterminated block
|
||||
comment, a lone `'`, empty code). The scanner must never throw; an
|
||||
unterminated thing runs to the end of the code.
|
||||
|
||||
## Verification
|
||||
|
||||
1. From `app/`: `. ./android-env.sh && ./gradlew :androidApp:ktfmtFormat
|
||||
:androidApp:compileDebugKotlin :androidApp:lintDebug
|
||||
:androidApp:testDebugUnitTest`. Lint must stay clean; an unused
|
||||
dependency or import it reports is a finding.
|
||||
2. Timing. `TRANSCRIPT_RENDERING.md` gives the library at 174ms for a
|
||||
two-hundred-line Kotlin fence on the emulator. Regenerate
|
||||
`/tmp/longfence.md` from that description and read `code highlighted`
|
||||
from the render report after `app/transcript-bench.sh`; expect single
|
||||
milliseconds. Run `app/transcript-bench.sh` and `app/stream-bench.sh`
|
||||
before and after, and put the report lines named in that doc's item 3
|
||||
into the commit message.
|
||||
3. Look at it. `app/run-android.sh`, send `/tmp/fixture.md` through
|
||||
`app/ui-sandbox.sh` (both described in `TRANSCRIPT_RENDERING.md`), and
|
||||
screenshot a fence holding a URL in a string, a Rust attribute, a
|
||||
shell line with a glob and a `$#`, a RON fence, and a TOML fence. Check
|
||||
a tool card's Bash command still colours, since that is the other
|
||||
caller. Check a fence in an unknown language is plain, and that a
|
||||
fence still streaming is plain and colours when the next block starts.
|
||||
Per the code rule about trying a fix on the half that cannot fail,
|
||||
look at a fence the change had no reason to touch -- a plain Python
|
||||
function -- and confirm it looks as it did.
|
||||
4. Commit on `main` with `git add -A`, and push.
|
||||
|
||||
## Traps
|
||||
|
||||
- `fenceContent` is shared by the drawing and the warming and must stay
|
||||
the single extraction; nothing here should touch it.
|
||||
- The `Color` constructor in `highlight` currently ORs in an alpha byte
|
||||
because the library's ints had none. With `Color` values in the palette
|
||||
that goes away; do not carry the mask over.
|
||||
- Keyword lists are per language in the library's `DEFAULT` case merged
|
||||
into one; there is no `DEFAULT` here, since a null language is plain.
|
||||
- The two callers pass different things: fences pass what the alias
|
||||
table answered for the info word, tool cards pass `SHELL` for Bash and
|
||||
null for everything else. Both must keep compiling without a cast.
|
||||
- ktfmt will reflow the keyword tables; let it.
|
||||
Reference in new issue
Block a user