iris/ai-app
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Colour code with a scanner of our own instead of the library

Browse Source 
dev.snipme:highlights 1.1.0 found comments before it knew the language and
paired /* with */ by ordinal, so `//` in any URL commented out the rest of
its line, every Rust `#[derive(...)]` greyed out as a comment, a `#` inside
a Kotlin string swallowed the line, and `x '*/a/*'` in shell produced a span
whose end preceded its start -- the one that crashed a card holding
`-path '*/.git/*'`. None of that could be post-processed away, because
comments won over strings before the language was known.

Highlighter.kt is one left-to-right scanner: at each position it is in a
line comment, a block comment, a string, or ordinary code, and every span is
emitted by advancing an index, so spans cannot overlap, arrive out of order
or run backwards. Languages.kt is a `Rules` row per language -- comment
tokens, block comment and whether it nests, the string forms, what opens an
attribute, and the keyword set -- so a new language is a table entry. The
keyword lists came from the library's SyntaxTokens.kt (Apache-2.0, noted at
the table) so nothing that is coloured today turns plain, and RON, TOML,
fish and JSON are coloured for the first time.

HighlighterTest.kt is a new JVM unit test source set -- 24 cases, the
library's mistakes kept as regressions, plus a sweep asserting no span
escapes the code for any language on unterminated and empty input.
AGENTS.md's app line now runs :androidApp:testDebugUnitTest.

Measured on the ai-app emulator, debug build, a ~200-line Kotlin fence sent
into a sandbox session:

  before  code highlighted: 1, 101.9ms total, 101.9ms mean, 101.9ms worst
  after   code highlighted: 1,  15.0ms total,  15.0ms mean,  15.0ms worst

and a second fence in the same run took 13.9ms, so that is the steady cost
rather than class loading. stream-bench.sh after the change:

  code highlighted: 1, 12.1ms total, 12.1ms mean, 12.1ms worst
  markdown reparsed while streaming: 1329, 2130.7ms total, 1.6ms mean, 8.7ms worst
  record: one block: 131, 11.4ms total, 0.1ms mean, 0.4ms worst
  draw phase 1.21ms per frame, the transcript 0.23ms of it

transcript-bench.sh after: draw phase 1.10ms per frame, the transcript
0.49ms (place 0.48), worst place 4.3ms -- unchanged within run-to-run noise,
as expected, since the scan happens in `warm` and not while drawing.

Looked at on the emulator: a URL inside a Kotlin string, a Rust attribute
with a lifetime and a raw string, a shell line with globs and `$#`, a RON
fence and a TOML fence all colour correctly; a Bash tool card still colours
its command; a plain Python fence -- which this change had no reason to
touch -- looks as it did; an unknown language stays plain; and a fence is
plain while it streams and colours when it freezes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
irisandClaude Opus 5 committed 2026-09-03 20:23:59 -04:00
1 parent 3d90e0947c
commit a2b11d516f
13 files changed
+1136 -495
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+5 -2
View File
@@ -220,8 +220,11 @@ first if a remote spawn ever mangles an argument.
warning-clean and rustfmt-clean at the defaults — there is no
`rustfmt.toml` and there should not be one.
- App: from `app/`, `. ./android-env.sh && ./gradlew :androidApp:ktfmtFormat
:androidApp:compileDebugKotlin :androidApp:lintDebug` — format, typecheck
and lint, the app-side equivalent of the line above. Then `./build-apk.sh`
:androidApp:compileDebugKotlin :androidApp:lintDebug
:androidApp:testDebugUnitTest` — format, typecheck, lint and test, the
app-side equivalent of the line above. The unit tests are JVM-only and
cover the syntax highlighter's scanner, which is the app's one piece of
pure logic with no Android in it. Then `./build-apk.sh`
to produce the APK to install on a phone (through Dev Updater), or
`./run-android.sh` to build, install, and launch on the emulator.
**The phone gets the release build**, signed with a key the script
HIGHLIGHTER_PLAN.md
-246
View File
@@ -1,246 +0,0 @@
# Replacing the code highlighter with a scanner of our own
Written 2026-09-03 for whichever session does the work. Bryan decided on
this after the assessment below; `TRANSCRIPT_RENDERING.md` records the
measurements that led here and stays the record of the transcript work,
and `PLAN.md` stays the design source of truth for the app as a whole.
Nothing here contradicts either. Work goes on `main`, committed and pushed
when it is verified, as usual.
## Why
The app colours code in two places -- a fenced block in a reply and the
command in a tool call's card -- through one function, `highlight` in
`app/androidApp/src/main/kotlin/com/example/aiapp/CodeFence.kt`, backed by
the library `dev.snipme:highlights` 1.1.0. That library finds comments
before it knows the language, using `//`, `#` and `/* */` for every
language, then suppresses keywords and strings under whatever it found;
it pairs quotes by ordinal per quote character; and it pairs `/*` with
`*/` by ordinal too. Measured against the library directly on 2026-09-03:
- Any language: `//` inside a URL comments out the rest of the line, and a
string holding a URL loses its colour inside the comment.
- Rust: every attribute line, `#[derive(Debug)]`, is greyed as a comment.
- Kotlin: a `#` inside a string (`"#FF0000"`) comments out the rest of the
line; an apostrophe inside a double-quoted string (`"don't"`) opens a
bogus single-quoted string that runs to the next apostrophe in the block.
- Shell: `x '*/a/*'` produced a span whose end preceded its start, which
crashed the app until `highlight` started dropping such spans.
Upstream merged a one-line fix for the last of these in August 2026 and has
not released it; nothing else moved. Post-processing the library's output
cannot restore what it already suppressed, and forking it means rewriting
its analyzer, which is the part that is wrong. No maintained pure-Kotlin
alternative exists on Maven Central; tree-sitter would cost the NDK on
every machine that builds the APK and one to five megabytes per language
per ABI. So the highlighter becomes ours: one sequential scanner and a
table of languages. The rule in `prefer-libraries-for-specs` still holds
in general; this is the case it allows, where the only library is a worse
hand-rolled scanner than the one we would write.
Counting fence words across the Claude Code transcripts on the dev VM
says which languages matter: shell in its variants 181, kotlin 119, rust
98, ron 67, then toml, python, java, js, groovy, json and diff at three to
eight each. RON, TOML and fish are not known to the library at all today,
so those fences are drawn plain; the scanner colours them from day one.
## Shape
Two new files under `app/androidApp/src/main/kotlin/com/example/aiapp/`,
and the library removed.
**`Highlighter.kt`** holds the scanner, the palette type, and `highlight`.
The signature stays `fun highlight(code: String, language: Language?):
AnnotatedString`, returning plain text for a null language, so the two
callers and the cache do not change shape. It is not a composable and
takes no colour from the theme, for the reason `CodeFence.kt` already
states: `ParsedReplies.warm` runs it off the drawing thread. The palette
is a plain `SyntaxPalette` value class of our own -- keyword, string,
literal, comment, metadata, punctuation, mark, as `Color` -- returned by
`catppuccinSyntax()` in `Theme.kt`, which currently builds the library's
`SyntaxTheme` from ARGB ints. The library's separate multiline-comment
colour collapses into comment; the palette already gave both `Overlay0`.
The scanner is one pass over the code with a small state: at each
position it is in a line comment, a block comment (with a nesting depth
where the language nests them), a string (with its closing delimiter and
whether backslash escapes apply), or ordinary code. In ordinary code it
recognises, in this order, a comment opener, a string opener, an
attribute, a number, a word, and finally single punctuation or mark
characters. Every span it emits is produced by advancing an index
forward, so spans are non-overlapping and ordered by construction and the
reversed-range filter goes with the library. Spans map onto the palette
as: keyword, string, literal (numbers), comment, metadata (attributes and
annotations), punctuation (`, . : ;`) and mark (`( ) = { } < > - + [ ] |
&`). Those two character sets are the library's, kept so nothing the
reader is used to changes colour except where it was wrong; punctuation
and marks are only coloured in ordinary code, never inside a string or a
comment, which is the one visible difference on correct input (a `.` in a
URL inside a string used to take the punctuation colour).
A word is `[A-Za-z_][A-Za-z0-9_]*` and is a keyword if the language's set
contains it. A number is a word starting with a digit and continuing
through letters, digits, `_` and `.`, which covers `0xFF`, `1_000`,
`1u32` and `3.14` without a grammar. A date in TOML comes out as digits
with marks between; leave that, it is untidy rather than misleading.
**`Languages.kt`** holds `enum class Language` and a `Rules` value per
member. Keep the vocabulary the code already uses: the enum replaces the
library's `SyntaxLanguage` under the name `Language`, and `fenceLanguage`
in `CodeFence.kt` keeps its name and its alias table, retargeted. `Rules`
is data, not code, and is what makes a new language a table row:
- `lineComments`: the tokens that open a comment to end of line, and
whether they count only at the start of a word. Shell and fish need
the word-start rule, since `$#`, `${#x}` and `a#b` are not comments; a
`#` counts there when it is preceded by nothing, whitespace, or one of
`;`, `|`, `&`, `(`. Python, Ruby, Perl, TOML: `#` anywhere. C-family,
Kotlin, Rust, RON, Go, Swift, Dart, JavaScript, TypeScript, Java,
C#, PHP: `//`; PHP also `#`.
- `blockComment`: the pair, or none, and whether it nests. `/* */` for
the C-family and friends; Rust, Kotlin and RON nest; C, Java,
JavaScript, Go do not. Python, Ruby, shell, fish, TOML: none.
- `quotes`: each string form as opener, closer, and whether backslash
escapes the closer. Double quotes with escapes nearly everywhere;
single quotes with escapes in the C-family (a character literal reads
fine as a string); single quotes with *no* escapes in shell and TOML;
fish single quotes escape only `\'` and `\\`; triple quotes in Kotlin
(`"""`, no escapes), Python (`"""` and `'''`) and TOML (`"""` and
`'''`). Longest opener wins, so `"""` is tried before `"`.
- Two string forms need a line of code rather than a row, and belong in
the scanner behind a flag on `Rules`. Rust and RON raw strings:
optional `b`, then `r`, then `n` hashes, then `"`, closing at `"`
followed by `n` hashes, no escapes. Rust lifetimes: a `'` followed by
a backslash or by one character and a `'` is a character literal;
otherwise it is a lifetime or label and no string starts. Without that
rule `'a` opens a string that runs to the next apostrophe.
- `attributes`: what opens a metadata span. `@` followed by a word for
Kotlin, Java, Dart, Swift, TypeScript, PHP, Python (decorators);
`#[` or `#![` through the matching `]` for Rust and RON; a `#` at the
start of a line through end of line for C and C++ (preprocessor
directives, which the library greyed out as comments); a `[` at the
start of a line through `]` for TOML (table headers). Nothing for the
rest.
- `keywords`: the set. Bring the lists over from the library's
`SyntaxTokens.kt` (Apache-2.0; say so in a comment at the table), for
every language the alias table names today, so no fence that is
coloured now turns plain. Drop entries that are not plain words --
Kotlin's `as?`, `!in`, `!is`; Swift's `#if` family; Ruby's `defined?`;
CoffeeScript's `=` and `->` -- since the word scanner cannot reach
them and the library only matched them by luck. Add rows for RON
(`true false Some None inf NaN`), TOML (`true false inf nan`), fish
(`and begin break builtin case command continue else end exec for
function if in not or return switch while set echo test string math
read source`) and JSON (`true false null`). Add `ron`, `toml`, `fish`
and `json` to the alias table in `CodeFence.kt`.
The `streaming` arrangement stays as it is: a fence still being written
is drawn plain and coloured when it freezes. That decision was made
against a lexer costing 174ms for two hundred lines; a linear scanner may
make live colouring affordable, but that is a follow-up to measure with
`app/stream-bench.sh`, not something to assume here.
## Files
- `Highlighter.kt`, new: scanner, `SyntaxPalette`, `highlight`. The
`DebugStats.timed("code highlighted")` wrapper stays, as the number the
before-and-after is read from.
- `Languages.kt`, new: `Language`, `Rules`, the rows and keyword sets.
- `CodeFence.kt`: remove `highlight` and the library imports; retarget
`fenceLanguage` and `FENCE_LANGUAGES`; add the four aliases.
- `ToolInput.kt`: `SyntaxLanguage` becomes `Language` in `ToolInput` and
`SUBJECTS`. `ToolInputView` keeps calling `highlight` uncached.
- `Markdown.kt`: the import at line 58 and the signature of
`ParsedReplies.highlighted`. The cache key `"$language\n$code"` works
unchanged with an enum.
- `Theme.kt`: `catppuccinSyntax()` returns `SyntaxPalette` of `Color`s
rather than the library's theme of ints.
- `app/gradle/libs.versions.toml` and `app/androidApp/build.gradle.kts`:
remove `highlights` and its comment.
- `app/highlights-repro.sh`: delete. It exists only to interrogate the
library, and its cases move into the test below; this is the harness
being replaced by a better one, not thrown away.
- `app/androidApp/src/test/kotlin/com/example/aiapp/HighlighterTest.kt`,
new: see next section.
- Docs, in the same commit: in `TRANSCRIPT_RENDERING.md` rewrite the
fences paragraph's mention of the library, replace the "highlights
1.1.0 finds comments before it knows the language" bullet with a short
description of the scanner and where its rows live, and drop item 2 of
"What is next". In `AGENTS.md`, add `:androidApp:testDebugUnitTest` to
the app line under "Checking your work". Prune rather than append.
## Tests
The app has no unit test source set today. The scanner is pure logic
with no Android imports, which is exactly the case CODE_RULES names for
adding tests, and the cost of the harness is one line,
`testImplementation(kotlin("test"))`, run by
`./gradlew :androidApp:testDebugUnitTest`. Keep `Highlighter.kt` and
`Languages.kt` free of Android and Compose imports other than
`AnnotatedString`, `SpanStyle` and `Color` (which are plain JVM classes),
so the test runs on the JVM without Robolectric. Better still, have the
scanner return a list of (range, kind) spans and let `highlight` be the
thin wrapper that turns them into an `AnnotatedString`: the test then
asserts on the list, which prints legibly on failure, and the test does
not depend on Compose loading at all.
Cases, each asserting the spans against the code so a failure prints
what was coloured:
- The four inputs from the deleted script: `x '*/a/*'` (shell), the
`find . -path '*/.git/*'` line, `curl https://example.com/x && echo
done`, and `val url = "https://example.com"\nfun f() = 1`.
- `#[derive(Debug)]` then a struct in Rust: metadata, then keywords.
- `"#FF0000"` and `"don't"` in Kotlin: two strings, no comment.
- `'a` as a Rust lifetime beside `'x'` as a character: one string.
- `r#"a "quoted" b"#` in Rust: one string.
- Nested `/* a /* b */ c */` in Rust and Kotlin: one comment to the end;
the same in C: comment ends at the first `*/`.
- `${#x} $# a#b # real` in shell: one comment, starting at the last `#`.
- `"""a "b" c"""` in Kotlin: one string.
- A TOML table header and a key with a `#` in a quoted value.
- A language the alias table does not know: `highlight` returns the
plain string.
- Every emitted span is inside the code and starts before it ends, over
a handful of nasty inputs (unterminated string, unterminated block
comment, a lone `'`, empty code). The scanner must never throw; an
unterminated thing runs to the end of the code.
## Verification
1. From `app/`: `. ./android-env.sh && ./gradlew :androidApp:ktfmtFormat
:androidApp:compileDebugKotlin :androidApp:lintDebug
:androidApp:testDebugUnitTest`. Lint must stay clean; an unused
dependency or import it reports is a finding.
2. Timing. `TRANSCRIPT_RENDERING.md` gives the library at 174ms for a
two-hundred-line Kotlin fence on the emulator. Regenerate
`/tmp/longfence.md` from that description and read `code highlighted`
from the render report after `app/transcript-bench.sh`; expect single
milliseconds. Run `app/transcript-bench.sh` and `app/stream-bench.sh`
before and after, and put the report lines named in that doc's item 3
into the commit message.
3. Look at it. `app/run-android.sh`, send `/tmp/fixture.md` through
`app/ui-sandbox.sh` (both described in `TRANSCRIPT_RENDERING.md`), and
screenshot a fence holding a URL in a string, a Rust attribute, a
shell line with a glob and a `$#`, a RON fence, and a TOML fence. Check
a tool card's Bash command still colours, since that is the other
caller. Check a fence in an unknown language is plain, and that a
fence still streaming is plain and colours when the next block starts.
Per the code rule about trying a fix on the half that cannot fail,
look at a fence the change had no reason to touch -- a plain Python
function -- and confirm it looks as it did.
4. Commit on `main` with `git add -A`, and push.
## Traps
- `fenceContent` is shared by the drawing and the warming and must stay
the single extraction; nothing here should touch it.
- The `Color` constructor in `highlight` currently ORs in an alpha byte
because the library's ints had none. With `Color` values in the palette
that goes away; do not carry the mask over.
- Keyword lists are per language in the library's `DEFAULT` case merged
into one; there is no `DEFAULT` here, since a null language is plain.
- The two callers pass different things: fences pass what the alias
table answered for the info word, tool cards pass `SHELL` for Bash and
null for everything else. Both must keep compiling without a cast.
- ktfmt will reflow the keyword tables; let it.
TRANSCRIPT_RENDERING.md
+36 -43
View File
@@ -107,17 +107,18 @@ time went from 2412ms of reparsing to 674ms, and `record: one block` from
1.8ms worst to 0.7ms.
**Fences are highlighted off the drawing thread, and a fence still being
written is drawn plain.** `CodeFence.kt` holds `highlight` (shared with a
tool call's input, so the same code is the same colours wherever it
appears), the `fenceLanguage` alias table, and `fenceContent`. A word not in
the table stays plain, because a fence coloured by the wrong language's
rules looks highlighted and is wrong in a way the reader cannot see.
Highlighting is warmed and cached exactly as parsing is
(`ParsedReplies.highlighted`, filled by `warm` from `fences(parse)`), and
`highlight` takes no colour from the theme, which is what lets it run off
the drawing thread: a two-hundred-line Kotlin fence costs 174ms to lex, and
a `remember` inside the fence was charged that again every time the block
scrolled back into composition. Because the warming has to ask for the same
written is drawn plain.** `Highlighter.kt` holds `highlight` and the scanner
behind it (shared with a tool call's input, so the same code is the same
colours wherever it appears); `CodeFence.kt` holds the `fenceLanguage` alias
table and `fenceContent`. A word not in the table stays plain, because a
fence coloured by the wrong language's rules looks highlighted and is wrong
in a way the reader cannot see. Highlighting is warmed and cached exactly as
parsing is (`ParsedReplies.highlighted`, filled by `warm` from
`fences(parse)`), and `highlight` takes no colour from the theme, which is
what lets it run off the drawing thread: a two-hundred-line Kotlin fence
costs 15ms to scan on the emulator's debug build -- it cost 102ms through
the library that used to do this -- and a `remember` inside the fence was
charged that again every time the block scrolled back into composition. Because the warming has to ask for the same
string the drawing does, `fenceContent` extracts the code and the language
word itself -- two extractions would be two keys, and the warmed answer
would be missed at every fence with nothing saying so. A fence still
@@ -209,25 +210,29 @@ and are the reason several tempting simplifications were rejected.
- **Compose `DropdownMenu` in an edge-to-edge activity** needs
`PopupProperties(clippingEnabled = false)` or it opens a status bar's
height away from its anchor (`~/.claude/TOOLCHAIN.md`).
- **highlights 1.1.0 finds comments before it knows the language, and pairs
`/*` with `*/` by ordinal.** Measured against the library directly with
`app/highlights-repro.sh`, which asks it for a piece of code outside the
app and prints every span with the text under it -- the mistakes are
invisible on a phone, where a line greyed out as a comment looks like a
comment. In `MultilineCommentLocator` and `CommentLocator`: it collects every `/*` and
every `*/` in the code, zips the two lists by position and never checks
that the end follows the start, so `x '*/a/*'` yields `start=6, end=5` --
a range `AnnotatedString` rejects, which crashed a card holding
`-path '*/.git/*'` until `highlight` started dropping such spans. The same
delimiters are used for every language, so a shell glob is read as a
comment opener and, worse, `//` in any URL comments out the rest of its
line: in `curl https://example.com/x && echo done` the comment runs to the
end and takes `echo` with it, and in Kotlin `val url = "https://..."` the
string span disappears inside it. Comments are located before strings and
win over them. What we can reach is `getCodeStructure()`, which is public,
plus a nine-line reconstruction of the library's private
`constructHighlights`; the locators themselves are `internal`. 1.1.0 is
the newest release, so there is nothing to upgrade to.
- **The syntax highlighter is ours: `Highlighter.kt` and `Languages.kt`.**
One left-to-right scanner with a small state -- in a line comment, in a
block comment, in a string, or in ordinary code -- and a `Rules` row per
language, so a new language is a table entry rather than code. Every span
is emitted by advancing an index, so spans cannot overlap, arrive out of
order or run backwards, and an unterminated string or comment simply runs
to the end of the code. `HighlighterTest.kt` is the JVM unit test
(`./gradlew :androidApp:testDebugUnitTest`); the cases in it are the
library's mistakes, kept as regressions.
It replaced dev.snipme:highlights 1.1.0 on 2026-09-03, which found
comments before it knew the language and paired `/*` with `*/` by
ordinal. That library used one set of delimiters for every language, so
`//` in any URL commented out the rest of its line (in `curl
https://example.com/x && echo done` the comment ran to the end and took
`echo` with it, and in Kotlin `val url = "https://..."` the string
disappeared inside it), every Rust `#[derive(...)]` greyed out as a
comment, a `#` inside a Kotlin string swallowed the line, and `x '*/a/*'`
in shell yielded `start=6, end=5` -- a range `AnnotatedString` rejects,
which crashed a card holding `-path '*/.git/*'`. Comments were located
before strings and won over them, so post-processing could not recover
what a wrong comment range had already suppressed. The scanner is also
about seven times faster on the same fixture, and it colours RON, TOML,
fish and JSON, which the library did not know at all.
## Rejected, and why
@@ -254,19 +259,7 @@ and are the reason several tempting simplifications were rejected.
against. The restore's one-event-per-request bug was part of what made it
so visible and has been fixed; whether this survives that fix is the
first thing to find out.
2. **Decide what to do about the highlighting bug above.** Two options, both
ours: post-process `getCodeStructure()` -- pair `/*` with the next `*/`
after it, and only for languages that have them, ignore `#` and `//`
where the language does not use them -- then build the annotated string
from the corrected structure, which costs no extra lexing but cannot
recover keywords and strings the wrong comment range already suppressed;
or fork the twenty-file library and fix the locators. Filing it upstream
needs Bryan or a token, since there is no `gh` and no GitHub credential
in this VM. One-line repro: lexing `x '*/a/*'` as `SyntaxLanguage.SHELL`
in highlights 1.1.0 returns a highlight whose `location.end` precedes its
`location.start`. Whichever is chosen, `app/highlights-repro.sh` is what
checks it.
3. **Regression runs.** `transcript-bench.sh` and `stream-bench.sh` before
2. **Regression runs.** `transcript-bench.sh` and `stream-bench.sh` before
and after any change to the files above, with the report in the commit.
The numbers to watch are the worst `record: one block`, the reparse mean
while streaming, and the draw phase's accounting line.
app/androidApp/build.gradle.kts
+10 -1
View File
@@ -173,6 +173,15 @@ dependencies {
implementation(libs.androidx.lifecycle.runtime.compose)
implementation(libs.zxing.embedded)
implementation(libs.markdown.renderer)
implementation(libs.highlights)
implementation(libs.androidx.exifinterface)
// The syntax scanner (Highlighter.kt) is pure logic with no Android imports, which is what
// lets it be tested on the JVM: `./gradlew :androidApp:testDebugUnitTest`. The assertions are
// `kotlin.test`, so the tests name no framework; JUnit is what runs them.
testImplementation(libs.kotlin.test.junit5)
testImplementation(libs.junit.jupiter)
testRuntimeOnly(libs.junit.platform.launcher)
}
// JUnit 6 runs on the Platform, which is not Gradle's default for a Test task.
tasks.withType<Test>().configureEach { useJUnitPlatform() }
app/androidApp/src/main/kotlin/com/example/aiapp/CodeFence.kt
+56 -115
View File
@@ -11,23 +11,14 @@ import androidx.compose.foundation.text.BasicText
import androidx.compose.runtime.Composable
import androidx.compose.runtime.remember
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.semantics.isTraversalGroup
import androidx.compose.ui.semantics.semantics
import androidx.compose.ui.text.AnnotatedString
import androidx.compose.ui.text.SpanStyle
import androidx.compose.ui.text.TextStyle
import androidx.compose.ui.text.buildAnnotatedString
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.unit.dp
import com.mikepenz.markdown.compose.LocalMarkdownColors
import com.mikepenz.markdown.compose.LocalMarkdownDimens
import com.mikepenz.markdown.compose.LocalMarkdownPadding
import com.mikepenz.markdown.model.State
import dev.snipme.highlights.Highlights
import dev.snipme.highlights.model.BoldHighlight
import dev.snipme.highlights.model.ColorHighlight
import dev.snipme.highlights.model.SyntaxLanguage
import org.intellij.markdown.MarkdownElementTypes
import org.intellij.markdown.MarkdownTokenTypes
import org.intellij.markdown.ast.ASTNode
@@ -38,10 +29,10 @@ import org.intellij.markdown.ast.getTextInNode
* A fenced code block in a reply: the code highlighted, on the dark surface every verbatim thing
* sits on, scrolling sideways rather than wrapping.
*
* The renderer's own fence drew the same block in plain text. The lexer that colours a tool call's
* command colours a reply's code the same way, through [highlighted] and one theme, so a `kotlin`
* fence and the Kotlin a tool wrote are the same colours. A fence in a language the lexer has no
* rules for is plain rather than wrongly coloured: [fenceLanguage] answers null for those, and
* The renderer's own fence drew the same block in plain text. The scanner that colours a tool
* call's command colours a reply's code the same way, through [highlighted] and one palette, so a
* `kotlin` fence and the Kotlin a tool wrote are the same colours. A fence in a language [scan] has
* no rules for is plain rather than wrongly coloured: [fenceLanguage] answers null for those, and
* plain is what the reader would have seen before.
*
* Finding the code is still the library's: which children of the node are the fence markers, the
@@ -74,7 +65,7 @@ fun CodeBlock(
}
/**
* The code inside a fence or indented block, and the lexer's language for its info word.
* The code inside a fence or indented block, and the highlighter's language for its info word.
*
* Which children of the node are the fence markers, the language word and the code between them is
* the library's knowledge of the parser, copied from its `MarkdownCodeFence` rather than called:
@@ -85,7 +76,7 @@ fun CodeBlock(
* Null for a fence too short to hold anything -- an unterminated one still arriving, which the
* library skips as invalid.
*/
fun fenceContent(content: String, node: ASTNode): Pair<String, SyntaxLanguage?>? {
fun fenceContent(content: String, node: ASTNode): Pair<String, Language?>? {
val word =
node.findChildOfType(MarkdownTokenTypes.FENCE_LANG)?.getTextInNode(content)?.toString()
val language = fenceLanguage(word)
@@ -111,7 +102,7 @@ fun fenceContent(content: String, node: ASTNode): Pair<String, SyntaxLanguage?>?
@Composable
private fun CodeBlockText(
code: String,
language: SyntaxLanguage?,
language: Language?,
style: TextStyle,
replies: ParsedReplies,
streaming: Boolean,
@@ -136,122 +127,72 @@ private fun CodeBlockText(
}
/**
* The lexer's language for a fence's info word, or null for one it has no lexer for.
* The highlighter's language for a fence's info word, or null for one it has no rules for.
*
* The aliases are what people actually write after the backticks: the file extension as often as
* the name. A word not here gets no colour rather than the nearest lexer's, because a fence
* the name. A word not here gets no colour rather than the nearest language's, because a fence
* coloured by the wrong language's rules looks highlighted and is wrong in a way the reader cannot
* see.
*/
fun fenceLanguage(name: String?): SyntaxLanguage? =
fun fenceLanguage(name: String?): Language? =
FENCE_LANGUAGES[name?.trim()?.lowercase() ?: return null]
private val FENCE_LANGUAGES: Map<String, SyntaxLanguage> =
private val FENCE_LANGUAGES: Map<String, Language> =
mapOf(
"kotlin" to SyntaxLanguage.KOTLIN,
"kt" to SyntaxLanguage.KOTLIN,
"kts" to SyntaxLanguage.KOTLIN,
"rust" to SyntaxLanguage.RUST,
"rs" to SyntaxLanguage.RUST,
"sh" to SyntaxLanguage.SHELL,
"bash" to SyntaxLanguage.SHELL,
"shell" to SyntaxLanguage.SHELL,
"zsh" to SyntaxLanguage.SHELL,
"console" to SyntaxLanguage.SHELL,
"python" to SyntaxLanguage.PYTHON,
"py" to SyntaxLanguage.PYTHON,
"javascript" to SyntaxLanguage.JAVASCRIPT,
"js" to SyntaxLanguage.JAVASCRIPT,
"jsx" to SyntaxLanguage.JAVASCRIPT,
"typescript" to SyntaxLanguage.TYPESCRIPT,
"ts" to SyntaxLanguage.TYPESCRIPT,
"tsx" to SyntaxLanguage.TYPESCRIPT,
"java" to SyntaxLanguage.JAVA,
"c" to SyntaxLanguage.C,
"h" to SyntaxLanguage.C,
"cpp" to SyntaxLanguage.CPP,
"c++" to SyntaxLanguage.CPP,
"cc" to SyntaxLanguage.CPP,
"hpp" to SyntaxLanguage.CPP,
"csharp" to SyntaxLanguage.CSHARP,
"cs" to SyntaxLanguage.CSHARP,
"c#" to SyntaxLanguage.CSHARP,
"go" to SyntaxLanguage.GO,
"golang" to SyntaxLanguage.GO,
"swift" to SyntaxLanguage.SWIFT,
"dart" to SyntaxLanguage.DART,
"ruby" to SyntaxLanguage.RUBY,
"rb" to SyntaxLanguage.RUBY,
"php" to SyntaxLanguage.PHP,
"perl" to SyntaxLanguage.PERL,
"pl" to SyntaxLanguage.PERL,
"coffeescript" to SyntaxLanguage.COFFEESCRIPT,
"coffee" to SyntaxLanguage.COFFEESCRIPT,
"kotlin" to Language.KOTLIN,
"kt" to Language.KOTLIN,
"kts" to Language.KOTLIN,
"rust" to Language.RUST,
"rs" to Language.RUST,
"sh" to Language.SHELL,
"bash" to Language.SHELL,
"shell" to Language.SHELL,
"zsh" to Language.SHELL,
"console" to Language.SHELL,
"python" to Language.PYTHON,
"py" to Language.PYTHON,
"javascript" to Language.JAVASCRIPT,
"js" to Language.JAVASCRIPT,
"jsx" to Language.JAVASCRIPT,
"typescript" to Language.TYPESCRIPT,
"ts" to Language.TYPESCRIPT,
"tsx" to Language.TYPESCRIPT,
"java" to Language.JAVA,
"c" to Language.C,
"h" to Language.C,
"cpp" to Language.CPP,
"c++" to Language.CPP,
"cc" to Language.CPP,
"hpp" to Language.CPP,
"csharp" to Language.CSHARP,
"cs" to Language.CSHARP,
"c#" to Language.CSHARP,
"go" to Language.GO,
"golang" to Language.GO,
"swift" to Language.SWIFT,
"dart" to Language.DART,
"ruby" to Language.RUBY,
"rb" to Language.RUBY,
"php" to Language.PHP,
"perl" to Language.PERL,
"pl" to Language.PERL,
"coffeescript" to Language.COFFEESCRIPT,
"coffee" to Language.COFFEESCRIPT,
"ron" to Language.RON,
"toml" to Language.TOML,
"fish" to Language.FISH,
"json" to Language.JSON,
)
/**
* [code] with its keywords and strings coloured, or plain if there is no language for it.
*
* The lexing is dev.snipme:highlights. The colours are this app's, mapped in [catppuccinSyntax] --
* a library's default theme would be the one place in the app whose palette came from somewhere
* else. Shared by a tool call's input ([ToolInputView]) and a reply's fences ([CodeFence]), so the
* same code is the same colours wherever it appears.
*
* Not a composable, and it takes no colour from the theme, because that is what lets [warm] run it
* off the drawing thread: the syntax palette is fixed, and a fence with no language is plain text
* which needs no colour of its own -- the style the caller draws it with carries that.
*
* Measured on the emulator before it was cached: a two-hundred-line Kotlin fence costs **174ms** to
* lex, and the lazy list charged that again every time the block scrolled back into composition.
* That is the whole reason [ParsedReplies.highlighted] exists rather than a `remember`.
*/
fun highlight(code: String, language: SyntaxLanguage?): AnnotatedString {
if (language == null) return AnnotatedString(code)
val marks =
DebugStats.timed("code highlighted") {
Highlights.Builder(code = code, language = language, theme = catppuccinSyntax())
.build()
.getHighlights()
// highlights 1.1.0's shell lexer answers a quoted glob that looks like a comment
// -- `x '*/a/*'` is the smallest input -- with a span whose end is before its
// start, and AnnotatedString refuses such a range. That crashed the app the
// moment a card holding `-path '*/.git/*'` was opened. Dropped rather than
// clamped: a span the lexer got backwards is not one it knows the colour of.
// Delete when snipme/highlights fixes it.
.filter {
it.location.start in 0..it.location.end && it.location.end <= code.length
}
}
return buildAnnotatedString {
append(code)
marks.forEach { mark ->
when (mark) {
is ColorHighlight ->
addStyle(
SpanStyle(color = Color(mark.rgb or 0xFF000000.toInt())),
mark.location.start,
mark.location.end,
)
is BoldHighlight ->
addStyle(
SpanStyle(fontWeight = FontWeight.Bold),
mark.location.start,
mark.location.end,
)
}
}
}
}
/**
* Every fence in [parse], as the code and language [highlight] will be asked for.
*
* Walks the whole tree rather than the top level: a fence inside a list item or a quote is drawn
* the same way and costs the same to lex.
*/
fun fences(parse: State): List<Pair<String, SyntaxLanguage?>> {
fun fences(parse: State): List<Pair<String, Language?>> {
val success = parse as? State.Success ?: return emptyList()
val out = ArrayList<Pair<String, SyntaxLanguage?>>()
val out = ArrayList<Pair<String, Language?>>()
fun walk(node: ASTNode) {
if (
node.type == MarkdownElementTypes.CODE_FENCE ||
app/androidApp/src/main/kotlin/com/example/aiapp/Highlighter.kt
+318
View File
@@ -0,0 +1,318 @@
package com.example.aiapp
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.text.AnnotatedString
import androidx.compose.ui.text.SpanStyle
import androidx.compose.ui.text.buildAnnotatedString
/** What a span of code is, in the terms the palette has a colour for. */
enum class Kind {
KEYWORD,
STRING,
LITERAL,
COMMENT,
METADATA,
PUNCTUATION,
MARK,
}
/** A run of [Kind] in the code, as a half-open range. */
data class Span(val start: Int, val end: Int, val kind: Kind)
/**
* The colours the highlighter draws with, ours rather than a library's; [catppuccinSyntax] is the
* one instance and lives with the rest of the palette.
*/
data class SyntaxPalette(
val keyword: Color,
val string: Color,
val literal: Color,
val comment: Color,
val metadata: Color,
val punctuation: Color,
val mark: Color,
) {
fun of(kind: Kind): Color =
when (kind) {
Kind.KEYWORD -> keyword
Kind.STRING -> string
Kind.LITERAL -> literal
Kind.COMMENT -> comment
Kind.METADATA -> metadata
Kind.PUNCTUATION -> punctuation
Kind.MARK -> mark
}
}
/**
* [code] with its keywords, strings and comments coloured, or plain if there is no language for it.
*
* Shared by a tool call's input ([ToolInputView]) and a reply's fences ([CodeFence]), so the same
* code is the same colours wherever it appears.
*
* Not a composable, and it takes no colour from the theme, because that is what lets [warm] run it
* off the drawing thread: the syntax palette is fixed, and a fence with no language is plain text
* which needs no colour of its own -- the style the caller draws it with carries that.
*
* The timing is the number the highlighter is judged by: the library this replaced took **174ms**
* on the emulator for a two-hundred-line Kotlin fence, which is why [ParsedReplies.highlighted]
* caches the answer rather than a `remember` inside the fence recomputing it on every scroll back.
*/
fun highlight(code: String, language: Language?): AnnotatedString {
if (language == null) return AnnotatedString(code)
val spans = DebugStats.timed("code highlighted") { scan(code, rulesOf(language)) }
val palette = catppuccinSyntax()
return buildAnnotatedString {
append(code)
spans.forEach { addStyle(SpanStyle(color = palette.of(it.kind)), it.start, it.end) }
}
}
/**
* [code] read once, left to right, into the spans that carry a colour.
*
* One pass with a small state -- in a comment, in a string, or in ordinary code -- rather than a
* locator per token kind over the whole text, which is what the library did and is why it found
* comments before it knew the language: a `#` inside a shell string, a `//` inside a URL and a
* block-comment opener inside a shell glob each commented out the rest of a line that was nothing
* of the sort.
*
* Every span is produced by advancing an index forward, so the result is ordered, non-overlapping
* and inside the code by construction. Nothing here throws: an unterminated string or comment runs
* to the end of the code, which is also what it looks like while a fence is still being written.
*
* In ordinary code the order of recognition is comment, string, attribute, number, word, and
* finally a single punctuation or mark character. Punctuation and marks are coloured only in
* ordinary code, never inside a string or a comment.
*/
fun scan(code: String, rules: Rules): List<Span> = Scanner(code, rules).run()
/** Characters coloured as punctuation, and as marks. Both sets are the ones the library used. */
private const val PUNCTUATION = ",.:;"
private const val MARKS = "()={}<>-+[]|&"
private class Scanner(private val code: String, private val rules: Rules) {
private val spans = ArrayList<Span>()
private var at = 0
fun run(): List<Span> {
while (at < code.length) {
// Every branch that answers true has advanced `at`, so this terminates.
val consumed =
blockComment() ||
lineComment() ||
rawString() ||
characterOrLifetime() ||
string() ||
attribute() ||
number() ||
word() ||
singleCharacter()
if (!consumed) at++
}
return spans
}
private fun emit(start: Int, kind: Kind) {
if (at > start) spans.add(Span(start, at, kind))
}
private fun starts(token: String) = code.startsWith(token, at)
/** Whether a line comment token here opens one; see [Rules.lineCommentsAtWordStart]. */
private fun atWordStart() = at == 0 || code[at - 1].isWhitespace() || code[at - 1] in ";|&("
/** Whether only whitespace stands between the start of this line and here. */
private fun atLineStart(): Boolean {
var back = at - 1
while (back >= 0 && code[back] != '\n') {
if (!code[back].isWhitespace()) return false
back--
}
return true
}
private fun toEndOfLine() {
while (at < code.length && code[at] != '\n') at++
}
/** From an open bracket through the one that matches it, or to the end if none does. */
private fun toMatchingBracket() {
var depth = 0
while (at < code.length) {
when (code[at]) {
'[' -> depth++
']' -> depth--
}
at++
if (depth == 0) return
}
}
private fun blockComment(): Boolean {
val comment = rules.blockComment ?: return false
if (!starts(comment.open)) return false
val start = at
at += comment.open.length
var depth = 1
while (at < code.length && depth > 0) {
// The closer is tried first so that a language whose two delimiters are the same
// string -- CoffeeScript's `###` -- closes rather than nesting forever.
if (starts(comment.close)) {
depth--
at += comment.close.length
} else if (comment.nests && starts(comment.open)) {
depth++
at += comment.open.length
} else {
at++
}
}
emit(start, Kind.COMMENT)
return true
}
private fun lineComment(): Boolean {
if (rules.lineComments.none { starts(it) }) return false
if (rules.lineCommentsAtWordStart && !atWordStart()) return false
val start = at
toEndOfLine()
emit(start, Kind.COMMENT)
return true
}
/** Rust and RON: `b`? `r` `#`* `"` … `"` `#`*, with no escapes inside. */
private fun rawString(): Boolean {
if (!rules.rawStrings) return false
var ahead = at
if (code.getOrNull(ahead) == 'b') ahead++
if (code.getOrNull(ahead) != 'r') return false
ahead++
var hashes = 0
while (code.getOrNull(ahead) == '#') {
ahead++
hashes++
}
if (code.getOrNull(ahead) != '"') return false
val start = at
val closer = "\"" + "#".repeat(hashes)
val closed = code.indexOf(closer, ahead + 1)
at = if (closed < 0) code.length else closed + closer.length
emit(start, Kind.STRING)
return true
}
/** See [Rules.lifetimes]: an apostrophe that is not a character literal opens nothing. */
private fun characterOrLifetime(): Boolean {
if (!rules.lifetimes || code[at] != '\'') return false
val next = code.getOrNull(at + 1) ?: return false
if (next == '\\' || code.getOrNull(at + 2) == '\'') {
quoted(Quote("'", "'", escapes = true))
} else {
at++
}
return true
}
private fun string(): Boolean {
// Longest opener wins, so Kotlin's `"""` is one delimiter rather than an empty string
// followed by a quote. A loop rather than filter/maxBy: this runs at every character of
// ordinary code, and the pair of lists that would allocate is the whole cost of the scan.
var quote: Quote? = null
for (candidate in rules.quotes) {
if (starts(candidate.open) && candidate.open.length > (quote?.open?.length ?: 0)) {
quote = candidate
}
}
quoted(quote ?: return false)
return true
}
private fun quoted(quote: Quote) {
val start = at
at += quote.open.length
while (at < code.length) {
if (quote.escapes && code[at] == '\\' && at + 1 < code.length) {
at += 2
continue
}
if (starts(quote.close)) {
at += quote.close.length
break
}
at++
}
at = at.coerceAtMost(code.length)
emit(start, Kind.STRING)
}
private fun attribute(): Boolean {
val start = at
when (rules.attributes) {
Attributes.NONE -> return false
Attributes.AT_WORD -> {
if (code[at] != '@' || !isWordStart(code.getOrNull(at + 1))) return false
at++
while (at < code.length && isWordPart(code[at])) at++
}
Attributes.HASH_BRACKET -> {
if (code[at] != '#') return false
var ahead = at + 1
if (code.getOrNull(ahead) == '!') ahead++
if (code.getOrNull(ahead) != '[') return false
at = ahead
toMatchingBracket()
}
Attributes.HASH_LINE -> {
if (code[at] != '#' || !atLineStart()) return false
toEndOfLine()
}
Attributes.LINE_BRACKET -> {
if (code[at] != '[' || !atLineStart()) return false
toMatchingBracket()
}
}
emit(start, Kind.METADATA)
return true
}
/**
* A number is a run starting with a digit and carrying on through letters, digits, `_` and `.`
* -- which covers `0xFF`, `1_000`, `1u32` and `3.14` without a grammar for any of them.
*/
private fun number(): Boolean {
if (!code[at].isDigit()) return false
val start = at
while (
at < code.length && (code[at].isLetterOrDigit() || code[at] == '_' || code[at] == '.')
) {
at++
}
emit(start, Kind.LITERAL)
return true
}
private fun word(): Boolean {
if (!isWordStart(code[at])) return false
val start = at
while (at < code.length && isWordPart(code[at])) at++
if (code.substring(start, at) in rules.keywords) emit(start, Kind.KEYWORD)
return true
}
private fun singleCharacter(): Boolean {
val kind =
when (code[at]) {
in PUNCTUATION -> Kind.PUNCTUATION
in MARKS -> Kind.MARK
else -> return false
}
at++
emit(at - 1, kind)
return true
}
}
private fun isWordStart(c: Char?) = c != null && (c.isLetter() || c == '_')
private fun isWordPart(c: Char) = c.isLetterOrDigit() || c == '_'
app/androidApp/src/main/kotlin/com/example/aiapp/Languages.kt
+437
View File
@@ -0,0 +1,437 @@
package com.example.aiapp
/**
* A language the highlighter has rules for.
*
* The names the reader writes after the backticks are aliases onto these; [fenceLanguage] holds
* that table. A word with no entry there is null, and null is drawn plain, because a fence coloured
* by another language's rules looks highlighted and is wrong in a way the reader cannot see.
*/
enum class Language {
C,
COFFEESCRIPT,
CPP,
CSHARP,
DART,
FISH,
GO,
JAVA,
JAVASCRIPT,
JSON,
KOTLIN,
PERL,
PHP,
PYTHON,
RON,
RUBY,
RUST,
SHELL,
SWIFT,
TOML,
TYPESCRIPT,
}
/**
* What [scan] needs to know about one language -- data, not code, so that adding a language is a
* row in [RULES] rather than a branch anywhere.
*
* The two forms that could not be expressed as data are flags here and a few lines in the scanner:
* [rawStrings], because the closing delimiter depends on how many hashes the opener had, and
* [lifetimes], because whether `'` opens anything at all depends on what follows it.
*/
data class Rules(
/** Words drawn as keywords. Only plain words; the scanner cannot reach anything else. */
val keywords: Set<String>,
/** Tokens that open a comment running to the end of the line. */
val lineComments: List<String> = emptyList(),
/**
* Whether [lineComments] count only at the start of a word.
*
* The shells need it: `$#`, `${#x}` and `a#b` are not comments, and greying the rest of those
* lines is one of the mistakes this scanner exists to stop.
*/
val lineCommentsAtWordStart: Boolean = false,
val blockComment: BlockComment? = null,
/** The string forms. The longest opener that matches wins, so `"""` is tried before `"`. */
val quotes: List<Quote> = emptyList(),
val attributes: Attributes = Attributes.NONE,
/** Rust and RON: an optional `b`, `r`, n hashes, `"`, closing at `"` and n hashes. */
val rawStrings: Boolean = false,
/**
* Rust: `'` opens a character literal only when a backslash or one character and a `'` follow.
* Otherwise it is a lifetime or a label and no string starts -- without this, `'a` opens a
* string that runs to the next apostrophe in the block.
*/
val lifetimes: Boolean = false,
)
data class BlockComment(val open: String, val close: String, val nests: Boolean)
/** One string form. [escapes] is whether a backslash escapes the closer (and itself). */
data class Quote(val open: String, val close: String, val escapes: Boolean)
/** What opens a metadata span, of the shapes that exist across these languages. */
enum class Attributes {
NONE,
/** `@` and a word: Kotlin and Java annotations, Python decorators. */
AT_WORD,
/** `#[` or `#![` through the matching `]`: Rust and RON attributes. */
HASH_BRACKET,
/** `#` at the start of a line, to the end of it: the C preprocessor. */
HASH_LINE,
/** `[` at the start of a line through the matching `]`: a TOML table header. */
LINE_BRACKET,
}
/** The rules for [language]. */
fun rulesOf(language: Language): Rules = RULES.getValue(language)
private val C_STYLE = BlockComment("/*", "*/", nests = false)
private val NESTING = BlockComment("/*", "*/", nests = true)
private val DOUBLE = Quote("\"", "\"", escapes = true)
private val SINGLE = Quote("'", "'", escapes = true)
private val TRIPLE_DOUBLE = Quote("\"\"\"", "\"\"\"", escapes = true)
private val TRIPLE_SINGLE = Quote("'''", "'''", escapes = true)
// Lazy because the keyword sets below are top-level properties too, and a file's properties
// initialize in the order they are written: read eagerly here, every set would be null.
private val RULES: Map<Language, Rules> by lazy {
mapOf(
Language.C to
Rules(
keywords = KEYWORDS_C,
lineComments = listOf("//"),
blockComment = C_STYLE,
quotes = listOf(DOUBLE, SINGLE),
attributes = Attributes.HASH_LINE,
),
Language.CPP to
Rules(
keywords = KEYWORDS_CPP,
lineComments = listOf("//"),
blockComment = C_STYLE,
quotes = listOf(DOUBLE, SINGLE),
attributes = Attributes.HASH_LINE,
),
Language.CSHARP to
Rules(
keywords = KEYWORDS_CSHARP,
lineComments = listOf("//"),
blockComment = C_STYLE,
quotes = listOf(DOUBLE, SINGLE),
),
// `###` opens and closes a block comment and `#` opens a line one, which is why the
// scanner tries the block opener first.
Language.COFFEESCRIPT to
Rules(
keywords = KEYWORDS_COFFEESCRIPT,
lineComments = listOf("#"),
blockComment = BlockComment("###", "###", nests = false),
quotes = listOf(TRIPLE_DOUBLE, TRIPLE_SINGLE, DOUBLE, SINGLE),
),
Language.DART to
Rules(
keywords = KEYWORDS_DART,
lineComments = listOf("//"),
blockComment = NESTING,
quotes = listOf(TRIPLE_DOUBLE, TRIPLE_SINGLE, DOUBLE, SINGLE),
attributes = Attributes.AT_WORD,
),
Language.FISH to
Rules(
keywords = KEYWORDS_FISH,
lineComments = listOf("#"),
lineCommentsAtWordStart = true,
// fish's single quotes escape only `\'` and `\\`, which is what "skip the
// character after a backslash" already does.
quotes = listOf(DOUBLE, SINGLE),
),
Language.GO to
Rules(
keywords = KEYWORDS_GO,
lineComments = listOf("//"),
blockComment = C_STYLE,
quotes = listOf(DOUBLE, SINGLE, Quote("`", "`", escapes = false)),
),
Language.JAVA to
Rules(
keywords = KEYWORDS_JAVA,
lineComments = listOf("//"),
blockComment = C_STYLE,
quotes = listOf(DOUBLE, SINGLE),
attributes = Attributes.AT_WORD,
),
Language.JAVASCRIPT to
Rules(
keywords = KEYWORDS_JAVASCRIPT,
lineComments = listOf("//"),
blockComment = C_STYLE,
quotes = listOf(DOUBLE, SINGLE, Quote("`", "`", escapes = true)),
),
Language.JSON to Rules(keywords = KEYWORDS_JSON, quotes = listOf(DOUBLE)),
Language.KOTLIN to
Rules(
keywords = KEYWORDS_KOTLIN,
lineComments = listOf("//"),
blockComment = NESTING,
quotes = listOf(Quote("\"\"\"", "\"\"\"", escapes = false), DOUBLE, SINGLE),
attributes = Attributes.AT_WORD,
),
Language.PERL to
Rules(
keywords = KEYWORDS_PERL,
lineComments = listOf("#"),
quotes = listOf(DOUBLE, SINGLE),
),
Language.PHP to
Rules(
keywords = KEYWORDS_PHP,
lineComments = listOf("//", "#"),
blockComment = C_STYLE,
quotes = listOf(DOUBLE, SINGLE),
attributes = Attributes.AT_WORD,
),
Language.PYTHON to
Rules(
keywords = KEYWORDS_PYTHON,
lineComments = listOf("#"),
quotes = listOf(TRIPLE_DOUBLE, TRIPLE_SINGLE, DOUBLE, SINGLE),
attributes = Attributes.AT_WORD,
),
Language.RON to
Rules(
keywords = KEYWORDS_RON,
lineComments = listOf("//"),
blockComment = NESTING,
quotes = listOf(DOUBLE, SINGLE),
attributes = Attributes.HASH_BRACKET,
rawStrings = true,
),
Language.RUBY to
Rules(
keywords = KEYWORDS_RUBY,
lineComments = listOf("#"),
quotes = listOf(DOUBLE, SINGLE),
),
Language.RUST to
Rules(
keywords = KEYWORDS_RUST,
lineComments = listOf("//"),
blockComment = NESTING,
// No `'` here: [Rules.lifetimes] decides when one opens a character literal.
quotes = listOf(DOUBLE),
attributes = Attributes.HASH_BRACKET,
rawStrings = true,
lifetimes = true,
),
Language.SHELL to
Rules(
keywords = KEYWORDS_SHELL,
lineComments = listOf("#"),
lineCommentsAtWordStart = true,
// A shell's single quotes are literal: `'a\'` is not one string.
quotes = listOf(DOUBLE, Quote("'", "'", escapes = false)),
),
Language.SWIFT to
Rules(
keywords = KEYWORDS_SWIFT,
lineComments = listOf("//"),
blockComment = NESTING,
quotes = listOf(TRIPLE_DOUBLE, DOUBLE),
attributes = Attributes.AT_WORD,
),
Language.TOML to
Rules(
keywords = KEYWORDS_TOML,
lineComments = listOf("#"),
quotes =
listOf(
TRIPLE_DOUBLE,
Quote("'''", "'''", escapes = false),
DOUBLE,
Quote("'", "'", escapes = false),
),
attributes = Attributes.LINE_BRACKET,
),
Language.TYPESCRIPT to
Rules(
keywords = KEYWORDS_TYPESCRIPT,
lineComments = listOf("//"),
blockComment = C_STYLE,
quotes = listOf(DOUBLE, SINGLE, Quote("`", "`", escapes = true)),
attributes = Attributes.AT_WORD,
),
)
}
/**
* The keyword sets.
*
* Every list below other than RON, TOML, fish and JSON came from dev.snipme:highlights 1.1.0
* (`SyntaxTokens.kt`, Apache-2.0), the library this scanner replaced, so that no fence which is
* coloured today turns plain. Entries that are not plain words were dropped -- Kotlin's `as?`,
* `!in` and `!is`, Swift's `#if` family, Ruby's `defined?`, CoffeeScript's `=` and `->` -- because
* the word scanner cannot reach them and the library only matched them by luck.
*/
private fun words(list: String): Set<String> =
list.split(Regex("\\s+")).filterNot(String::isEmpty).toSet()
private val KEYWORDS_C =
words(
"""auto break case char const continue default do double else enum extern float for goto if
int long register return short signed sizeof static struct switch typedef union unsigned
void volatile while"""
)
private val KEYWORDS_CPP =
words(
"""asm auto bool break case catch char class const const_cast continue default delete do
double dynamic_cast else enum explicit export extern false float for friend goto if inline
int long mutable namespace new operator private protected public register reinterpret_cast
return short signed sizeof static static_cast struct switch template this throw true try
typedef typeid typename union unsigned using virtual void volatile wchar_t while"""
)
private val KEYWORDS_CSHARP =
words(
"""abstract as base bool break byte case catch char checked class const continue decimal
default delegate do double else enum event explicit extern false finally fixed float for
foreach goto if implicit in int interface internal is lock long namespace new null object
operator out override params private protected public readonly ref return sbyte sealed short
sizeof stackalloc static string struct switch this throw true try typeof uint ulong unchecked
unsafe ushort using virtual void volatile while"""
)
private val KEYWORDS_COFFEESCRIPT =
words(
"""Infinity NaN and arguments await break by case catch class continue debugger delete defer
default do else export extends false finally for function if import in instanceof is isnt
let loop new no not null of on or package return super switch this throw true try typeof
unless undefined var wait when with yield"""
)
private val KEYWORDS_DART =
words(
"""abstract as assert async await base break case catch class const continue covariant
default deferred do dynamic else enum export extends external factory false final finally
for get if implements import in interface is late library mixin new null on operator part
required rethrow return sealed set show static super switch this throw true try var void
when with while yield"""
)
/**
* fish is not in the library at all, so its fences are drawn plain today. The list is the shell's
* own words, which is what a fish fence is mostly made of.
*/
private val KEYWORDS_FISH =
words(
"""and begin break builtin case command continue else end exec for function if in not or
return switch while set echo test string math read source"""
)
private val KEYWORDS_GO =
words(
"""break case chan const continue default defer else fallthrough false for func go goto if
import interface map package range return select struct switch true type var"""
)
private val KEYWORDS_JAVA =
words(
"""abstract assert boolean break byte case catch char class const continue default do double
else enum extends final finally float for goto if implements import instanceof int interface
long native new null package private protected public return short static strictfp super
switch synchronized this throw throws transient try void volatile while"""
)
private val KEYWORDS_JAVASCRIPT =
words(
"""async await boolean break case catch class const continue debugger default delete do else
enum export extends false finally for function if implements import in instanceof interface
let new null package private protected public return super switch this throw true try typeof
var void while with yield"""
)
private val KEYWORDS_JSON = words("true false null")
private val KEYWORDS_KOTLIN =
words(
"""actual abstract annotation as break by catch class companion const constructor continue
coroutine crossinline data delegate dynamic do else enum expect external false final finally
for fun get if import in infix inline interface internal is lazy lateinit native null object
open operator out override package private protected public reified return sealed set super
suspend tailrec this throw true try typealias typeof val var vararg when while yield"""
)
private val KEYWORDS_PERL =
words(
"""__DATA__ __END__ __FILE__ __LINE__ __PACKAGE__ and cmp continue do else elsif eq eval for
foreach goto gt if last le lt my ne next no not or package redo ref return sub unless until
use while xor"""
)
private val KEYWORDS_PHP =
words(
"""__halt_compiler abstract and array as break callable case catch class clone const continue
declare default die do echo else elseif empty enddeclare endfor endforeach endif endswitch
endwhile eval exit extends final finally fn for foreach function global goto if implements
include include_once instanceof insteadof interface isset list match new or print private
protected public require require_once return static switch throw trait try unset use var
while xor yield"""
)
private val KEYWORDS_PYTHON =
words(
"""False True and as assert async await break class continue def del elif else except finally
for from global if import in is lambda nonlocal not or pass raise return try while with
yield"""
)
/** RON is not in the library either; these are the words a RON file can hold. */
private val KEYWORDS_RON = words("true false Some None inf NaN")
private val KEYWORDS_RUBY =
words(
"""__ENCODING__ __END__ __FILE__ __LINE__ BEGIN END alias and begin break case class def do
else elsif end ensure false for if in module next nil not or redo rescue retry return self
super then true undef unless until when while yield"""
)
private val KEYWORDS_RUST =
words(
"""as async await break const continue crate dyn else enum extern false fn for if impl in
let loop match mod move mut pub ref return Self self static struct super trait true type
union unsafe use where while abstract become box do final macro override priv try typeof
unsized virtual yield"""
)
private val KEYWORDS_SHELL =
words(
"""alias bg bind break builtin caller cd command compgen complete compopt continue declare
dirs disown echo enable eval exec exit export fc fg getopts hash help history jobs kill let
local logout popd printf pushd pwd read readonly return set shift shopt source suspend
test"""
)
private val KEYWORDS_SWIFT =
words(
"""_ associatedtype class deinit enum extension fileprivate func import init inout internal
let open operator private precedencegroup protocol public rethrows static struct subscript
typealias var break case catch continue default defer do else fallthrough for guard if in
repeat return throw switch where while Any as await false is nil self Self super throws true
try associativity convenience didSet dynamic final get indirect infix lazy left mutating none
nonmutating optional override postfix precedence prefix Protocol required right set some Type
unowned weak willSet"""
)
/** TOML is not in the library; `inf` and `nan` are values rather than names, like the booleans. */
private val KEYWORDS_TOML = words("true false inf nan")
private val KEYWORDS_TYPESCRIPT =
words(
"""abstract as asserts await break case catch class const constructor continue debugger
default delete do else enum export extends false finally for from function get if implements
import in infer instanceof interface is keyof let module namespace new null number object
package private protected public readonly require global return set static string super
switch this throw true try type typeof undefined unique unknown var void while with yield"""
)
app/androidApp/src/main/kotlin/com/example/aiapp/Markdown.kt
+1 -2
View File
@@ -55,7 +55,6 @@ import com.mikepenz.markdown.model.markdownAnimations
import com.mikepenz.markdown.model.markdownDimens
import com.mikepenz.markdown.model.markdownPadding
import com.mikepenz.markdown.model.parseMarkdown
import dev.snipme.highlights.model.SyntaxLanguage
import java.util.concurrent.ConcurrentHashMap
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
@@ -670,7 +669,7 @@ class ParsedReplies {
*
* The key carries the language, because the same code lexes differently under two of them.
*/
fun highlighted(code: String, language: SyntaxLanguage?): AnnotatedString =
fun highlighted(code: String, language: Language?): AnnotatedString =
if (language == null) AnnotatedString(code)
else highlights.computeIfAbsent("$language\n$code") { highlight(code, language) }
app/androidApp/src/main/kotlin/com/example/aiapp/Theme.kt
+16 -18
View File
@@ -6,8 +6,6 @@ import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.darkColorScheme
import androidx.compose.runtime.Composable
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.graphics.toArgb
import dev.snipme.highlights.model.SyntaxTheme
/**
* Catppuccin Mocha, as published in `catppuccin/palette`.
@@ -198,24 +196,24 @@ val rawSurface: Color
@Composable get() = Mocha.Crust
/**
* Catppuccin Mocha as a syntax theme, for the highlighter used on a tool call's input.
* Catppuccin Mocha as the highlighter's palette; see [SyntaxPalette].
*
* Here with the rest of the palette rather than beside the code that highlights: a library's own
* theme would otherwise be the one surface in the app whose colours came from somewhere else, and
* the accents below are the same ones every other coloured thing already uses.
* Here with the rest of the palette rather than beside the code that highlights: the colours a
* fence is drawn in are the same accents every other coloured thing in the app already uses, and
* splitting them out would make code the one surface whose palette came from somewhere else.
*
* Not a composable, because [highlight] runs off the drawing thread; these colours never vary with
* the theme.
*/
fun catppuccinSyntax(): SyntaxTheme =
SyntaxTheme(
key = "catppuccin-mocha",
code = Mocha.Text.toArgb(),
keyword = Mocha.Mauve.toArgb(),
string = Mocha.Green.toArgb(),
literal = Mocha.Peach.toArgb(),
comment = Mocha.Overlay0.toArgb(),
metadata = Mocha.Yellow.toArgb(),
multilineComment = Mocha.Overlay0.toArgb(),
punctuation = Mocha.Subtext0.toArgb(),
mark = Mocha.Sky.toArgb(),
fun catppuccinSyntax(): SyntaxPalette =
SyntaxPalette(
keyword = Mocha.Mauve,
string = Mocha.Green,
literal = Mocha.Peach,
comment = Mocha.Overlay0,
metadata = Mocha.Yellow,
punctuation = Mocha.Subtext0,
mark = Mocha.Sky,
)
/**
app/androidApp/src/main/kotlin/com/example/aiapp/ToolInput.kt
+3 -4
View File
@@ -11,7 +11,6 @@ import androidx.compose.runtime.remember
import androidx.compose.ui.Modifier
import androidx.compose.ui.text.font.FontFamily
import androidx.compose.ui.unit.dp
import dev.snipme.highlights.model.SyntaxLanguage
import org.json.JSONObject
/**
@@ -27,7 +26,7 @@ data class ToolInput(
/** The thing that will actually be run or read, if this tool has one. */
val subject: String?,
/** The language [subject] is written in, for highlighting. */
val language: SyntaxLanguage?,
val language: Language?,
/** The tool's own one-line summary, when it wrote one. */
val description: String?,
/**
@@ -50,9 +49,9 @@ data class ToolInput(
* from being the special case that gets its own code path. Unknown tools fall through to "no
* subject, everything is rest", which is what the card always did.
*/
private val SUBJECTS: Map<String, Pair<String, SyntaxLanguage?>> =
private val SUBJECTS: Map<String, Pair<String, Language?>> =
mapOf(
"Bash" to ("command" to SyntaxLanguage.SHELL),
"Bash" to ("command" to Language.SHELL),
"Read" to ("file_path" to null),
"Write" to ("file_path" to null),
"Edit" to ("file_path" to null),
app/androidApp/src/test/kotlin/com/example/aiapp/HighlighterTest.kt
+240
View File
@@ -0,0 +1,240 @@
package com.example.aiapp
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
/**
* What the scanner colours, asserted as the text under each span rather than as offsets, so a
* failure prints the code that was got wrong instead of a pair of numbers.
*
* Most of these are the mistakes dev.snipme:highlights 1.1.0 made -- the library this scanner
* replaced -- measured against it directly before it was removed. They are here rather than in the
* `highlights-repro.sh` script they came from because a case that only a script can ask about is a
* case nobody asks about.
*/
class HighlighterTest {
/** Every span of [kind] in [code], as the text it covers. */
private fun spans(code: String, language: Language, kind: Kind): List<String> =
scan(code, rulesOf(language))
.filter { it.kind == kind }
.map { code.substring(it.start, it.end) }
private fun assertSpans(
code: String,
language: Language,
kind: Kind,
vararg expected: String,
) {
assertEquals(expected.toList(), spans(code, language, kind), "$kind in: $code")
}
// The four inputs the repro script asked the library about, and what it answered.
@Test
fun `a quoted glob is one string, not a comment`() {
// The library answered a span whose end preceded its start here, which crashed the app.
assertSpans("x '*/a/*'", Language.SHELL, Kind.STRING, "'*/a/*'")
assertSpans("x '*/a/*'", Language.SHELL, Kind.COMMENT)
}
@Test
fun `a find with globs has no comment in it`() {
val code = "find . -path '*/.git/*' -prune -o -name '*.kt' -print"
assertSpans(code, Language.SHELL, Kind.STRING, "'*/.git/*'", "'*.kt'")
assertSpans(code, Language.SHELL, Kind.COMMENT)
}
@Test
fun `a URL does not comment out the rest of a shell line`() {
val code = "curl https://example.com/x && echo done"
assertSpans(code, Language.SHELL, Kind.COMMENT)
assertSpans(code, Language.SHELL, Kind.KEYWORD, "echo")
}
@Test
fun `a URL inside a Kotlin string stays a string`() {
val code = "val url = \"https://example.com\"\nfun f() = 1"
assertSpans(code, Language.KOTLIN, Kind.COMMENT)
assertSpans(code, Language.KOTLIN, Kind.STRING, "\"https://example.com\"")
assertSpans(code, Language.KOTLIN, Kind.KEYWORD, "val", "fun")
}
// Attributes, which the library greyed out as comments.
@Test
fun `a Rust attribute is metadata and the struct after it still colours`() {
val code = "#[derive(Debug)]\nstruct A { b: u8 }"
assertSpans(code, Language.RUST, Kind.METADATA, "#[derive(Debug)]")
assertSpans(code, Language.RUST, Kind.COMMENT)
assertSpans(code, Language.RUST, Kind.KEYWORD, "struct")
}
@Test
fun `an inner Rust attribute closes at its own bracket`() {
val code = "#![allow(dead_code)]\nfn f() {}"
assertSpans(code, Language.RUST, Kind.METADATA, "#![allow(dead_code)]")
assertSpans(code, Language.RUST, Kind.KEYWORD, "fn")
}
@Test
fun `a C preprocessor line is metadata rather than a comment`() {
val code = "#include <stdio.h>\nint main() { return 0; }"
assertSpans(code, Language.C, Kind.METADATA, "#include <stdio.h>")
assertSpans(code, Language.C, Kind.COMMENT)
assertSpans(code, Language.C, Kind.KEYWORD, "int", "return")
}
@Test
fun `a Kotlin annotation is metadata`() {
assertSpans("@Composable fun f() {}", Language.KOTLIN, Kind.METADATA, "@Composable")
}
// Strings whose contents the library read as code.
@Test
fun `a hash inside a Kotlin string is not a comment`() {
val code = "val c = \"#FF0000\"\nval d = 1"
assertSpans(code, Language.KOTLIN, Kind.COMMENT)
assertSpans(code, Language.KOTLIN, Kind.STRING, "\"#FF0000\"")
}
@Test
fun `an apostrophe inside a Kotlin string does not open one`() {
val code = "val a = \"don't\"\nval b = \"x\""
assertSpans(code, Language.KOTLIN, Kind.STRING, "\"don't\"", "\"x\"")
}
@Test
fun `a Rust lifetime does not open a string but a character literal does`() {
val code = "fn f<'a>(x: &'a str) { let c = 'x'; }"
assertSpans(code, Language.RUST, Kind.STRING, "'x'")
}
@Test
fun `an escaped quote is inside the Rust character literal`() {
assertSpans("let c = '\\'';", Language.RUST, Kind.STRING, "'\\''")
}
@Test
fun `a Rust raw string keeps its inner quotes`() {
val code = "let s = r#\"a \"quoted\" b\"#;"
assertSpans(code, Language.RUST, Kind.STRING, "r#\"a \"quoted\" b\"#")
}
@Test
fun `a Kotlin triple quoted string is one string`() {
assertSpans(
"val s = \"\"\"a \"b\" c\"\"\"",
Language.KOTLIN,
Kind.STRING,
"\"\"\"a \"b\" c\"\"\"",
)
}
@Test
fun `a shell single quoted string takes no escapes`() {
// `\` is literal inside shell single quotes, so the string ends at the next apostrophe.
assertSpans("echo 'a\\' b", Language.SHELL, Kind.STRING, "'a\\'")
}
// Comments.
@Test
fun `Rust and Kotlin nest block comments`() {
val code = "/* a /* b */ c */ x"
assertSpans(code, Language.RUST, Kind.COMMENT, "/* a /* b */ c */")
assertSpans(code, Language.KOTLIN, Kind.COMMENT, "/* a /* b */ c */")
}
@Test
fun `C ends a block comment at the first close`() {
assertSpans("/* a /* b */ c */ x", Language.C, Kind.COMMENT, "/* a /* b */")
}
@Test
fun `a shell comment starts only at a word boundary`() {
val code = "\${#x} \$# a#b # real"
assertSpans(code, Language.SHELL, Kind.COMMENT, "# real")
}
@Test
fun `a hash anywhere is a Python comment`() {
assertSpans("x = 1 # note", Language.PYTHON, Kind.COMMENT, "# note")
}
// TOML, which the library has no rules for at all.
@Test
fun `a TOML table header is metadata and a hash in a value is not a comment`() {
val code = "[server]\ncolour = \"#FF0000\"\nport = 8080 # the real one"
assertSpans(code, Language.TOML, Kind.METADATA, "[server]")
assertSpans(code, Language.TOML, Kind.STRING, "\"#FF0000\"")
assertSpans(code, Language.TOML, Kind.COMMENT, "# the real one")
assertSpans(code, Language.TOML, Kind.LITERAL, "8080")
}
@Test
fun `a RON attribute and its values colour`() {
val code = "#![enable(implicit_some)]\n(count: 3, on: true)"
assertSpans(code, Language.RON, Kind.METADATA, "#![enable(implicit_some)]")
assertSpans(code, Language.RON, Kind.KEYWORD, "true")
assertSpans(code, Language.RON, Kind.LITERAL, "3")
}
@Test
fun `an unknown fence language is drawn plain`() {
assertEquals(null, fenceLanguage("brainfuck"))
assertEquals("+[-]", highlight("+[-]", fenceLanguage("brainfuck")).text)
}
@Test
fun `every alias the fence table knows has rules`() {
Language.entries.forEach { rulesOf(it) }
}
/**
* The scanner must never throw and must never answer a span the code does not contain: the
* library's reversed range is exactly the shape that crashed a card, and a fence still being
* written is an unterminated string or comment on every keystroke.
*/
@Test
fun `spans stay inside the code for every language and every nasty input`() {
val nasty =
listOf(
"",
"'",
"\"",
"\"unterminated",
"/* unterminated",
"###",
"#",
"#![",
"[",
"r#\"",
"\\",
"'''",
"\"\"\"",
"0x",
"1.2.3",
"a#b//c/*d*/'e\"f",
"\n\n \n",
)
for (language in Language.entries) {
for (code in nasty) {
val spans = scan(code, rulesOf(language))
spans.forEach {
assertTrue(
it.start in 0..it.end && it.end <= code.length,
"$language answered $it for ${code.replace("\n", "\\n")}",
)
}
assertEquals(
spans.sortedBy { it.start },
spans,
"$language answered spans out of order for ${code.replace("\n", "\\n")}",
)
}
}
}
}
app/gradle/libs.versions.toml
+14 -5
View File
@@ -27,10 +27,6 @@ zxing-embedded = "4.3.0"
# https://repo1.maven.org/maven2/com/mikepenz/multiplatform-markdown-renderer-m3/maven-metadata.xml
# Latest stable there, checked 2026-08-29.
markdown-renderer = "0.45.0"
# Syntax highlighting for a tool call's input. Same reasoning as the markdown
# renderer: a language's lexical rules are somebody else's specification.
# Latest stable, checked 2026-08-31 against Maven Central.
highlights = "1.1.0"
# The support ExifInterface rather than android.media's, which lint warns off:
# the framework one is missing formats and the fixes for parsing hostile
# images, and images here arrive from outside the phone. Latest stable,
@@ -46,6 +42,12 @@ androidx-lifecycle = "2.11.0"
# itself is Kotlin-org owned and has almost nothing to configure, which is
# the point; this is the Gradle wrapper for it. Checked 2026-08-28.
ktfmt-gradle = "0.27.0"
# The test framework for the app's JVM unit tests (Highlighter.kt's scanner).
# JUnit 6 is the current line -- Jupiter and the Platform ship on one version
# now -- and the tests themselves are written against `kotlin.test`, so the
# framework is a build-file choice rather than something the source names.
# Latest stable, checked 2026-09-03 against Maven Central.
junit = "6.1.3"
# Backports java.time (and more) to API 24, which UsageScreen needs: its
# reset countdown is OffsetDateTime/Duration, both API 26. Checked
# 2026-08-28 against Google Maven.
@@ -64,8 +66,15 @@ desugar-jdk-libs = { module = "com.android.tools:desugar_jdk_libs", version.ref
# The -m3 flavour: it takes its colours and type from the ambient Material 3
# theme, so the app's Catppuccin scheme is what it draws with.
markdown-renderer = { module = "com.mikepenz:multiplatform-markdown-renderer-m3", version.ref = "markdown-renderer" }
highlights = { module = "dev.snipme:highlights", version.ref = "highlights" }
androidx-exifinterface = { module = "androidx.exifinterface:exifinterface", version.ref = "androidx-exifinterface" }
# The framework-neutral assertions the tests are written against. The `-junit5`
# artifact rather than plain `kotlin-test`: AGP 9 compiles Kotlin itself rather
# than through the Kotlin Android plugin, so nothing here resolves the variant
# of `kotlin-test` that carries `kotlin.test.Test` -- naming the artifact is
# what makes the annotation exist.
kotlin-test-junit5 = { module = "org.jetbrains.kotlin:kotlin-test-junit5", version.ref = "kotlin" }
junit-jupiter = { module = "org.junit.jupiter:junit-jupiter", version.ref = "junit" }
junit-platform-launcher = { module = "org.junit.platform:junit-platform-launcher", version.ref = "junit" }
# Declared directly rather than through the plugin's `compose.*` accessors,
# which are deprecated as of CMP 1.11.
compose-runtime = { module = "org.jetbrains.compose.runtime:runtime", version.ref = "compose-multiplatform" }
app/highlights-repro.sh
-59
View File
@@ -1,59 +0,0 @@
#!/usr/bin/env bash
# What dev.snipme:highlights actually answers for a piece of code, outside the app.
#
# The lexer's mistakes are invisible on a phone -- a line greyed out as a comment looks
# like a comment -- so this asks it directly and prints every span with the text under
# it. It is how the `/*`-pairing and URL-comment bugs in TRANSCRIPT_RENDERING.md were
# measured, and it is what any fix has to be checked against.
#
# Usage: ./highlights-repro.sh [LANGUAGE FILE] (default: the known-bad cases)
set -euo pipefail
cache=~/.gradle/caches/modules-2/files-2.1
jar() { find "$cache/$1" -name "$2" ! -name '*sources*' | sort | tail -1; }
cp=$(jar dev.snipme/highlights-jvm/1.1.0 '*.jar')
cp=$cp:$(jar org.jetbrains.kotlin/kotlin-stdlib '*[0-9].jar')
cp=$cp:$(jar org.jetbrains.kotlinx/kotlinx-coroutines-core-jvm '*.jar')
cp=$cp:$(jar org.jetbrains.kotlinx/kotlinx-serialization-core-jvm '*.jar')
cp=$cp:$(jar org.jetbrains.kotlinx/kotlinx-serialization-json-jvm '*.jar')
for entry in ${cp//:/ }; do
[ -f "$entry" ] || { echo "missing jar in the gradle cache: build the app once first" >&2; exit 1; }
done
out=$(mktemp -d)
trap 'rm -rf "$out"' EXIT
cat > "$out/Repro.java" <<'JAVA'
import dev.snipme.highlights.Highlights;
import dev.snipme.highlights.model.*;
import java.nio.file.*;
public class Repro {
public static void main(String[] args) throws Exception {
if (args.length == 2) {
report(SyntaxLanguage.valueOf(args[0].toUpperCase()), Files.readString(Path.of(args[1])));
return;
}
report(SyntaxLanguage.SHELL, "x '*/a/*'");
report(SyntaxLanguage.SHELL, "find . -path '*/.git/*' -prune -o -name '*.kt' -print");
report(SyntaxLanguage.SHELL, "curl https://example.com/x && echo done");
report(SyntaxLanguage.KOTLIN, "val url = \"https://example.com\"\nfun f() = 1");
}
static void report(SyntaxLanguage language, String code) {
System.out.println("== [" + language + "] " + code.replace("\n", "\\n"));
for (CodeHighlight highlight : new Highlights.Builder().code(code).language(language)
.build().getHighlights()) {
PhraseLocation at = highlight instanceof ColorHighlight
? ((ColorHighlight) highlight).getLocation()
: ((BoldHighlight) highlight).getLocation();
boolean sane = at.getStart() >= 0 && at.getStart() <= at.getEnd()
&& at.getEnd() <= code.length();
System.out.println((sane ? " " : " BAD ") + at + " "
+ (sane ? "\"" + code.substring(at.getStart(), at.getEnd()) + "\"" : ""));
}
}
}
JAVA
javac -cp "$cp" -d "$out" "$out/Repro.java"
java -cp "$cp:$out" Repro "$@"