diff --git a/AGENTS.md b/AGENTS.md index 004eb2c..442c77b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -220,8 +220,11 @@ first if a remote spawn ever mangles an argument. warning-clean and rustfmt-clean at the defaults — there is no `rustfmt.toml` and there should not be one. - App: from `app/`, `. ./android-env.sh && ./gradlew :androidApp:ktfmtFormat - :androidApp:compileDebugKotlin :androidApp:lintDebug` — format, typecheck - and lint, the app-side equivalent of the line above. Then `./build-apk.sh` + :androidApp:compileDebugKotlin :androidApp:lintDebug + :androidApp:testDebugUnitTest` — format, typecheck, lint and test, the + app-side equivalent of the line above. The unit tests are JVM-only and + cover the syntax highlighter's scanner, which is the app's one piece of + pure logic with no Android in it. Then `./build-apk.sh` to produce the APK to install on a phone (through Dev Updater), or `./run-android.sh` to build, install, and launch on the emulator. **The phone gets the release build**, signed with a key the script diff --git a/HIGHLIGHTER_PLAN.md b/HIGHLIGHTER_PLAN.md deleted file mode 100644 index 8acd0d6..0000000 --- a/HIGHLIGHTER_PLAN.md +++ /dev/null @@ -1,246 +0,0 @@ -# Replacing the code highlighter with a scanner of our own - -Written 2026-09-03 for whichever session does the work. Bryan decided on -this after the assessment below; `TRANSCRIPT_RENDERING.md` records the -measurements that led here and stays the record of the transcript work, -and `PLAN.md` stays the design source of truth for the app as a whole. -Nothing here contradicts either. Work goes on `main`, committed and pushed -when it is verified, as usual. - -## Why - -The app colours code in two places -- a fenced block in a reply and the -command in a tool call's card -- through one function, `highlight` in -`app/androidApp/src/main/kotlin/com/example/aiapp/CodeFence.kt`, backed by -the library `dev.snipme:highlights` 1.1.0. That library finds comments -before it knows the language, using `//`, `#` and `/* */` for every -language, then suppresses keywords and strings under whatever it found; -it pairs quotes by ordinal per quote character; and it pairs `/*` with -`*/` by ordinal too. Measured against the library directly on 2026-09-03: - -- Any language: `//` inside a URL comments out the rest of the line, and a - string holding a URL loses its colour inside the comment. -- Rust: every attribute line, `#[derive(Debug)]`, is greyed as a comment. -- Kotlin: a `#` inside a string (`"#FF0000"`) comments out the rest of the - line; an apostrophe inside a double-quoted string (`"don't"`) opens a - bogus single-quoted string that runs to the next apostrophe in the block. -- Shell: `x '*/a/*'` produced a span whose end preceded its start, which - crashed the app until `highlight` started dropping such spans. - -Upstream merged a one-line fix for the last of these in August 2026 and has -not released it; nothing else moved. Post-processing the library's output -cannot restore what it already suppressed, and forking it means rewriting -its analyzer, which is the part that is wrong. No maintained pure-Kotlin -alternative exists on Maven Central; tree-sitter would cost the NDK on -every machine that builds the APK and one to five megabytes per language -per ABI. So the highlighter becomes ours: one sequential scanner and a -table of languages. The rule in `prefer-libraries-for-specs` still holds -in general; this is the case it allows, where the only library is a worse -hand-rolled scanner than the one we would write. - -Counting fence words across the Claude Code transcripts on the dev VM -says which languages matter: shell in its variants 181, kotlin 119, rust -98, ron 67, then toml, python, java, js, groovy, json and diff at three to -eight each. RON, TOML and fish are not known to the library at all today, -so those fences are drawn plain; the scanner colours them from day one. - -## Shape - -Two new files under `app/androidApp/src/main/kotlin/com/example/aiapp/`, -and the library removed. - -**`Highlighter.kt`** holds the scanner, the palette type, and `highlight`. -The signature stays `fun highlight(code: String, language: Language?): -AnnotatedString`, returning plain text for a null language, so the two -callers and the cache do not change shape. It is not a composable and -takes no colour from the theme, for the reason `CodeFence.kt` already -states: `ParsedReplies.warm` runs it off the drawing thread. The palette -is a plain `SyntaxPalette` value class of our own -- keyword, string, -literal, comment, metadata, punctuation, mark, as `Color` -- returned by -`catppuccinSyntax()` in `Theme.kt`, which currently builds the library's -`SyntaxTheme` from ARGB ints. The library's separate multiline-comment -colour collapses into comment; the palette already gave both `Overlay0`. - -The scanner is one pass over the code with a small state: at each -position it is in a line comment, a block comment (with a nesting depth -where the language nests them), a string (with its closing delimiter and -whether backslash escapes apply), or ordinary code. In ordinary code it -recognises, in this order, a comment opener, a string opener, an -attribute, a number, a word, and finally single punctuation or mark -characters. Every span it emits is produced by advancing an index -forward, so spans are non-overlapping and ordered by construction and the -reversed-range filter goes with the library. Spans map onto the palette -as: keyword, string, literal (numbers), comment, metadata (attributes and -annotations), punctuation (`, . : ;`) and mark (`( ) = { } < > - + [ ] | -&`). Those two character sets are the library's, kept so nothing the -reader is used to changes colour except where it was wrong; punctuation -and marks are only coloured in ordinary code, never inside a string or a -comment, which is the one visible difference on correct input (a `.` in a -URL inside a string used to take the punctuation colour). - -A word is `[A-Za-z_][A-Za-z0-9_]*` and is a keyword if the language's set -contains it. A number is a word starting with a digit and continuing -through letters, digits, `_` and `.`, which covers `0xFF`, `1_000`, -`1u32` and `3.14` without a grammar. A date in TOML comes out as digits -with marks between; leave that, it is untidy rather than misleading. - -**`Languages.kt`** holds `enum class Language` and a `Rules` value per -member. Keep the vocabulary the code already uses: the enum replaces the -library's `SyntaxLanguage` under the name `Language`, and `fenceLanguage` -in `CodeFence.kt` keeps its name and its alias table, retargeted. `Rules` -is data, not code, and is what makes a new language a table row: - -- `lineComments`: the tokens that open a comment to end of line, and - whether they count only at the start of a word. Shell and fish need - the word-start rule, since `$#`, `${#x}` and `a#b` are not comments; a - `#` counts there when it is preceded by nothing, whitespace, or one of - `;`, `|`, `&`, `(`. Python, Ruby, Perl, TOML: `#` anywhere. C-family, - Kotlin, Rust, RON, Go, Swift, Dart, JavaScript, TypeScript, Java, - C#, PHP: `//`; PHP also `#`. -- `blockComment`: the pair, or none, and whether it nests. `/* */` for - the C-family and friends; Rust, Kotlin and RON nest; C, Java, - JavaScript, Go do not. Python, Ruby, shell, fish, TOML: none. -- `quotes`: each string form as opener, closer, and whether backslash - escapes the closer. Double quotes with escapes nearly everywhere; - single quotes with escapes in the C-family (a character literal reads - fine as a string); single quotes with *no* escapes in shell and TOML; - fish single quotes escape only `\'` and `\\`; triple quotes in Kotlin - (`"""`, no escapes), Python (`"""` and `'''`) and TOML (`"""` and - `'''`). Longest opener wins, so `"""` is tried before `"`. -- Two string forms need a line of code rather than a row, and belong in - the scanner behind a flag on `Rules`. Rust and RON raw strings: - optional `b`, then `r`, then `n` hashes, then `"`, closing at `"` - followed by `n` hashes, no escapes. Rust lifetimes: a `'` followed by - a backslash or by one character and a `'` is a character literal; - otherwise it is a lifetime or label and no string starts. Without that - rule `'a` opens a string that runs to the next apostrophe. -- `attributes`: what opens a metadata span. `@` followed by a word for - Kotlin, Java, Dart, Swift, TypeScript, PHP, Python (decorators); - `#[` or `#![` through the matching `]` for Rust and RON; a `#` at the - start of a line through end of line for C and C++ (preprocessor - directives, which the library greyed out as comments); a `[` at the - start of a line through `]` for TOML (table headers). Nothing for the - rest. -- `keywords`: the set. Bring the lists over from the library's - `SyntaxTokens.kt` (Apache-2.0; say so in a comment at the table), for - every language the alias table names today, so no fence that is - coloured now turns plain. Drop entries that are not plain words -- - Kotlin's `as?`, `!in`, `!is`; Swift's `#if` family; Ruby's `defined?`; - CoffeeScript's `=` and `->` -- since the word scanner cannot reach - them and the library only matched them by luck. Add rows for RON - (`true false Some None inf NaN`), TOML (`true false inf nan`), fish - (`and begin break builtin case command continue else end exec for - function if in not or return switch while set echo test string math - read source`) and JSON (`true false null`). Add `ron`, `toml`, `fish` - and `json` to the alias table in `CodeFence.kt`. - -The `streaming` arrangement stays as it is: a fence still being written -is drawn plain and coloured when it freezes. That decision was made -against a lexer costing 174ms for two hundred lines; a linear scanner may -make live colouring affordable, but that is a follow-up to measure with -`app/stream-bench.sh`, not something to assume here. - -## Files - -- `Highlighter.kt`, new: scanner, `SyntaxPalette`, `highlight`. The - `DebugStats.timed("code highlighted")` wrapper stays, as the number the - before-and-after is read from. -- `Languages.kt`, new: `Language`, `Rules`, the rows and keyword sets. -- `CodeFence.kt`: remove `highlight` and the library imports; retarget - `fenceLanguage` and `FENCE_LANGUAGES`; add the four aliases. -- `ToolInput.kt`: `SyntaxLanguage` becomes `Language` in `ToolInput` and - `SUBJECTS`. `ToolInputView` keeps calling `highlight` uncached. -- `Markdown.kt`: the import at line 58 and the signature of - `ParsedReplies.highlighted`. The cache key `"$language\n$code"` works - unchanged with an enum. -- `Theme.kt`: `catppuccinSyntax()` returns `SyntaxPalette` of `Color`s - rather than the library's theme of ints. -- `app/gradle/libs.versions.toml` and `app/androidApp/build.gradle.kts`: - remove `highlights` and its comment. -- `app/highlights-repro.sh`: delete. It exists only to interrogate the - library, and its cases move into the test below; this is the harness - being replaced by a better one, not thrown away. -- `app/androidApp/src/test/kotlin/com/example/aiapp/HighlighterTest.kt`, - new: see next section. -- Docs, in the same commit: in `TRANSCRIPT_RENDERING.md` rewrite the - fences paragraph's mention of the library, replace the "highlights - 1.1.0 finds comments before it knows the language" bullet with a short - description of the scanner and where its rows live, and drop item 2 of - "What is next". In `AGENTS.md`, add `:androidApp:testDebugUnitTest` to - the app line under "Checking your work". Prune rather than append. - -## Tests - -The app has no unit test source set today. The scanner is pure logic -with no Android imports, which is exactly the case CODE_RULES names for -adding tests, and the cost of the harness is one line, -`testImplementation(kotlin("test"))`, run by -`./gradlew :androidApp:testDebugUnitTest`. Keep `Highlighter.kt` and -`Languages.kt` free of Android and Compose imports other than -`AnnotatedString`, `SpanStyle` and `Color` (which are plain JVM classes), -so the test runs on the JVM without Robolectric. Better still, have the -scanner return a list of (range, kind) spans and let `highlight` be the -thin wrapper that turns them into an `AnnotatedString`: the test then -asserts on the list, which prints legibly on failure, and the test does -not depend on Compose loading at all. - -Cases, each asserting the spans against the code so a failure prints -what was coloured: - -- The four inputs from the deleted script: `x '*/a/*'` (shell), the - `find . -path '*/.git/*'` line, `curl https://example.com/x && echo - done`, and `val url = "https://example.com"\nfun f() = 1`. -- `#[derive(Debug)]` then a struct in Rust: metadata, then keywords. -- `"#FF0000"` and `"don't"` in Kotlin: two strings, no comment. -- `'a` as a Rust lifetime beside `'x'` as a character: one string. -- `r#"a "quoted" b"#` in Rust: one string. -- Nested `/* a /* b */ c */` in Rust and Kotlin: one comment to the end; - the same in C: comment ends at the first `*/`. -- `${#x} $# a#b # real` in shell: one comment, starting at the last `#`. -- `"""a "b" c"""` in Kotlin: one string. -- A TOML table header and a key with a `#` in a quoted value. -- A language the alias table does not know: `highlight` returns the - plain string. -- Every emitted span is inside the code and starts before it ends, over - a handful of nasty inputs (unterminated string, unterminated block - comment, a lone `'`, empty code). The scanner must never throw; an - unterminated thing runs to the end of the code. - -## Verification - -1. From `app/`: `. ./android-env.sh && ./gradlew :androidApp:ktfmtFormat - :androidApp:compileDebugKotlin :androidApp:lintDebug - :androidApp:testDebugUnitTest`. Lint must stay clean; an unused - dependency or import it reports is a finding. -2. Timing. `TRANSCRIPT_RENDERING.md` gives the library at 174ms for a - two-hundred-line Kotlin fence on the emulator. Regenerate - `/tmp/longfence.md` from that description and read `code highlighted` - from the render report after `app/transcript-bench.sh`; expect single - milliseconds. Run `app/transcript-bench.sh` and `app/stream-bench.sh` - before and after, and put the report lines named in that doc's item 3 - into the commit message. -3. Look at it. `app/run-android.sh`, send `/tmp/fixture.md` through - `app/ui-sandbox.sh` (both described in `TRANSCRIPT_RENDERING.md`), and - screenshot a fence holding a URL in a string, a Rust attribute, a - shell line with a glob and a `$#`, a RON fence, and a TOML fence. Check - a tool card's Bash command still colours, since that is the other - caller. Check a fence in an unknown language is plain, and that a - fence still streaming is plain and colours when the next block starts. - Per the code rule about trying a fix on the half that cannot fail, - look at a fence the change had no reason to touch -- a plain Python - function -- and confirm it looks as it did. -4. Commit on `main` with `git add -A`, and push. - -## Traps - -- `fenceContent` is shared by the drawing and the warming and must stay - the single extraction; nothing here should touch it. -- The `Color` constructor in `highlight` currently ORs in an alpha byte - because the library's ints had none. With `Color` values in the palette - that goes away; do not carry the mask over. -- Keyword lists are per language in the library's `DEFAULT` case merged - into one; there is no `DEFAULT` here, since a null language is plain. -- The two callers pass different things: fences pass what the alias - table answered for the info word, tool cards pass `SHELL` for Bash and - null for everything else. Both must keep compiling without a cast. -- ktfmt will reflow the keyword tables; let it. diff --git a/TRANSCRIPT_RENDERING.md b/TRANSCRIPT_RENDERING.md index 541d24f..3d9b11f 100644 --- a/TRANSCRIPT_RENDERING.md +++ b/TRANSCRIPT_RENDERING.md @@ -107,17 +107,18 @@ time went from 2412ms of reparsing to 674ms, and `record: one block` from 1.8ms worst to 0.7ms. **Fences are highlighted off the drawing thread, and a fence still being -written is drawn plain.** `CodeFence.kt` holds `highlight` (shared with a -tool call's input, so the same code is the same colours wherever it -appears), the `fenceLanguage` alias table, and `fenceContent`. A word not in -the table stays plain, because a fence coloured by the wrong language's -rules looks highlighted and is wrong in a way the reader cannot see. -Highlighting is warmed and cached exactly as parsing is -(`ParsedReplies.highlighted`, filled by `warm` from `fences(parse)`), and -`highlight` takes no colour from the theme, which is what lets it run off -the drawing thread: a two-hundred-line Kotlin fence costs 174ms to lex, and -a `remember` inside the fence was charged that again every time the block -scrolled back into composition. Because the warming has to ask for the same +written is drawn plain.** `Highlighter.kt` holds `highlight` and the scanner +behind it (shared with a tool call's input, so the same code is the same +colours wherever it appears); `CodeFence.kt` holds the `fenceLanguage` alias +table and `fenceContent`. A word not in the table stays plain, because a +fence coloured by the wrong language's rules looks highlighted and is wrong +in a way the reader cannot see. Highlighting is warmed and cached exactly as +parsing is (`ParsedReplies.highlighted`, filled by `warm` from +`fences(parse)`), and `highlight` takes no colour from the theme, which is +what lets it run off the drawing thread: a two-hundred-line Kotlin fence +costs 15ms to scan on the emulator's debug build -- it cost 102ms through +the library that used to do this -- and a `remember` inside the fence was +charged that again every time the block scrolled back into composition. Because the warming has to ask for the same string the drawing does, `fenceContent` extracts the code and the language word itself -- two extractions would be two keys, and the warmed answer would be missed at every fence with nothing saying so. A fence still @@ -209,25 +210,29 @@ and are the reason several tempting simplifications were rejected. - **Compose `DropdownMenu` in an edge-to-edge activity** needs `PopupProperties(clippingEnabled = false)` or it opens a status bar's height away from its anchor (`~/.claude/TOOLCHAIN.md`). -- **highlights 1.1.0 finds comments before it knows the language, and pairs - `/*` with `*/` by ordinal.** Measured against the library directly with - `app/highlights-repro.sh`, which asks it for a piece of code outside the - app and prints every span with the text under it -- the mistakes are - invisible on a phone, where a line greyed out as a comment looks like a - comment. In `MultilineCommentLocator` and `CommentLocator`: it collects every `/*` and - every `*/` in the code, zips the two lists by position and never checks - that the end follows the start, so `x '*/a/*'` yields `start=6, end=5` -- - a range `AnnotatedString` rejects, which crashed a card holding - `-path '*/.git/*'` until `highlight` started dropping such spans. The same - delimiters are used for every language, so a shell glob is read as a - comment opener and, worse, `//` in any URL comments out the rest of its - line: in `curl https://example.com/x && echo done` the comment runs to the - end and takes `echo` with it, and in Kotlin `val url = "https://..."` the - string span disappears inside it. Comments are located before strings and - win over them. What we can reach is `getCodeStructure()`, which is public, - plus a nine-line reconstruction of the library's private - `constructHighlights`; the locators themselves are `internal`. 1.1.0 is - the newest release, so there is nothing to upgrade to. +- **The syntax highlighter is ours: `Highlighter.kt` and `Languages.kt`.** + One left-to-right scanner with a small state -- in a line comment, in a + block comment, in a string, or in ordinary code -- and a `Rules` row per + language, so a new language is a table entry rather than code. Every span + is emitted by advancing an index, so spans cannot overlap, arrive out of + order or run backwards, and an unterminated string or comment simply runs + to the end of the code. `HighlighterTest.kt` is the JVM unit test + (`./gradlew :androidApp:testDebugUnitTest`); the cases in it are the + library's mistakes, kept as regressions. + It replaced dev.snipme:highlights 1.1.0 on 2026-09-03, which found + comments before it knew the language and paired `/*` with `*/` by + ordinal. That library used one set of delimiters for every language, so + `//` in any URL commented out the rest of its line (in `curl + https://example.com/x && echo done` the comment ran to the end and took + `echo` with it, and in Kotlin `val url = "https://..."` the string + disappeared inside it), every Rust `#[derive(...)]` greyed out as a + comment, a `#` inside a Kotlin string swallowed the line, and `x '*/a/*'` + in shell yielded `start=6, end=5` -- a range `AnnotatedString` rejects, + which crashed a card holding `-path '*/.git/*'`. Comments were located + before strings and won over them, so post-processing could not recover + what a wrong comment range had already suppressed. The scanner is also + about seven times faster on the same fixture, and it colours RON, TOML, + fish and JSON, which the library did not know at all. ## Rejected, and why @@ -254,19 +259,7 @@ and are the reason several tempting simplifications were rejected. against. The restore's one-event-per-request bug was part of what made it so visible and has been fixed; whether this survives that fix is the first thing to find out. -2. **Decide what to do about the highlighting bug above.** Two options, both - ours: post-process `getCodeStructure()` -- pair `/*` with the next `*/` - after it, and only for languages that have them, ignore `#` and `//` - where the language does not use them -- then build the annotated string - from the corrected structure, which costs no extra lexing but cannot - recover keywords and strings the wrong comment range already suppressed; - or fork the twenty-file library and fix the locators. Filing it upstream - needs Bryan or a token, since there is no `gh` and no GitHub credential - in this VM. One-line repro: lexing `x '*/a/*'` as `SyntaxLanguage.SHELL` - in highlights 1.1.0 returns a highlight whose `location.end` precedes its - `location.start`. Whichever is chosen, `app/highlights-repro.sh` is what - checks it. -3. **Regression runs.** `transcript-bench.sh` and `stream-bench.sh` before +2. **Regression runs.** `transcript-bench.sh` and `stream-bench.sh` before and after any change to the files above, with the report in the commit. The numbers to watch are the worst `record: one block`, the reparse mean while streaming, and the draw phase's accounting line. diff --git a/app/androidApp/build.gradle.kts b/app/androidApp/build.gradle.kts index c016cac..9b5d6c9 100644 --- a/app/androidApp/build.gradle.kts +++ b/app/androidApp/build.gradle.kts @@ -173,6 +173,15 @@ dependencies { implementation(libs.androidx.lifecycle.runtime.compose) implementation(libs.zxing.embedded) implementation(libs.markdown.renderer) - implementation(libs.highlights) implementation(libs.androidx.exifinterface) + + // The syntax scanner (Highlighter.kt) is pure logic with no Android imports, which is what + // lets it be tested on the JVM: `./gradlew :androidApp:testDebugUnitTest`. The assertions are + // `kotlin.test`, so the tests name no framework; JUnit is what runs them. + testImplementation(libs.kotlin.test.junit5) + testImplementation(libs.junit.jupiter) + testRuntimeOnly(libs.junit.platform.launcher) } + +// JUnit 6 runs on the Platform, which is not Gradle's default for a Test task. +tasks.withType().configureEach { useJUnitPlatform() } diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/CodeFence.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/CodeFence.kt index 344ef46..0e9cfe9 100644 --- a/app/androidApp/src/main/kotlin/com/example/aiapp/CodeFence.kt +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/CodeFence.kt @@ -11,23 +11,14 @@ import androidx.compose.foundation.text.BasicText import androidx.compose.runtime.Composable import androidx.compose.runtime.remember import androidx.compose.ui.Modifier -import androidx.compose.ui.graphics.Color import androidx.compose.ui.semantics.isTraversalGroup import androidx.compose.ui.semantics.semantics -import androidx.compose.ui.text.AnnotatedString -import androidx.compose.ui.text.SpanStyle import androidx.compose.ui.text.TextStyle -import androidx.compose.ui.text.buildAnnotatedString -import androidx.compose.ui.text.font.FontWeight import androidx.compose.ui.unit.dp import com.mikepenz.markdown.compose.LocalMarkdownColors import com.mikepenz.markdown.compose.LocalMarkdownDimens import com.mikepenz.markdown.compose.LocalMarkdownPadding import com.mikepenz.markdown.model.State -import dev.snipme.highlights.Highlights -import dev.snipme.highlights.model.BoldHighlight -import dev.snipme.highlights.model.ColorHighlight -import dev.snipme.highlights.model.SyntaxLanguage import org.intellij.markdown.MarkdownElementTypes import org.intellij.markdown.MarkdownTokenTypes import org.intellij.markdown.ast.ASTNode @@ -38,10 +29,10 @@ import org.intellij.markdown.ast.getTextInNode * A fenced code block in a reply: the code highlighted, on the dark surface every verbatim thing * sits on, scrolling sideways rather than wrapping. * - * The renderer's own fence drew the same block in plain text. The lexer that colours a tool call's - * command colours a reply's code the same way, through [highlighted] and one theme, so a `kotlin` - * fence and the Kotlin a tool wrote are the same colours. A fence in a language the lexer has no - * rules for is plain rather than wrongly coloured: [fenceLanguage] answers null for those, and + * The renderer's own fence drew the same block in plain text. The scanner that colours a tool + * call's command colours a reply's code the same way, through [highlighted] and one palette, so a + * `kotlin` fence and the Kotlin a tool wrote are the same colours. A fence in a language [scan] has + * no rules for is plain rather than wrongly coloured: [fenceLanguage] answers null for those, and * plain is what the reader would have seen before. * * Finding the code is still the library's: which children of the node are the fence markers, the @@ -74,7 +65,7 @@ fun CodeBlock( } /** - * The code inside a fence or indented block, and the lexer's language for its info word. + * The code inside a fence or indented block, and the highlighter's language for its info word. * * Which children of the node are the fence markers, the language word and the code between them is * the library's knowledge of the parser, copied from its `MarkdownCodeFence` rather than called: @@ -85,7 +76,7 @@ fun CodeBlock( * Null for a fence too short to hold anything -- an unterminated one still arriving, which the * library skips as invalid. */ -fun fenceContent(content: String, node: ASTNode): Pair? { +fun fenceContent(content: String, node: ASTNode): Pair? { val word = node.findChildOfType(MarkdownTokenTypes.FENCE_LANG)?.getTextInNode(content)?.toString() val language = fenceLanguage(word) @@ -111,7 +102,7 @@ fun fenceContent(content: String, node: ASTNode): Pair? @Composable private fun CodeBlockText( code: String, - language: SyntaxLanguage?, + language: Language?, style: TextStyle, replies: ParsedReplies, streaming: Boolean, @@ -136,122 +127,72 @@ private fun CodeBlockText( } /** - * The lexer's language for a fence's info word, or null for one it has no lexer for. + * The highlighter's language for a fence's info word, or null for one it has no rules for. * * The aliases are what people actually write after the backticks: the file extension as often as - * the name. A word not here gets no colour rather than the nearest lexer's, because a fence + * the name. A word not here gets no colour rather than the nearest language's, because a fence * coloured by the wrong language's rules looks highlighted and is wrong in a way the reader cannot * see. */ -fun fenceLanguage(name: String?): SyntaxLanguage? = +fun fenceLanguage(name: String?): Language? = FENCE_LANGUAGES[name?.trim()?.lowercase() ?: return null] -private val FENCE_LANGUAGES: Map = +private val FENCE_LANGUAGES: Map = mapOf( - "kotlin" to SyntaxLanguage.KOTLIN, - "kt" to SyntaxLanguage.KOTLIN, - "kts" to SyntaxLanguage.KOTLIN, - "rust" to SyntaxLanguage.RUST, - "rs" to SyntaxLanguage.RUST, - "sh" to SyntaxLanguage.SHELL, - "bash" to SyntaxLanguage.SHELL, - "shell" to SyntaxLanguage.SHELL, - "zsh" to SyntaxLanguage.SHELL, - "console" to SyntaxLanguage.SHELL, - "python" to SyntaxLanguage.PYTHON, - "py" to SyntaxLanguage.PYTHON, - "javascript" to SyntaxLanguage.JAVASCRIPT, - "js" to SyntaxLanguage.JAVASCRIPT, - "jsx" to SyntaxLanguage.JAVASCRIPT, - "typescript" to SyntaxLanguage.TYPESCRIPT, - "ts" to SyntaxLanguage.TYPESCRIPT, - "tsx" to SyntaxLanguage.TYPESCRIPT, - "java" to SyntaxLanguage.JAVA, - "c" to SyntaxLanguage.C, - "h" to SyntaxLanguage.C, - "cpp" to SyntaxLanguage.CPP, - "c++" to SyntaxLanguage.CPP, - "cc" to SyntaxLanguage.CPP, - "hpp" to SyntaxLanguage.CPP, - "csharp" to SyntaxLanguage.CSHARP, - "cs" to SyntaxLanguage.CSHARP, - "c#" to SyntaxLanguage.CSHARP, - "go" to SyntaxLanguage.GO, - "golang" to SyntaxLanguage.GO, - "swift" to SyntaxLanguage.SWIFT, - "dart" to SyntaxLanguage.DART, - "ruby" to SyntaxLanguage.RUBY, - "rb" to SyntaxLanguage.RUBY, - "php" to SyntaxLanguage.PHP, - "perl" to SyntaxLanguage.PERL, - "pl" to SyntaxLanguage.PERL, - "coffeescript" to SyntaxLanguage.COFFEESCRIPT, - "coffee" to SyntaxLanguage.COFFEESCRIPT, + "kotlin" to Language.KOTLIN, + "kt" to Language.KOTLIN, + "kts" to Language.KOTLIN, + "rust" to Language.RUST, + "rs" to Language.RUST, + "sh" to Language.SHELL, + "bash" to Language.SHELL, + "shell" to Language.SHELL, + "zsh" to Language.SHELL, + "console" to Language.SHELL, + "python" to Language.PYTHON, + "py" to Language.PYTHON, + "javascript" to Language.JAVASCRIPT, + "js" to Language.JAVASCRIPT, + "jsx" to Language.JAVASCRIPT, + "typescript" to Language.TYPESCRIPT, + "ts" to Language.TYPESCRIPT, + "tsx" to Language.TYPESCRIPT, + "java" to Language.JAVA, + "c" to Language.C, + "h" to Language.C, + "cpp" to Language.CPP, + "c++" to Language.CPP, + "cc" to Language.CPP, + "hpp" to Language.CPP, + "csharp" to Language.CSHARP, + "cs" to Language.CSHARP, + "c#" to Language.CSHARP, + "go" to Language.GO, + "golang" to Language.GO, + "swift" to Language.SWIFT, + "dart" to Language.DART, + "ruby" to Language.RUBY, + "rb" to Language.RUBY, + "php" to Language.PHP, + "perl" to Language.PERL, + "pl" to Language.PERL, + "coffeescript" to Language.COFFEESCRIPT, + "coffee" to Language.COFFEESCRIPT, + "ron" to Language.RON, + "toml" to Language.TOML, + "fish" to Language.FISH, + "json" to Language.JSON, ) -/** - * [code] with its keywords and strings coloured, or plain if there is no language for it. - * - * The lexing is dev.snipme:highlights. The colours are this app's, mapped in [catppuccinSyntax] -- - * a library's default theme would be the one place in the app whose palette came from somewhere - * else. Shared by a tool call's input ([ToolInputView]) and a reply's fences ([CodeFence]), so the - * same code is the same colours wherever it appears. - * - * Not a composable, and it takes no colour from the theme, because that is what lets [warm] run it - * off the drawing thread: the syntax palette is fixed, and a fence with no language is plain text - * which needs no colour of its own -- the style the caller draws it with carries that. - * - * Measured on the emulator before it was cached: a two-hundred-line Kotlin fence costs **174ms** to - * lex, and the lazy list charged that again every time the block scrolled back into composition. - * That is the whole reason [ParsedReplies.highlighted] exists rather than a `remember`. - */ -fun highlight(code: String, language: SyntaxLanguage?): AnnotatedString { - if (language == null) return AnnotatedString(code) - val marks = - DebugStats.timed("code highlighted") { - Highlights.Builder(code = code, language = language, theme = catppuccinSyntax()) - .build() - .getHighlights() - // highlights 1.1.0's shell lexer answers a quoted glob that looks like a comment - // -- `x '*/a/*'` is the smallest input -- with a span whose end is before its - // start, and AnnotatedString refuses such a range. That crashed the app the - // moment a card holding `-path '*/.git/*'` was opened. Dropped rather than - // clamped: a span the lexer got backwards is not one it knows the colour of. - // Delete when snipme/highlights fixes it. - .filter { - it.location.start in 0..it.location.end && it.location.end <= code.length - } - } - return buildAnnotatedString { - append(code) - marks.forEach { mark -> - when (mark) { - is ColorHighlight -> - addStyle( - SpanStyle(color = Color(mark.rgb or 0xFF000000.toInt())), - mark.location.start, - mark.location.end, - ) - is BoldHighlight -> - addStyle( - SpanStyle(fontWeight = FontWeight.Bold), - mark.location.start, - mark.location.end, - ) - } - } - } -} - /** * Every fence in [parse], as the code and language [highlight] will be asked for. * * Walks the whole tree rather than the top level: a fence inside a list item or a quote is drawn * the same way and costs the same to lex. */ -fun fences(parse: State): List> { +fun fences(parse: State): List> { val success = parse as? State.Success ?: return emptyList() - val out = ArrayList>() + val out = ArrayList>() fun walk(node: ASTNode) { if ( node.type == MarkdownElementTypes.CODE_FENCE || diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/Highlighter.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/Highlighter.kt new file mode 100644 index 0000000..e5122bb --- /dev/null +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/Highlighter.kt @@ -0,0 +1,318 @@ +package com.example.aiapp + +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.text.AnnotatedString +import androidx.compose.ui.text.SpanStyle +import androidx.compose.ui.text.buildAnnotatedString + +/** What a span of code is, in the terms the palette has a colour for. */ +enum class Kind { + KEYWORD, + STRING, + LITERAL, + COMMENT, + METADATA, + PUNCTUATION, + MARK, +} + +/** A run of [Kind] in the code, as a half-open range. */ +data class Span(val start: Int, val end: Int, val kind: Kind) + +/** + * The colours the highlighter draws with, ours rather than a library's; [catppuccinSyntax] is the + * one instance and lives with the rest of the palette. + */ +data class SyntaxPalette( + val keyword: Color, + val string: Color, + val literal: Color, + val comment: Color, + val metadata: Color, + val punctuation: Color, + val mark: Color, +) { + fun of(kind: Kind): Color = + when (kind) { + Kind.KEYWORD -> keyword + Kind.STRING -> string + Kind.LITERAL -> literal + Kind.COMMENT -> comment + Kind.METADATA -> metadata + Kind.PUNCTUATION -> punctuation + Kind.MARK -> mark + } +} + +/** + * [code] with its keywords, strings and comments coloured, or plain if there is no language for it. + * + * Shared by a tool call's input ([ToolInputView]) and a reply's fences ([CodeFence]), so the same + * code is the same colours wherever it appears. + * + * Not a composable, and it takes no colour from the theme, because that is what lets [warm] run it + * off the drawing thread: the syntax palette is fixed, and a fence with no language is plain text + * which needs no colour of its own -- the style the caller draws it with carries that. + * + * The timing is the number the highlighter is judged by: the library this replaced took **174ms** + * on the emulator for a two-hundred-line Kotlin fence, which is why [ParsedReplies.highlighted] + * caches the answer rather than a `remember` inside the fence recomputing it on every scroll back. + */ +fun highlight(code: String, language: Language?): AnnotatedString { + if (language == null) return AnnotatedString(code) + val spans = DebugStats.timed("code highlighted") { scan(code, rulesOf(language)) } + val palette = catppuccinSyntax() + return buildAnnotatedString { + append(code) + spans.forEach { addStyle(SpanStyle(color = palette.of(it.kind)), it.start, it.end) } + } +} + +/** + * [code] read once, left to right, into the spans that carry a colour. + * + * One pass with a small state -- in a comment, in a string, or in ordinary code -- rather than a + * locator per token kind over the whole text, which is what the library did and is why it found + * comments before it knew the language: a `#` inside a shell string, a `//` inside a URL and a + * block-comment opener inside a shell glob each commented out the rest of a line that was nothing + * of the sort. + * + * Every span is produced by advancing an index forward, so the result is ordered, non-overlapping + * and inside the code by construction. Nothing here throws: an unterminated string or comment runs + * to the end of the code, which is also what it looks like while a fence is still being written. + * + * In ordinary code the order of recognition is comment, string, attribute, number, word, and + * finally a single punctuation or mark character. Punctuation and marks are coloured only in + * ordinary code, never inside a string or a comment. + */ +fun scan(code: String, rules: Rules): List = Scanner(code, rules).run() + +/** Characters coloured as punctuation, and as marks. Both sets are the ones the library used. */ +private const val PUNCTUATION = ",.:;" +private const val MARKS = "()={}<>-+[]|&" + +private class Scanner(private val code: String, private val rules: Rules) { + private val spans = ArrayList() + private var at = 0 + + fun run(): List { + while (at < code.length) { + // Every branch that answers true has advanced `at`, so this terminates. + val consumed = + blockComment() || + lineComment() || + rawString() || + characterOrLifetime() || + string() || + attribute() || + number() || + word() || + singleCharacter() + if (!consumed) at++ + } + return spans + } + + private fun emit(start: Int, kind: Kind) { + if (at > start) spans.add(Span(start, at, kind)) + } + + private fun starts(token: String) = code.startsWith(token, at) + + /** Whether a line comment token here opens one; see [Rules.lineCommentsAtWordStart]. */ + private fun atWordStart() = at == 0 || code[at - 1].isWhitespace() || code[at - 1] in ";|&(" + + /** Whether only whitespace stands between the start of this line and here. */ + private fun atLineStart(): Boolean { + var back = at - 1 + while (back >= 0 && code[back] != '\n') { + if (!code[back].isWhitespace()) return false + back-- + } + return true + } + + private fun toEndOfLine() { + while (at < code.length && code[at] != '\n') at++ + } + + /** From an open bracket through the one that matches it, or to the end if none does. */ + private fun toMatchingBracket() { + var depth = 0 + while (at < code.length) { + when (code[at]) { + '[' -> depth++ + ']' -> depth-- + } + at++ + if (depth == 0) return + } + } + + private fun blockComment(): Boolean { + val comment = rules.blockComment ?: return false + if (!starts(comment.open)) return false + val start = at + at += comment.open.length + var depth = 1 + while (at < code.length && depth > 0) { + // The closer is tried first so that a language whose two delimiters are the same + // string -- CoffeeScript's `###` -- closes rather than nesting forever. + if (starts(comment.close)) { + depth-- + at += comment.close.length + } else if (comment.nests && starts(comment.open)) { + depth++ + at += comment.open.length + } else { + at++ + } + } + emit(start, Kind.COMMENT) + return true + } + + private fun lineComment(): Boolean { + if (rules.lineComments.none { starts(it) }) return false + if (rules.lineCommentsAtWordStart && !atWordStart()) return false + val start = at + toEndOfLine() + emit(start, Kind.COMMENT) + return true + } + + /** Rust and RON: `b`? `r` `#`* `"` … `"` `#`*, with no escapes inside. */ + private fun rawString(): Boolean { + if (!rules.rawStrings) return false + var ahead = at + if (code.getOrNull(ahead) == 'b') ahead++ + if (code.getOrNull(ahead) != 'r') return false + ahead++ + var hashes = 0 + while (code.getOrNull(ahead) == '#') { + ahead++ + hashes++ + } + if (code.getOrNull(ahead) != '"') return false + val start = at + val closer = "\"" + "#".repeat(hashes) + val closed = code.indexOf(closer, ahead + 1) + at = if (closed < 0) code.length else closed + closer.length + emit(start, Kind.STRING) + return true + } + + /** See [Rules.lifetimes]: an apostrophe that is not a character literal opens nothing. */ + private fun characterOrLifetime(): Boolean { + if (!rules.lifetimes || code[at] != '\'') return false + val next = code.getOrNull(at + 1) ?: return false + if (next == '\\' || code.getOrNull(at + 2) == '\'') { + quoted(Quote("'", "'", escapes = true)) + } else { + at++ + } + return true + } + + private fun string(): Boolean { + // Longest opener wins, so Kotlin's `"""` is one delimiter rather than an empty string + // followed by a quote. A loop rather than filter/maxBy: this runs at every character of + // ordinary code, and the pair of lists that would allocate is the whole cost of the scan. + var quote: Quote? = null + for (candidate in rules.quotes) { + if (starts(candidate.open) && candidate.open.length > (quote?.open?.length ?: 0)) { + quote = candidate + } + } + quoted(quote ?: return false) + return true + } + + private fun quoted(quote: Quote) { + val start = at + at += quote.open.length + while (at < code.length) { + if (quote.escapes && code[at] == '\\' && at + 1 < code.length) { + at += 2 + continue + } + if (starts(quote.close)) { + at += quote.close.length + break + } + at++ + } + at = at.coerceAtMost(code.length) + emit(start, Kind.STRING) + } + + private fun attribute(): Boolean { + val start = at + when (rules.attributes) { + Attributes.NONE -> return false + Attributes.AT_WORD -> { + if (code[at] != '@' || !isWordStart(code.getOrNull(at + 1))) return false + at++ + while (at < code.length && isWordPart(code[at])) at++ + } + Attributes.HASH_BRACKET -> { + if (code[at] != '#') return false + var ahead = at + 1 + if (code.getOrNull(ahead) == '!') ahead++ + if (code.getOrNull(ahead) != '[') return false + at = ahead + toMatchingBracket() + } + Attributes.HASH_LINE -> { + if (code[at] != '#' || !atLineStart()) return false + toEndOfLine() + } + Attributes.LINE_BRACKET -> { + if (code[at] != '[' || !atLineStart()) return false + toMatchingBracket() + } + } + emit(start, Kind.METADATA) + return true + } + + /** + * A number is a run starting with a digit and carrying on through letters, digits, `_` and `.` + * -- which covers `0xFF`, `1_000`, `1u32` and `3.14` without a grammar for any of them. + */ + private fun number(): Boolean { + if (!code[at].isDigit()) return false + val start = at + while ( + at < code.length && (code[at].isLetterOrDigit() || code[at] == '_' || code[at] == '.') + ) { + at++ + } + emit(start, Kind.LITERAL) + return true + } + + private fun word(): Boolean { + if (!isWordStart(code[at])) return false + val start = at + while (at < code.length && isWordPart(code[at])) at++ + if (code.substring(start, at) in rules.keywords) emit(start, Kind.KEYWORD) + return true + } + + private fun singleCharacter(): Boolean { + val kind = + when (code[at]) { + in PUNCTUATION -> Kind.PUNCTUATION + in MARKS -> Kind.MARK + else -> return false + } + at++ + emit(at - 1, kind) + return true + } +} + +private fun isWordStart(c: Char?) = c != null && (c.isLetter() || c == '_') + +private fun isWordPart(c: Char) = c.isLetterOrDigit() || c == '_' diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/Languages.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/Languages.kt new file mode 100644 index 0000000..c03c715 --- /dev/null +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/Languages.kt @@ -0,0 +1,437 @@ +package com.example.aiapp + +/** + * A language the highlighter has rules for. + * + * The names the reader writes after the backticks are aliases onto these; [fenceLanguage] holds + * that table. A word with no entry there is null, and null is drawn plain, because a fence coloured + * by another language's rules looks highlighted and is wrong in a way the reader cannot see. + */ +enum class Language { + C, + COFFEESCRIPT, + CPP, + CSHARP, + DART, + FISH, + GO, + JAVA, + JAVASCRIPT, + JSON, + KOTLIN, + PERL, + PHP, + PYTHON, + RON, + RUBY, + RUST, + SHELL, + SWIFT, + TOML, + TYPESCRIPT, +} + +/** + * What [scan] needs to know about one language -- data, not code, so that adding a language is a + * row in [RULES] rather than a branch anywhere. + * + * The two forms that could not be expressed as data are flags here and a few lines in the scanner: + * [rawStrings], because the closing delimiter depends on how many hashes the opener had, and + * [lifetimes], because whether `'` opens anything at all depends on what follows it. + */ +data class Rules( + /** Words drawn as keywords. Only plain words; the scanner cannot reach anything else. */ + val keywords: Set, + /** Tokens that open a comment running to the end of the line. */ + val lineComments: List = emptyList(), + /** + * Whether [lineComments] count only at the start of a word. + * + * The shells need it: `$#`, `${#x}` and `a#b` are not comments, and greying the rest of those + * lines is one of the mistakes this scanner exists to stop. + */ + val lineCommentsAtWordStart: Boolean = false, + val blockComment: BlockComment? = null, + /** The string forms. The longest opener that matches wins, so `"""` is tried before `"`. */ + val quotes: List = emptyList(), + val attributes: Attributes = Attributes.NONE, + /** Rust and RON: an optional `b`, `r`, n hashes, `"`, closing at `"` and n hashes. */ + val rawStrings: Boolean = false, + /** + * Rust: `'` opens a character literal only when a backslash or one character and a `'` follow. + * Otherwise it is a lifetime or a label and no string starts -- without this, `'a` opens a + * string that runs to the next apostrophe in the block. + */ + val lifetimes: Boolean = false, +) + +data class BlockComment(val open: String, val close: String, val nests: Boolean) + +/** One string form. [escapes] is whether a backslash escapes the closer (and itself). */ +data class Quote(val open: String, val close: String, val escapes: Boolean) + +/** What opens a metadata span, of the shapes that exist across these languages. */ +enum class Attributes { + NONE, + /** `@` and a word: Kotlin and Java annotations, Python decorators. */ + AT_WORD, + /** `#[` or `#![` through the matching `]`: Rust and RON attributes. */ + HASH_BRACKET, + /** `#` at the start of a line, to the end of it: the C preprocessor. */ + HASH_LINE, + /** `[` at the start of a line through the matching `]`: a TOML table header. */ + LINE_BRACKET, +} + +/** The rules for [language]. */ +fun rulesOf(language: Language): Rules = RULES.getValue(language) + +private val C_STYLE = BlockComment("/*", "*/", nests = false) +private val NESTING = BlockComment("/*", "*/", nests = true) + +private val DOUBLE = Quote("\"", "\"", escapes = true) +private val SINGLE = Quote("'", "'", escapes = true) +private val TRIPLE_DOUBLE = Quote("\"\"\"", "\"\"\"", escapes = true) +private val TRIPLE_SINGLE = Quote("'''", "'''", escapes = true) + +// Lazy because the keyword sets below are top-level properties too, and a file's properties +// initialize in the order they are written: read eagerly here, every set would be null. +private val RULES: Map by lazy { + mapOf( + Language.C to + Rules( + keywords = KEYWORDS_C, + lineComments = listOf("//"), + blockComment = C_STYLE, + quotes = listOf(DOUBLE, SINGLE), + attributes = Attributes.HASH_LINE, + ), + Language.CPP to + Rules( + keywords = KEYWORDS_CPP, + lineComments = listOf("//"), + blockComment = C_STYLE, + quotes = listOf(DOUBLE, SINGLE), + attributes = Attributes.HASH_LINE, + ), + Language.CSHARP to + Rules( + keywords = KEYWORDS_CSHARP, + lineComments = listOf("//"), + blockComment = C_STYLE, + quotes = listOf(DOUBLE, SINGLE), + ), + // `###` opens and closes a block comment and `#` opens a line one, which is why the + // scanner tries the block opener first. + Language.COFFEESCRIPT to + Rules( + keywords = KEYWORDS_COFFEESCRIPT, + lineComments = listOf("#"), + blockComment = BlockComment("###", "###", nests = false), + quotes = listOf(TRIPLE_DOUBLE, TRIPLE_SINGLE, DOUBLE, SINGLE), + ), + Language.DART to + Rules( + keywords = KEYWORDS_DART, + lineComments = listOf("//"), + blockComment = NESTING, + quotes = listOf(TRIPLE_DOUBLE, TRIPLE_SINGLE, DOUBLE, SINGLE), + attributes = Attributes.AT_WORD, + ), + Language.FISH to + Rules( + keywords = KEYWORDS_FISH, + lineComments = listOf("#"), + lineCommentsAtWordStart = true, + // fish's single quotes escape only `\'` and `\\`, which is what "skip the + // character after a backslash" already does. + quotes = listOf(DOUBLE, SINGLE), + ), + Language.GO to + Rules( + keywords = KEYWORDS_GO, + lineComments = listOf("//"), + blockComment = C_STYLE, + quotes = listOf(DOUBLE, SINGLE, Quote("`", "`", escapes = false)), + ), + Language.JAVA to + Rules( + keywords = KEYWORDS_JAVA, + lineComments = listOf("//"), + blockComment = C_STYLE, + quotes = listOf(DOUBLE, SINGLE), + attributes = Attributes.AT_WORD, + ), + Language.JAVASCRIPT to + Rules( + keywords = KEYWORDS_JAVASCRIPT, + lineComments = listOf("//"), + blockComment = C_STYLE, + quotes = listOf(DOUBLE, SINGLE, Quote("`", "`", escapes = true)), + ), + Language.JSON to Rules(keywords = KEYWORDS_JSON, quotes = listOf(DOUBLE)), + Language.KOTLIN to + Rules( + keywords = KEYWORDS_KOTLIN, + lineComments = listOf("//"), + blockComment = NESTING, + quotes = listOf(Quote("\"\"\"", "\"\"\"", escapes = false), DOUBLE, SINGLE), + attributes = Attributes.AT_WORD, + ), + Language.PERL to + Rules( + keywords = KEYWORDS_PERL, + lineComments = listOf("#"), + quotes = listOf(DOUBLE, SINGLE), + ), + Language.PHP to + Rules( + keywords = KEYWORDS_PHP, + lineComments = listOf("//", "#"), + blockComment = C_STYLE, + quotes = listOf(DOUBLE, SINGLE), + attributes = Attributes.AT_WORD, + ), + Language.PYTHON to + Rules( + keywords = KEYWORDS_PYTHON, + lineComments = listOf("#"), + quotes = listOf(TRIPLE_DOUBLE, TRIPLE_SINGLE, DOUBLE, SINGLE), + attributes = Attributes.AT_WORD, + ), + Language.RON to + Rules( + keywords = KEYWORDS_RON, + lineComments = listOf("//"), + blockComment = NESTING, + quotes = listOf(DOUBLE, SINGLE), + attributes = Attributes.HASH_BRACKET, + rawStrings = true, + ), + Language.RUBY to + Rules( + keywords = KEYWORDS_RUBY, + lineComments = listOf("#"), + quotes = listOf(DOUBLE, SINGLE), + ), + Language.RUST to + Rules( + keywords = KEYWORDS_RUST, + lineComments = listOf("//"), + blockComment = NESTING, + // No `'` here: [Rules.lifetimes] decides when one opens a character literal. + quotes = listOf(DOUBLE), + attributes = Attributes.HASH_BRACKET, + rawStrings = true, + lifetimes = true, + ), + Language.SHELL to + Rules( + keywords = KEYWORDS_SHELL, + lineComments = listOf("#"), + lineCommentsAtWordStart = true, + // A shell's single quotes are literal: `'a\'` is not one string. + quotes = listOf(DOUBLE, Quote("'", "'", escapes = false)), + ), + Language.SWIFT to + Rules( + keywords = KEYWORDS_SWIFT, + lineComments = listOf("//"), + blockComment = NESTING, + quotes = listOf(TRIPLE_DOUBLE, DOUBLE), + attributes = Attributes.AT_WORD, + ), + Language.TOML to + Rules( + keywords = KEYWORDS_TOML, + lineComments = listOf("#"), + quotes = + listOf( + TRIPLE_DOUBLE, + Quote("'''", "'''", escapes = false), + DOUBLE, + Quote("'", "'", escapes = false), + ), + attributes = Attributes.LINE_BRACKET, + ), + Language.TYPESCRIPT to + Rules( + keywords = KEYWORDS_TYPESCRIPT, + lineComments = listOf("//"), + blockComment = C_STYLE, + quotes = listOf(DOUBLE, SINGLE, Quote("`", "`", escapes = true)), + attributes = Attributes.AT_WORD, + ), + ) +} + +/** + * The keyword sets. + * + * Every list below other than RON, TOML, fish and JSON came from dev.snipme:highlights 1.1.0 + * (`SyntaxTokens.kt`, Apache-2.0), the library this scanner replaced, so that no fence which is + * coloured today turns plain. Entries that are not plain words were dropped -- Kotlin's `as?`, + * `!in` and `!is`, Swift's `#if` family, Ruby's `defined?`, CoffeeScript's `=` and `->` -- because + * the word scanner cannot reach them and the library only matched them by luck. + */ +private fun words(list: String): Set = + list.split(Regex("\\s+")).filterNot(String::isEmpty).toSet() + +private val KEYWORDS_C = + words( + """auto break case char const continue default do double else enum extern float for goto if + int long register return short signed sizeof static struct switch typedef union unsigned + void volatile while""" + ) + +private val KEYWORDS_CPP = + words( + """asm auto bool break case catch char class const const_cast continue default delete do + double dynamic_cast else enum explicit export extern false float for friend goto if inline + int long mutable namespace new operator private protected public register reinterpret_cast + return short signed sizeof static static_cast struct switch template this throw true try + typedef typeid typename union unsigned using virtual void volatile wchar_t while""" + ) + +private val KEYWORDS_CSHARP = + words( + """abstract as base bool break byte case catch char checked class const continue decimal + default delegate do double else enum event explicit extern false finally fixed float for + foreach goto if implicit in int interface internal is lock long namespace new null object + operator out override params private protected public readonly ref return sbyte sealed short + sizeof stackalloc static string struct switch this throw true try typeof uint ulong unchecked + unsafe ushort using virtual void volatile while""" + ) + +private val KEYWORDS_COFFEESCRIPT = + words( + """Infinity NaN and arguments await break by case catch class continue debugger delete defer + default do else export extends false finally for function if import in instanceof is isnt + let loop new no not null of on or package return super switch this throw true try typeof + unless undefined var wait when with yield""" + ) + +private val KEYWORDS_DART = + words( + """abstract as assert async await base break case catch class const continue covariant + default deferred do dynamic else enum export extends external factory false final finally + for get if implements import in interface is late library mixin new null on operator part + required rethrow return sealed set show static super switch this throw true try var void + when with while yield""" + ) + +/** + * fish is not in the library at all, so its fences are drawn plain today. The list is the shell's + * own words, which is what a fish fence is mostly made of. + */ +private val KEYWORDS_FISH = + words( + """and begin break builtin case command continue else end exec for function if in not or + return switch while set echo test string math read source""" + ) + +private val KEYWORDS_GO = + words( + """break case chan const continue default defer else fallthrough false for func go goto if + import interface map package range return select struct switch true type var""" + ) + +private val KEYWORDS_JAVA = + words( + """abstract assert boolean break byte case catch char class const continue default do double + else enum extends final finally float for goto if implements import instanceof int interface + long native new null package private protected public return short static strictfp super + switch synchronized this throw throws transient try void volatile while""" + ) + +private val KEYWORDS_JAVASCRIPT = + words( + """async await boolean break case catch class const continue debugger default delete do else + enum export extends false finally for function if implements import in instanceof interface + let new null package private protected public return super switch this throw true try typeof + var void while with yield""" + ) + +private val KEYWORDS_JSON = words("true false null") + +private val KEYWORDS_KOTLIN = + words( + """actual abstract annotation as break by catch class companion const constructor continue + coroutine crossinline data delegate dynamic do else enum expect external false final finally + for fun get if import in infix inline interface internal is lazy lateinit native null object + open operator out override package private protected public reified return sealed set super + suspend tailrec this throw true try typealias typeof val var vararg when while yield""" + ) + +private val KEYWORDS_PERL = + words( + """__DATA__ __END__ __FILE__ __LINE__ __PACKAGE__ and cmp continue do else elsif eq eval for + foreach goto gt if last le lt my ne next no not or package redo ref return sub unless until + use while xor""" + ) + +private val KEYWORDS_PHP = + words( + """__halt_compiler abstract and array as break callable case catch class clone const continue + declare default die do echo else elseif empty enddeclare endfor endforeach endif endswitch + endwhile eval exit extends final finally fn for foreach function global goto if implements + include include_once instanceof insteadof interface isset list match new or print private + protected public require require_once return static switch throw trait try unset use var + while xor yield""" + ) + +private val KEYWORDS_PYTHON = + words( + """False True and as assert async await break class continue def del elif else except finally + for from global if import in is lambda nonlocal not or pass raise return try while with + yield""" + ) + +/** RON is not in the library either; these are the words a RON file can hold. */ +private val KEYWORDS_RON = words("true false Some None inf NaN") + +private val KEYWORDS_RUBY = + words( + """__ENCODING__ __END__ __FILE__ __LINE__ BEGIN END alias and begin break case class def do + else elsif end ensure false for if in module next nil not or redo rescue retry return self + super then true undef unless until when while yield""" + ) + +private val KEYWORDS_RUST = + words( + """as async await break const continue crate dyn else enum extern false fn for if impl in + let loop match mod move mut pub ref return Self self static struct super trait true type + union unsafe use where while abstract become box do final macro override priv try typeof + unsized virtual yield""" + ) + +private val KEYWORDS_SHELL = + words( + """alias bg bind break builtin caller cd command compgen complete compopt continue declare + dirs disown echo enable eval exec exit export fc fg getopts hash help history jobs kill let + local logout popd printf pushd pwd read readonly return set shift shopt source suspend + test""" + ) + +private val KEYWORDS_SWIFT = + words( + """_ associatedtype class deinit enum extension fileprivate func import init inout internal + let open operator private precedencegroup protocol public rethrows static struct subscript + typealias var break case catch continue default defer do else fallthrough for guard if in + repeat return throw switch where while Any as await false is nil self Self super throws true + try associativity convenience didSet dynamic final get indirect infix lazy left mutating none + nonmutating optional override postfix precedence prefix Protocol required right set some Type + unowned weak willSet""" + ) + +/** TOML is not in the library; `inf` and `nan` are values rather than names, like the booleans. */ +private val KEYWORDS_TOML = words("true false inf nan") + +private val KEYWORDS_TYPESCRIPT = + words( + """abstract as asserts await break case catch class const constructor continue debugger + default delete do else enum export extends false finally for from function get if implements + import in infer instanceof interface is keyof let module namespace new null number object + package private protected public readonly require global return set static string super + switch this throw true try type typeof undefined unique unknown var void while with yield""" + ) diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/Markdown.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/Markdown.kt index a95074f..d9699fc 100644 --- a/app/androidApp/src/main/kotlin/com/example/aiapp/Markdown.kt +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/Markdown.kt @@ -55,7 +55,6 @@ import com.mikepenz.markdown.model.markdownAnimations import com.mikepenz.markdown.model.markdownDimens import com.mikepenz.markdown.model.markdownPadding import com.mikepenz.markdown.model.parseMarkdown -import dev.snipme.highlights.model.SyntaxLanguage import java.util.concurrent.ConcurrentHashMap import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.withContext @@ -670,7 +669,7 @@ class ParsedReplies { * * The key carries the language, because the same code lexes differently under two of them. */ - fun highlighted(code: String, language: SyntaxLanguage?): AnnotatedString = + fun highlighted(code: String, language: Language?): AnnotatedString = if (language == null) AnnotatedString(code) else highlights.computeIfAbsent("$language\n$code") { highlight(code, language) } diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/Theme.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/Theme.kt index e83c45c..f8c9ce4 100644 --- a/app/androidApp/src/main/kotlin/com/example/aiapp/Theme.kt +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/Theme.kt @@ -6,8 +6,6 @@ import androidx.compose.material3.MaterialTheme import androidx.compose.material3.darkColorScheme import androidx.compose.runtime.Composable import androidx.compose.ui.graphics.Color -import androidx.compose.ui.graphics.toArgb -import dev.snipme.highlights.model.SyntaxTheme /** * Catppuccin Mocha, as published in `catppuccin/palette`. @@ -198,24 +196,24 @@ val rawSurface: Color @Composable get() = Mocha.Crust /** - * Catppuccin Mocha as a syntax theme, for the highlighter used on a tool call's input. + * Catppuccin Mocha as the highlighter's palette; see [SyntaxPalette]. * - * Here with the rest of the palette rather than beside the code that highlights: a library's own - * theme would otherwise be the one surface in the app whose colours came from somewhere else, and - * the accents below are the same ones every other coloured thing already uses. + * Here with the rest of the palette rather than beside the code that highlights: the colours a + * fence is drawn in are the same accents every other coloured thing in the app already uses, and + * splitting them out would make code the one surface whose palette came from somewhere else. + * + * Not a composable, because [highlight] runs off the drawing thread; these colours never vary with + * the theme. */ -fun catppuccinSyntax(): SyntaxTheme = - SyntaxTheme( - key = "catppuccin-mocha", - code = Mocha.Text.toArgb(), - keyword = Mocha.Mauve.toArgb(), - string = Mocha.Green.toArgb(), - literal = Mocha.Peach.toArgb(), - comment = Mocha.Overlay0.toArgb(), - metadata = Mocha.Yellow.toArgb(), - multilineComment = Mocha.Overlay0.toArgb(), - punctuation = Mocha.Subtext0.toArgb(), - mark = Mocha.Sky.toArgb(), +fun catppuccinSyntax(): SyntaxPalette = + SyntaxPalette( + keyword = Mocha.Mauve, + string = Mocha.Green, + literal = Mocha.Peach, + comment = Mocha.Overlay0, + metadata = Mocha.Yellow, + punctuation = Mocha.Subtext0, + mark = Mocha.Sky, ) /** diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/ToolInput.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/ToolInput.kt index 4ed473f..ff4350c 100644 --- a/app/androidApp/src/main/kotlin/com/example/aiapp/ToolInput.kt +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/ToolInput.kt @@ -11,7 +11,6 @@ import androidx.compose.runtime.remember import androidx.compose.ui.Modifier import androidx.compose.ui.text.font.FontFamily import androidx.compose.ui.unit.dp -import dev.snipme.highlights.model.SyntaxLanguage import org.json.JSONObject /** @@ -27,7 +26,7 @@ data class ToolInput( /** The thing that will actually be run or read, if this tool has one. */ val subject: String?, /** The language [subject] is written in, for highlighting. */ - val language: SyntaxLanguage?, + val language: Language?, /** The tool's own one-line summary, when it wrote one. */ val description: String?, /** @@ -50,9 +49,9 @@ data class ToolInput( * from being the special case that gets its own code path. Unknown tools fall through to "no * subject, everything is rest", which is what the card always did. */ -private val SUBJECTS: Map> = +private val SUBJECTS: Map> = mapOf( - "Bash" to ("command" to SyntaxLanguage.SHELL), + "Bash" to ("command" to Language.SHELL), "Read" to ("file_path" to null), "Write" to ("file_path" to null), "Edit" to ("file_path" to null), diff --git a/app/androidApp/src/test/kotlin/com/example/aiapp/HighlighterTest.kt b/app/androidApp/src/test/kotlin/com/example/aiapp/HighlighterTest.kt new file mode 100644 index 0000000..4e5a43d --- /dev/null +++ b/app/androidApp/src/test/kotlin/com/example/aiapp/HighlighterTest.kt @@ -0,0 +1,240 @@ +package com.example.aiapp + +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertTrue + +/** + * What the scanner colours, asserted as the text under each span rather than as offsets, so a + * failure prints the code that was got wrong instead of a pair of numbers. + * + * Most of these are the mistakes dev.snipme:highlights 1.1.0 made -- the library this scanner + * replaced -- measured against it directly before it was removed. They are here rather than in the + * `highlights-repro.sh` script they came from because a case that only a script can ask about is a + * case nobody asks about. + */ +class HighlighterTest { + /** Every span of [kind] in [code], as the text it covers. */ + private fun spans(code: String, language: Language, kind: Kind): List = + scan(code, rulesOf(language)) + .filter { it.kind == kind } + .map { code.substring(it.start, it.end) } + + private fun assertSpans( + code: String, + language: Language, + kind: Kind, + vararg expected: String, + ) { + assertEquals(expected.toList(), spans(code, language, kind), "$kind in: $code") + } + + // The four inputs the repro script asked the library about, and what it answered. + + @Test + fun `a quoted glob is one string, not a comment`() { + // The library answered a span whose end preceded its start here, which crashed the app. + assertSpans("x '*/a/*'", Language.SHELL, Kind.STRING, "'*/a/*'") + assertSpans("x '*/a/*'", Language.SHELL, Kind.COMMENT) + } + + @Test + fun `a find with globs has no comment in it`() { + val code = "find . -path '*/.git/*' -prune -o -name '*.kt' -print" + assertSpans(code, Language.SHELL, Kind.STRING, "'*/.git/*'", "'*.kt'") + assertSpans(code, Language.SHELL, Kind.COMMENT) + } + + @Test + fun `a URL does not comment out the rest of a shell line`() { + val code = "curl https://example.com/x && echo done" + assertSpans(code, Language.SHELL, Kind.COMMENT) + assertSpans(code, Language.SHELL, Kind.KEYWORD, "echo") + } + + @Test + fun `a URL inside a Kotlin string stays a string`() { + val code = "val url = \"https://example.com\"\nfun f() = 1" + assertSpans(code, Language.KOTLIN, Kind.COMMENT) + assertSpans(code, Language.KOTLIN, Kind.STRING, "\"https://example.com\"") + assertSpans(code, Language.KOTLIN, Kind.KEYWORD, "val", "fun") + } + + // Attributes, which the library greyed out as comments. + + @Test + fun `a Rust attribute is metadata and the struct after it still colours`() { + val code = "#[derive(Debug)]\nstruct A { b: u8 }" + assertSpans(code, Language.RUST, Kind.METADATA, "#[derive(Debug)]") + assertSpans(code, Language.RUST, Kind.COMMENT) + assertSpans(code, Language.RUST, Kind.KEYWORD, "struct") + } + + @Test + fun `an inner Rust attribute closes at its own bracket`() { + val code = "#![allow(dead_code)]\nfn f() {}" + assertSpans(code, Language.RUST, Kind.METADATA, "#![allow(dead_code)]") + assertSpans(code, Language.RUST, Kind.KEYWORD, "fn") + } + + @Test + fun `a C preprocessor line is metadata rather than a comment`() { + val code = "#include \nint main() { return 0; }" + assertSpans(code, Language.C, Kind.METADATA, "#include ") + assertSpans(code, Language.C, Kind.COMMENT) + assertSpans(code, Language.C, Kind.KEYWORD, "int", "return") + } + + @Test + fun `a Kotlin annotation is metadata`() { + assertSpans("@Composable fun f() {}", Language.KOTLIN, Kind.METADATA, "@Composable") + } + + // Strings whose contents the library read as code. + + @Test + fun `a hash inside a Kotlin string is not a comment`() { + val code = "val c = \"#FF0000\"\nval d = 1" + assertSpans(code, Language.KOTLIN, Kind.COMMENT) + assertSpans(code, Language.KOTLIN, Kind.STRING, "\"#FF0000\"") + } + + @Test + fun `an apostrophe inside a Kotlin string does not open one`() { + val code = "val a = \"don't\"\nval b = \"x\"" + assertSpans(code, Language.KOTLIN, Kind.STRING, "\"don't\"", "\"x\"") + } + + @Test + fun `a Rust lifetime does not open a string but a character literal does`() { + val code = "fn f<'a>(x: &'a str) { let c = 'x'; }" + assertSpans(code, Language.RUST, Kind.STRING, "'x'") + } + + @Test + fun `an escaped quote is inside the Rust character literal`() { + assertSpans("let c = '\\'';", Language.RUST, Kind.STRING, "'\\''") + } + + @Test + fun `a Rust raw string keeps its inner quotes`() { + val code = "let s = r#\"a \"quoted\" b\"#;" + assertSpans(code, Language.RUST, Kind.STRING, "r#\"a \"quoted\" b\"#") + } + + @Test + fun `a Kotlin triple quoted string is one string`() { + assertSpans( + "val s = \"\"\"a \"b\" c\"\"\"", + Language.KOTLIN, + Kind.STRING, + "\"\"\"a \"b\" c\"\"\"", + ) + } + + @Test + fun `a shell single quoted string takes no escapes`() { + // `\` is literal inside shell single quotes, so the string ends at the next apostrophe. + assertSpans("echo 'a\\' b", Language.SHELL, Kind.STRING, "'a\\'") + } + + // Comments. + + @Test + fun `Rust and Kotlin nest block comments`() { + val code = "/* a /* b */ c */ x" + assertSpans(code, Language.RUST, Kind.COMMENT, "/* a /* b */ c */") + assertSpans(code, Language.KOTLIN, Kind.COMMENT, "/* a /* b */ c */") + } + + @Test + fun `C ends a block comment at the first close`() { + assertSpans("/* a /* b */ c */ x", Language.C, Kind.COMMENT, "/* a /* b */") + } + + @Test + fun `a shell comment starts only at a word boundary`() { + val code = "\${#x} \$# a#b # real" + assertSpans(code, Language.SHELL, Kind.COMMENT, "# real") + } + + @Test + fun `a hash anywhere is a Python comment`() { + assertSpans("x = 1 # note", Language.PYTHON, Kind.COMMENT, "# note") + } + + // TOML, which the library has no rules for at all. + + @Test + fun `a TOML table header is metadata and a hash in a value is not a comment`() { + val code = "[server]\ncolour = \"#FF0000\"\nport = 8080 # the real one" + assertSpans(code, Language.TOML, Kind.METADATA, "[server]") + assertSpans(code, Language.TOML, Kind.STRING, "\"#FF0000\"") + assertSpans(code, Language.TOML, Kind.COMMENT, "# the real one") + assertSpans(code, Language.TOML, Kind.LITERAL, "8080") + } + + @Test + fun `a RON attribute and its values colour`() { + val code = "#![enable(implicit_some)]\n(count: 3, on: true)" + assertSpans(code, Language.RON, Kind.METADATA, "#![enable(implicit_some)]") + assertSpans(code, Language.RON, Kind.KEYWORD, "true") + assertSpans(code, Language.RON, Kind.LITERAL, "3") + } + + @Test + fun `an unknown fence language is drawn plain`() { + assertEquals(null, fenceLanguage("brainfuck")) + assertEquals("+[-]", highlight("+[-]", fenceLanguage("brainfuck")).text) + } + + @Test + fun `every alias the fence table knows has rules`() { + Language.entries.forEach { rulesOf(it) } + } + + /** + * The scanner must never throw and must never answer a span the code does not contain: the + * library's reversed range is exactly the shape that crashed a card, and a fence still being + * written is an unterminated string or comment on every keystroke. + */ + @Test + fun `spans stay inside the code for every language and every nasty input`() { + val nasty = + listOf( + "", + "'", + "\"", + "\"unterminated", + "/* unterminated", + "###", + "#", + "#![", + "[", + "r#\"", + "\\", + "'''", + "\"\"\"", + "0x", + "1.2.3", + "a#b//c/*d*/'e\"f", + "\n\n \n", + ) + for (language in Language.entries) { + for (code in nasty) { + val spans = scan(code, rulesOf(language)) + spans.forEach { + assertTrue( + it.start in 0..it.end && it.end <= code.length, + "$language answered $it for ${code.replace("\n", "\\n")}", + ) + } + assertEquals( + spans.sortedBy { it.start }, + spans, + "$language answered spans out of order for ${code.replace("\n", "\\n")}", + ) + } + } + } +} diff --git a/app/gradle/libs.versions.toml b/app/gradle/libs.versions.toml index ad7ebba..43ab95b 100644 --- a/app/gradle/libs.versions.toml +++ b/app/gradle/libs.versions.toml @@ -27,10 +27,6 @@ zxing-embedded = "4.3.0" # https://repo1.maven.org/maven2/com/mikepenz/multiplatform-markdown-renderer-m3/maven-metadata.xml # Latest stable there, checked 2026-08-29. markdown-renderer = "0.45.0" -# Syntax highlighting for a tool call's input. Same reasoning as the markdown -# renderer: a language's lexical rules are somebody else's specification. -# Latest stable, checked 2026-08-31 against Maven Central. -highlights = "1.1.0" # The support ExifInterface rather than android.media's, which lint warns off: # the framework one is missing formats and the fixes for parsing hostile # images, and images here arrive from outside the phone. Latest stable, @@ -46,6 +42,12 @@ androidx-lifecycle = "2.11.0" # itself is Kotlin-org owned and has almost nothing to configure, which is # the point; this is the Gradle wrapper for it. Checked 2026-08-28. ktfmt-gradle = "0.27.0" +# The test framework for the app's JVM unit tests (Highlighter.kt's scanner). +# JUnit 6 is the current line -- Jupiter and the Platform ship on one version +# now -- and the tests themselves are written against `kotlin.test`, so the +# framework is a build-file choice rather than something the source names. +# Latest stable, checked 2026-09-03 against Maven Central. +junit = "6.1.3" # Backports java.time (and more) to API 24, which UsageScreen needs: its # reset countdown is OffsetDateTime/Duration, both API 26. Checked # 2026-08-28 against Google Maven. @@ -64,8 +66,15 @@ desugar-jdk-libs = { module = "com.android.tools:desugar_jdk_libs", version.ref # The -m3 flavour: it takes its colours and type from the ambient Material 3 # theme, so the app's Catppuccin scheme is what it draws with. markdown-renderer = { module = "com.mikepenz:multiplatform-markdown-renderer-m3", version.ref = "markdown-renderer" } -highlights = { module = "dev.snipme:highlights", version.ref = "highlights" } androidx-exifinterface = { module = "androidx.exifinterface:exifinterface", version.ref = "androidx-exifinterface" } +# The framework-neutral assertions the tests are written against. The `-junit5` +# artifact rather than plain `kotlin-test`: AGP 9 compiles Kotlin itself rather +# than through the Kotlin Android plugin, so nothing here resolves the variant +# of `kotlin-test` that carries `kotlin.test.Test` -- naming the artifact is +# what makes the annotation exist. +kotlin-test-junit5 = { module = "org.jetbrains.kotlin:kotlin-test-junit5", version.ref = "kotlin" } +junit-jupiter = { module = "org.junit.jupiter:junit-jupiter", version.ref = "junit" } +junit-platform-launcher = { module = "org.junit.platform:junit-platform-launcher", version.ref = "junit" } # Declared directly rather than through the plugin's `compose.*` accessors, # which are deprecated as of CMP 1.11. compose-runtime = { module = "org.jetbrains.compose.runtime:runtime", version.ref = "compose-multiplatform" } diff --git a/app/highlights-repro.sh b/app/highlights-repro.sh deleted file mode 100755 index b3d9039..0000000 --- a/app/highlights-repro.sh +++ /dev/null @@ -1,59 +0,0 @@ -#!/usr/bin/env bash -# What dev.snipme:highlights actually answers for a piece of code, outside the app. -# -# The lexer's mistakes are invisible on a phone -- a line greyed out as a comment looks -# like a comment -- so this asks it directly and prints every span with the text under -# it. It is how the `/*`-pairing and URL-comment bugs in TRANSCRIPT_RENDERING.md were -# measured, and it is what any fix has to be checked against. -# -# Usage: ./highlights-repro.sh [LANGUAGE FILE] (default: the known-bad cases) -set -euo pipefail - -cache=~/.gradle/caches/modules-2/files-2.1 -jar() { find "$cache/$1" -name "$2" ! -name '*sources*' | sort | tail -1; } - -cp=$(jar dev.snipme/highlights-jvm/1.1.0 '*.jar') -cp=$cp:$(jar org.jetbrains.kotlin/kotlin-stdlib '*[0-9].jar') -cp=$cp:$(jar org.jetbrains.kotlinx/kotlinx-coroutines-core-jvm '*.jar') -cp=$cp:$(jar org.jetbrains.kotlinx/kotlinx-serialization-core-jvm '*.jar') -cp=$cp:$(jar org.jetbrains.kotlinx/kotlinx-serialization-json-jvm '*.jar') -for entry in ${cp//:/ }; do - [ -f "$entry" ] || { echo "missing jar in the gradle cache: build the app once first" >&2; exit 1; } -done - -out=$(mktemp -d) -trap 'rm -rf "$out"' EXIT -cat > "$out/Repro.java" <<'JAVA' -import dev.snipme.highlights.Highlights; -import dev.snipme.highlights.model.*; -import java.nio.file.*; - -public class Repro { - public static void main(String[] args) throws Exception { - if (args.length == 2) { - report(SyntaxLanguage.valueOf(args[0].toUpperCase()), Files.readString(Path.of(args[1]))); - return; - } - report(SyntaxLanguage.SHELL, "x '*/a/*'"); - report(SyntaxLanguage.SHELL, "find . -path '*/.git/*' -prune -o -name '*.kt' -print"); - report(SyntaxLanguage.SHELL, "curl https://example.com/x && echo done"); - report(SyntaxLanguage.KOTLIN, "val url = \"https://example.com\"\nfun f() = 1"); - } - - static void report(SyntaxLanguage language, String code) { - System.out.println("== [" + language + "] " + code.replace("\n", "\\n")); - for (CodeHighlight highlight : new Highlights.Builder().code(code).language(language) - .build().getHighlights()) { - PhraseLocation at = highlight instanceof ColorHighlight - ? ((ColorHighlight) highlight).getLocation() - : ((BoldHighlight) highlight).getLocation(); - boolean sane = at.getStart() >= 0 && at.getStart() <= at.getEnd() - && at.getEnd() <= code.length(); - System.out.println((sane ? " " : " BAD ") + at + " " - + (sane ? "\"" + code.substring(at.getStart(), at.getEnd()) + "\"" : "")); - } - } -} -JAVA -javac -cp "$cp" -d "$out" "$out/Repro.java" -java -cp "$cp:$out" Repro "$@"