Colour code with a scanner of our own instead of the library
dev.snipme:highlights 1.1.0 found comments before it knew the language and paired /* with */ by ordinal, so `//` in any URL commented out the rest of its line, every Rust `#[derive(...)]` greyed out as a comment, a `#` inside a Kotlin string swallowed the line, and `x '*/a/*'` in shell produced a span whose end preceded its start -- the one that crashed a card holding `-path '*/.git/*'`. None of that could be post-processed away, because comments won over strings before the language was known. Highlighter.kt is one left-to-right scanner: at each position it is in a line comment, a block comment, a string, or ordinary code, and every span is emitted by advancing an index, so spans cannot overlap, arrive out of order or run backwards. Languages.kt is a `Rules` row per language -- comment tokens, block comment and whether it nests, the string forms, what opens an attribute, and the keyword set -- so a new language is a table entry. The keyword lists came from the library's SyntaxTokens.kt (Apache-2.0, noted at the table) so nothing that is coloured today turns plain, and RON, TOML, fish and JSON are coloured for the first time. HighlighterTest.kt is a new JVM unit test source set -- 24 cases, the library's mistakes kept as regressions, plus a sweep asserting no span escapes the code for any language on unterminated and empty input. AGENTS.md's app line now runs :androidApp:testDebugUnitTest. Measured on the ai-app emulator, debug build, a ~200-line Kotlin fence sent into a sandbox session: before code highlighted: 1, 101.9ms total, 101.9ms mean, 101.9ms worst after code highlighted: 1, 15.0ms total, 15.0ms mean, 15.0ms worst and a second fence in the same run took 13.9ms, so that is the steady cost rather than class loading. stream-bench.sh after the change: code highlighted: 1, 12.1ms total, 12.1ms mean, 12.1ms worst markdown reparsed while streaming: 1329, 2130.7ms total, 1.6ms mean, 8.7ms worst record: one block: 131, 11.4ms total, 0.1ms mean, 0.4ms worst draw phase 1.21ms per frame, the transcript 0.23ms of it transcript-bench.sh after: draw phase 1.10ms per frame, the transcript 0.49ms (place 0.48), worst place 4.3ms -- unchanged within run-to-run noise, as expected, since the scan happens in `warm` and not while drawing. Looked at on the emulator: a URL inside a Kotlin string, a Rust attribute with a lifetime and a raw string, a shell line with globs and `$#`, a RON fence and a TOML fence all colour correctly; a Bash tool card still colours its command; a plain Python fence -- which this change had no reason to touch -- looks as it did; an unknown language stays plain; and a fence is plain while it streams and colours when it freezes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
irisandClaude Opus 5 committed
1 parent
3d90e0947c
commit
a2b11d516f
13 files changed
+1136
-495
No files matched your search
@@ -11,23 +11,14 @@ import androidx.compose.foundation.text.BasicText
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.graphics.Color
|
||||
import androidx.compose.ui.semantics.isTraversalGroup
|
||||
import androidx.compose.ui.semantics.semantics
|
||||
import androidx.compose.ui.text.AnnotatedString
|
||||
import androidx.compose.ui.text.SpanStyle
|
||||
import androidx.compose.ui.text.TextStyle
|
||||
import androidx.compose.ui.text.buildAnnotatedString
|
||||
import androidx.compose.ui.text.font.FontWeight
|
||||
import androidx.compose.ui.unit.dp
|
||||
import com.mikepenz.markdown.compose.LocalMarkdownColors
|
||||
import com.mikepenz.markdown.compose.LocalMarkdownDimens
|
||||
import com.mikepenz.markdown.compose.LocalMarkdownPadding
|
||||
import com.mikepenz.markdown.model.State
|
||||
import dev.snipme.highlights.Highlights
|
||||
import dev.snipme.highlights.model.BoldHighlight
|
||||
import dev.snipme.highlights.model.ColorHighlight
|
||||
import dev.snipme.highlights.model.SyntaxLanguage
|
||||
import org.intellij.markdown.MarkdownElementTypes
|
||||
import org.intellij.markdown.MarkdownTokenTypes
|
||||
import org.intellij.markdown.ast.ASTNode
|
||||
@@ -38,10 +29,10 @@ import org.intellij.markdown.ast.getTextInNode
|
||||
* A fenced code block in a reply: the code highlighted, on the dark surface every verbatim thing
|
||||
* sits on, scrolling sideways rather than wrapping.
|
||||
*
|
||||
* The renderer's own fence drew the same block in plain text. The lexer that colours a tool call's
|
||||
* command colours a reply's code the same way, through [highlighted] and one theme, so a `kotlin`
|
||||
* fence and the Kotlin a tool wrote are the same colours. A fence in a language the lexer has no
|
||||
* rules for is plain rather than wrongly coloured: [fenceLanguage] answers null for those, and
|
||||
* The renderer's own fence drew the same block in plain text. The scanner that colours a tool
|
||||
* call's command colours a reply's code the same way, through [highlighted] and one palette, so a
|
||||
* `kotlin` fence and the Kotlin a tool wrote are the same colours. A fence in a language [scan] has
|
||||
* no rules for is plain rather than wrongly coloured: [fenceLanguage] answers null for those, and
|
||||
* plain is what the reader would have seen before.
|
||||
*
|
||||
* Finding the code is still the library's: which children of the node are the fence markers, the
|
||||
@@ -74,7 +65,7 @@ fun CodeBlock(
|
||||
}
|
||||
|
||||
/**
|
||||
* The code inside a fence or indented block, and the lexer's language for its info word.
|
||||
* The code inside a fence or indented block, and the highlighter's language for its info word.
|
||||
*
|
||||
* Which children of the node are the fence markers, the language word and the code between them is
|
||||
* the library's knowledge of the parser, copied from its `MarkdownCodeFence` rather than called:
|
||||
@@ -85,7 +76,7 @@ fun CodeBlock(
|
||||
* Null for a fence too short to hold anything -- an unterminated one still arriving, which the
|
||||
* library skips as invalid.
|
||||
*/
|
||||
fun fenceContent(content: String, node: ASTNode): Pair<String, SyntaxLanguage?>? {
|
||||
fun fenceContent(content: String, node: ASTNode): Pair<String, Language?>? {
|
||||
val word =
|
||||
node.findChildOfType(MarkdownTokenTypes.FENCE_LANG)?.getTextInNode(content)?.toString()
|
||||
val language = fenceLanguage(word)
|
||||
@@ -111,7 +102,7 @@ fun fenceContent(content: String, node: ASTNode): Pair<String, SyntaxLanguage?>?
|
||||
@Composable
|
||||
private fun CodeBlockText(
|
||||
code: String,
|
||||
language: SyntaxLanguage?,
|
||||
language: Language?,
|
||||
style: TextStyle,
|
||||
replies: ParsedReplies,
|
||||
streaming: Boolean,
|
||||
@@ -136,122 +127,72 @@ private fun CodeBlockText(
|
||||
}
|
||||
|
||||
/**
|
||||
* The lexer's language for a fence's info word, or null for one it has no lexer for.
|
||||
* The highlighter's language for a fence's info word, or null for one it has no rules for.
|
||||
*
|
||||
* The aliases are what people actually write after the backticks: the file extension as often as
|
||||
* the name. A word not here gets no colour rather than the nearest lexer's, because a fence
|
||||
* the name. A word not here gets no colour rather than the nearest language's, because a fence
|
||||
* coloured by the wrong language's rules looks highlighted and is wrong in a way the reader cannot
|
||||
* see.
|
||||
*/
|
||||
fun fenceLanguage(name: String?): SyntaxLanguage? =
|
||||
fun fenceLanguage(name: String?): Language? =
|
||||
FENCE_LANGUAGES[name?.trim()?.lowercase() ?: return null]
|
||||
|
||||
private val FENCE_LANGUAGES: Map<String, SyntaxLanguage> =
|
||||
private val FENCE_LANGUAGES: Map<String, Language> =
|
||||
mapOf(
|
||||
"kotlin" to SyntaxLanguage.KOTLIN,
|
||||
"kt" to SyntaxLanguage.KOTLIN,
|
||||
"kts" to SyntaxLanguage.KOTLIN,
|
||||
"rust" to SyntaxLanguage.RUST,
|
||||
"rs" to SyntaxLanguage.RUST,
|
||||
"sh" to SyntaxLanguage.SHELL,
|
||||
"bash" to SyntaxLanguage.SHELL,
|
||||
"shell" to SyntaxLanguage.SHELL,
|
||||
"zsh" to SyntaxLanguage.SHELL,
|
||||
"console" to SyntaxLanguage.SHELL,
|
||||
"python" to SyntaxLanguage.PYTHON,
|
||||
"py" to SyntaxLanguage.PYTHON,
|
||||
"javascript" to SyntaxLanguage.JAVASCRIPT,
|
||||
"js" to SyntaxLanguage.JAVASCRIPT,
|
||||
"jsx" to SyntaxLanguage.JAVASCRIPT,
|
||||
"typescript" to SyntaxLanguage.TYPESCRIPT,
|
||||
"ts" to SyntaxLanguage.TYPESCRIPT,
|
||||
"tsx" to SyntaxLanguage.TYPESCRIPT,
|
||||
"java" to SyntaxLanguage.JAVA,
|
||||
"c" to SyntaxLanguage.C,
|
||||
"h" to SyntaxLanguage.C,
|
||||
"cpp" to SyntaxLanguage.CPP,
|
||||
"c++" to SyntaxLanguage.CPP,
|
||||
"cc" to SyntaxLanguage.CPP,
|
||||
"hpp" to SyntaxLanguage.CPP,
|
||||
"csharp" to SyntaxLanguage.CSHARP,
|
||||
"cs" to SyntaxLanguage.CSHARP,
|
||||
"c#" to SyntaxLanguage.CSHARP,
|
||||
"go" to SyntaxLanguage.GO,
|
||||
"golang" to SyntaxLanguage.GO,
|
||||
"swift" to SyntaxLanguage.SWIFT,
|
||||
"dart" to SyntaxLanguage.DART,
|
||||
"ruby" to SyntaxLanguage.RUBY,
|
||||
"rb" to SyntaxLanguage.RUBY,
|
||||
"php" to SyntaxLanguage.PHP,
|
||||
"perl" to SyntaxLanguage.PERL,
|
||||
"pl" to SyntaxLanguage.PERL,
|
||||
"coffeescript" to SyntaxLanguage.COFFEESCRIPT,
|
||||
"coffee" to SyntaxLanguage.COFFEESCRIPT,
|
||||
"kotlin" to Language.KOTLIN,
|
||||
"kt" to Language.KOTLIN,
|
||||
"kts" to Language.KOTLIN,
|
||||
"rust" to Language.RUST,
|
||||
"rs" to Language.RUST,
|
||||
"sh" to Language.SHELL,
|
||||
"bash" to Language.SHELL,
|
||||
"shell" to Language.SHELL,
|
||||
"zsh" to Language.SHELL,
|
||||
"console" to Language.SHELL,
|
||||
"python" to Language.PYTHON,
|
||||
"py" to Language.PYTHON,
|
||||
"javascript" to Language.JAVASCRIPT,
|
||||
"js" to Language.JAVASCRIPT,
|
||||
"jsx" to Language.JAVASCRIPT,
|
||||
"typescript" to Language.TYPESCRIPT,
|
||||
"ts" to Language.TYPESCRIPT,
|
||||
"tsx" to Language.TYPESCRIPT,
|
||||
"java" to Language.JAVA,
|
||||
"c" to Language.C,
|
||||
"h" to Language.C,
|
||||
"cpp" to Language.CPP,
|
||||
"c++" to Language.CPP,
|
||||
"cc" to Language.CPP,
|
||||
"hpp" to Language.CPP,
|
||||
"csharp" to Language.CSHARP,
|
||||
"cs" to Language.CSHARP,
|
||||
"c#" to Language.CSHARP,
|
||||
"go" to Language.GO,
|
||||
"golang" to Language.GO,
|
||||
"swift" to Language.SWIFT,
|
||||
"dart" to Language.DART,
|
||||
"ruby" to Language.RUBY,
|
||||
"rb" to Language.RUBY,
|
||||
"php" to Language.PHP,
|
||||
"perl" to Language.PERL,
|
||||
"pl" to Language.PERL,
|
||||
"coffeescript" to Language.COFFEESCRIPT,
|
||||
"coffee" to Language.COFFEESCRIPT,
|
||||
"ron" to Language.RON,
|
||||
"toml" to Language.TOML,
|
||||
"fish" to Language.FISH,
|
||||
"json" to Language.JSON,
|
||||
)
|
||||
|
||||
/**
|
||||
* [code] with its keywords and strings coloured, or plain if there is no language for it.
|
||||
*
|
||||
* The lexing is dev.snipme:highlights. The colours are this app's, mapped in [catppuccinSyntax] --
|
||||
* a library's default theme would be the one place in the app whose palette came from somewhere
|
||||
* else. Shared by a tool call's input ([ToolInputView]) and a reply's fences ([CodeFence]), so the
|
||||
* same code is the same colours wherever it appears.
|
||||
*
|
||||
* Not a composable, and it takes no colour from the theme, because that is what lets [warm] run it
|
||||
* off the drawing thread: the syntax palette is fixed, and a fence with no language is plain text
|
||||
* which needs no colour of its own -- the style the caller draws it with carries that.
|
||||
*
|
||||
* Measured on the emulator before it was cached: a two-hundred-line Kotlin fence costs **174ms** to
|
||||
* lex, and the lazy list charged that again every time the block scrolled back into composition.
|
||||
* That is the whole reason [ParsedReplies.highlighted] exists rather than a `remember`.
|
||||
*/
|
||||
fun highlight(code: String, language: SyntaxLanguage?): AnnotatedString {
|
||||
if (language == null) return AnnotatedString(code)
|
||||
val marks =
|
||||
DebugStats.timed("code highlighted") {
|
||||
Highlights.Builder(code = code, language = language, theme = catppuccinSyntax())
|
||||
.build()
|
||||
.getHighlights()
|
||||
// highlights 1.1.0's shell lexer answers a quoted glob that looks like a comment
|
||||
// -- `x '*/a/*'` is the smallest input -- with a span whose end is before its
|
||||
// start, and AnnotatedString refuses such a range. That crashed the app the
|
||||
// moment a card holding `-path '*/.git/*'` was opened. Dropped rather than
|
||||
// clamped: a span the lexer got backwards is not one it knows the colour of.
|
||||
// Delete when snipme/highlights fixes it.
|
||||
.filter {
|
||||
it.location.start in 0..it.location.end && it.location.end <= code.length
|
||||
}
|
||||
}
|
||||
return buildAnnotatedString {
|
||||
append(code)
|
||||
marks.forEach { mark ->
|
||||
when (mark) {
|
||||
is ColorHighlight ->
|
||||
addStyle(
|
||||
SpanStyle(color = Color(mark.rgb or 0xFF000000.toInt())),
|
||||
mark.location.start,
|
||||
mark.location.end,
|
||||
)
|
||||
is BoldHighlight ->
|
||||
addStyle(
|
||||
SpanStyle(fontWeight = FontWeight.Bold),
|
||||
mark.location.start,
|
||||
mark.location.end,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Every fence in [parse], as the code and language [highlight] will be asked for.
|
||||
*
|
||||
* Walks the whole tree rather than the top level: a fence inside a list item or a quote is drawn
|
||||
* the same way and costs the same to lex.
|
||||
*/
|
||||
fun fences(parse: State): List<Pair<String, SyntaxLanguage?>> {
|
||||
fun fences(parse: State): List<Pair<String, Language?>> {
|
||||
val success = parse as? State.Success ?: return emptyList()
|
||||
val out = ArrayList<Pair<String, SyntaxLanguage?>>()
|
||||
val out = ArrayList<Pair<String, Language?>>()
|
||||
fun walk(node: ASTNode) {
|
||||
if (
|
||||
node.type == MarkdownElementTypes.CODE_FENCE ||
|
||||
|
||||
Reference in new issue
Block a user