iris/ai-app
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Colour code with a scanner of our own instead of the library

Browse Source 
dev.snipme:highlights 1.1.0 found comments before it knew the language and
paired /* with */ by ordinal, so `//` in any URL commented out the rest of
its line, every Rust `#[derive(...)]` greyed out as a comment, a `#` inside
a Kotlin string swallowed the line, and `x '*/a/*'` in shell produced a span
whose end preceded its start -- the one that crashed a card holding
`-path '*/.git/*'`. None of that could be post-processed away, because
comments won over strings before the language was known.

Highlighter.kt is one left-to-right scanner: at each position it is in a
line comment, a block comment, a string, or ordinary code, and every span is
emitted by advancing an index, so spans cannot overlap, arrive out of order
or run backwards. Languages.kt is a `Rules` row per language -- comment
tokens, block comment and whether it nests, the string forms, what opens an
attribute, and the keyword set -- so a new language is a table entry. The
keyword lists came from the library's SyntaxTokens.kt (Apache-2.0, noted at
the table) so nothing that is coloured today turns plain, and RON, TOML,
fish and JSON are coloured for the first time.

HighlighterTest.kt is a new JVM unit test source set -- 24 cases, the
library's mistakes kept as regressions, plus a sweep asserting no span
escapes the code for any language on unterminated and empty input.
AGENTS.md's app line now runs :androidApp:testDebugUnitTest.

Measured on the ai-app emulator, debug build, a ~200-line Kotlin fence sent
into a sandbox session:

  before  code highlighted: 1, 101.9ms total, 101.9ms mean, 101.9ms worst
  after   code highlighted: 1,  15.0ms total,  15.0ms mean,  15.0ms worst

and a second fence in the same run took 13.9ms, so that is the steady cost
rather than class loading. stream-bench.sh after the change:

  code highlighted: 1, 12.1ms total, 12.1ms mean, 12.1ms worst
  markdown reparsed while streaming: 1329, 2130.7ms total, 1.6ms mean, 8.7ms worst
  record: one block: 131, 11.4ms total, 0.1ms mean, 0.4ms worst
  draw phase 1.21ms per frame, the transcript 0.23ms of it

transcript-bench.sh after: draw phase 1.10ms per frame, the transcript
0.49ms (place 0.48), worst place 4.3ms -- unchanged within run-to-run noise,
as expected, since the scan happens in `warm` and not while drawing.

Looked at on the emulator: a URL inside a Kotlin string, a Rust attribute
with a lifetime and a raw string, a shell line with globs and `$#`, a RON
fence and a TOML fence all colour correctly; a Bash tool card still colours
its command; a plain Python fence -- which this change had no reason to
touch -- looks as it did; an unknown language stays plain; and a fence is
plain while it streams and colours when it freezes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
irisandClaude Opus 5 committed 2026-09-03 20:23:59 -04:00
1 parent 3d90e0947c
commit a2b11d516f
13 files changed
+1136 -495
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

app/androidApp/build.gradle.kts
+10 -1
View File
@@ -173,6 +173,15 @@ dependencies {
implementation(libs.androidx.lifecycle.runtime.compose)
implementation(libs.zxing.embedded)
implementation(libs.markdown.renderer)
implementation(libs.highlights)
implementation(libs.androidx.exifinterface)
// The syntax scanner (Highlighter.kt) is pure logic with no Android imports, which is what
// lets it be tested on the JVM: `./gradlew :androidApp:testDebugUnitTest`. The assertions are
// `kotlin.test`, so the tests name no framework; JUnit is what runs them.
testImplementation(libs.kotlin.test.junit5)
testImplementation(libs.junit.jupiter)
testRuntimeOnly(libs.junit.platform.launcher)
}
// JUnit 6 runs on the Platform, which is not Gradle's default for a Test task.
tasks.withType<Test>().configureEach { useJUnitPlatform() }
app/androidApp/src/main/kotlin/com/example/aiapp/CodeFence.kt
+56 -115
View File
@@ -11,23 +11,14 @@ import androidx.compose.foundation.text.BasicText
import androidx.compose.runtime.Composable
import androidx.compose.runtime.remember
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.semantics.isTraversalGroup
import androidx.compose.ui.semantics.semantics
import androidx.compose.ui.text.AnnotatedString
import androidx.compose.ui.text.SpanStyle
import androidx.compose.ui.text.TextStyle
import androidx.compose.ui.text.buildAnnotatedString
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.unit.dp
import com.mikepenz.markdown.compose.LocalMarkdownColors
import com.mikepenz.markdown.compose.LocalMarkdownDimens
import com.mikepenz.markdown.compose.LocalMarkdownPadding
import com.mikepenz.markdown.model.State
import dev.snipme.highlights.Highlights
import dev.snipme.highlights.model.BoldHighlight
import dev.snipme.highlights.model.ColorHighlight
import dev.snipme.highlights.model.SyntaxLanguage
import org.intellij.markdown.MarkdownElementTypes
import org.intellij.markdown.MarkdownTokenTypes
import org.intellij.markdown.ast.ASTNode
@@ -38,10 +29,10 @@ import org.intellij.markdown.ast.getTextInNode
* A fenced code block in a reply: the code highlighted, on the dark surface every verbatim thing
* sits on, scrolling sideways rather than wrapping.
*
* The renderer's own fence drew the same block in plain text. The lexer that colours a tool call's
* command colours a reply's code the same way, through [highlighted] and one theme, so a `kotlin`
* fence and the Kotlin a tool wrote are the same colours. A fence in a language the lexer has no
* rules for is plain rather than wrongly coloured: [fenceLanguage] answers null for those, and
* The renderer's own fence drew the same block in plain text. The scanner that colours a tool
* call's command colours a reply's code the same way, through [highlighted] and one palette, so a
* `kotlin` fence and the Kotlin a tool wrote are the same colours. A fence in a language [scan] has
* no rules for is plain rather than wrongly coloured: [fenceLanguage] answers null for those, and
* plain is what the reader would have seen before.
*
* Finding the code is still the library's: which children of the node are the fence markers, the
@@ -74,7 +65,7 @@ fun CodeBlock(
}
/**
* The code inside a fence or indented block, and the lexer's language for its info word.
* The code inside a fence or indented block, and the highlighter's language for its info word.
*
* Which children of the node are the fence markers, the language word and the code between them is
* the library's knowledge of the parser, copied from its `MarkdownCodeFence` rather than called:
@@ -85,7 +76,7 @@ fun CodeBlock(
* Null for a fence too short to hold anything -- an unterminated one still arriving, which the
* library skips as invalid.
*/
fun fenceContent(content: String, node: ASTNode): Pair<String, SyntaxLanguage?>? {
fun fenceContent(content: String, node: ASTNode): Pair<String, Language?>? {
val word =
node.findChildOfType(MarkdownTokenTypes.FENCE_LANG)?.getTextInNode(content)?.toString()
val language = fenceLanguage(word)
@@ -111,7 +102,7 @@ fun fenceContent(content: String, node: ASTNode): Pair<String, SyntaxLanguage?>?
@Composable
private fun CodeBlockText(
code: String,
language: SyntaxLanguage?,
language: Language?,
style: TextStyle,
replies: ParsedReplies,
streaming: Boolean,
@@ -136,122 +127,72 @@ private fun CodeBlockText(
}
/**
* The lexer's language for a fence's info word, or null for one it has no lexer for.
* The highlighter's language for a fence's info word, or null for one it has no rules for.
*
* The aliases are what people actually write after the backticks: the file extension as often as
* the name. A word not here gets no colour rather than the nearest lexer's, because a fence
* the name. A word not here gets no colour rather than the nearest language's, because a fence
* coloured by the wrong language's rules looks highlighted and is wrong in a way the reader cannot
* see.
*/
fun fenceLanguage(name: String?): SyntaxLanguage? =
fun fenceLanguage(name: String?): Language? =
FENCE_LANGUAGES[name?.trim()?.lowercase() ?: return null]
private val FENCE_LANGUAGES: Map<String, SyntaxLanguage> =
private val FENCE_LANGUAGES: Map<String, Language> =
mapOf(
"kotlin" to SyntaxLanguage.KOTLIN,
"kt" to SyntaxLanguage.KOTLIN,
"kts" to SyntaxLanguage.KOTLIN,
"rust" to SyntaxLanguage.RUST,
"rs" to SyntaxLanguage.RUST,
"sh" to SyntaxLanguage.SHELL,
"bash" to SyntaxLanguage.SHELL,
"shell" to SyntaxLanguage.SHELL,
"zsh" to SyntaxLanguage.SHELL,
"console" to SyntaxLanguage.SHELL,
"python" to SyntaxLanguage.PYTHON,
"py" to SyntaxLanguage.PYTHON,
"javascript" to SyntaxLanguage.JAVASCRIPT,
"js" to SyntaxLanguage.JAVASCRIPT,
"jsx" to SyntaxLanguage.JAVASCRIPT,
"typescript" to SyntaxLanguage.TYPESCRIPT,
"ts" to SyntaxLanguage.TYPESCRIPT,
"tsx" to SyntaxLanguage.TYPESCRIPT,
"java" to SyntaxLanguage.JAVA,
"c" to SyntaxLanguage.C,
"h" to SyntaxLanguage.C,
"cpp" to SyntaxLanguage.CPP,
"c++" to SyntaxLanguage.CPP,
"cc" to SyntaxLanguage.CPP,
"hpp" to SyntaxLanguage.CPP,
"csharp" to SyntaxLanguage.CSHARP,
"cs" to SyntaxLanguage.CSHARP,
"c#" to SyntaxLanguage.CSHARP,
"go" to SyntaxLanguage.GO,
"golang" to SyntaxLanguage.GO,
"swift" to SyntaxLanguage.SWIFT,
"dart" to SyntaxLanguage.DART,
"ruby" to SyntaxLanguage.RUBY,
"rb" to SyntaxLanguage.RUBY,
"php" to SyntaxLanguage.PHP,
"perl" to SyntaxLanguage.PERL,
"pl" to SyntaxLanguage.PERL,
"coffeescript" to SyntaxLanguage.COFFEESCRIPT,
"coffee" to SyntaxLanguage.COFFEESCRIPT,
"kotlin" to Language.KOTLIN,
"kt" to Language.KOTLIN,
"kts" to Language.KOTLIN,
"rust" to Language.RUST,
"rs" to Language.RUST,
"sh" to Language.SHELL,
"bash" to Language.SHELL,
"shell" to Language.SHELL,
"zsh" to Language.SHELL,
"console" to Language.SHELL,
"python" to Language.PYTHON,
"py" to Language.PYTHON,
"javascript" to Language.JAVASCRIPT,
"js" to Language.JAVASCRIPT,
"jsx" to Language.JAVASCRIPT,
"typescript" to Language.TYPESCRIPT,
"ts" to Language.TYPESCRIPT,
"tsx" to Language.TYPESCRIPT,
"java" to Language.JAVA,
"c" to Language.C,
"h" to Language.C,
"cpp" to Language.CPP,
"c++" to Language.CPP,
"cc" to Language.CPP,
"hpp" to Language.CPP,
"csharp" to Language.CSHARP,
"cs" to Language.CSHARP,
"c#" to Language.CSHARP,
"go" to Language.GO,
"golang" to Language.GO,
"swift" to Language.SWIFT,
"dart" to Language.DART,
"ruby" to Language.RUBY,
"rb" to Language.RUBY,
"php" to Language.PHP,
"perl" to Language.PERL,
"pl" to Language.PERL,
"coffeescript" to Language.COFFEESCRIPT,
"coffee" to Language.COFFEESCRIPT,
"ron" to Language.RON,
"toml" to Language.TOML,
"fish" to Language.FISH,
"json" to Language.JSON,
)
/**
* [code] with its keywords and strings coloured, or plain if there is no language for it.
*
* The lexing is dev.snipme:highlights. The colours are this app's, mapped in [catppuccinSyntax] --
* a library's default theme would be the one place in the app whose palette came from somewhere
* else. Shared by a tool call's input ([ToolInputView]) and a reply's fences ([CodeFence]), so the
* same code is the same colours wherever it appears.
*
* Not a composable, and it takes no colour from the theme, because that is what lets [warm] run it
* off the drawing thread: the syntax palette is fixed, and a fence with no language is plain text
* which needs no colour of its own -- the style the caller draws it with carries that.
*
* Measured on the emulator before it was cached: a two-hundred-line Kotlin fence costs **174ms** to
* lex, and the lazy list charged that again every time the block scrolled back into composition.
* That is the whole reason [ParsedReplies.highlighted] exists rather than a `remember`.
*/
fun highlight(code: String, language: SyntaxLanguage?): AnnotatedString {
if (language == null) return AnnotatedString(code)
val marks =
DebugStats.timed("code highlighted") {
Highlights.Builder(code = code, language = language, theme = catppuccinSyntax())
.build()
.getHighlights()
// highlights 1.1.0's shell lexer answers a quoted glob that looks like a comment
// -- `x '*/a/*'` is the smallest input -- with a span whose end is before its
// start, and AnnotatedString refuses such a range. That crashed the app the
// moment a card holding `-path '*/.git/*'` was opened. Dropped rather than
// clamped: a span the lexer got backwards is not one it knows the colour of.
// Delete when snipme/highlights fixes it.
.filter {
it.location.start in 0..it.location.end && it.location.end <= code.length
}
}
return buildAnnotatedString {
append(code)
marks.forEach { mark ->
when (mark) {
is ColorHighlight ->
addStyle(
SpanStyle(color = Color(mark.rgb or 0xFF000000.toInt())),
mark.location.start,
mark.location.end,
)
is BoldHighlight ->
addStyle(
SpanStyle(fontWeight = FontWeight.Bold),
mark.location.start,
mark.location.end,
)
}
}
}
}
/**
* Every fence in [parse], as the code and language [highlight] will be asked for.
*
* Walks the whole tree rather than the top level: a fence inside a list item or a quote is drawn
* the same way and costs the same to lex.
*/
fun fences(parse: State): List<Pair<String, SyntaxLanguage?>> {
fun fences(parse: State): List<Pair<String, Language?>> {
val success = parse as? State.Success ?: return emptyList()
val out = ArrayList<Pair<String, SyntaxLanguage?>>()
val out = ArrayList<Pair<String, Language?>>()
fun walk(node: ASTNode) {
if (
node.type == MarkdownElementTypes.CODE_FENCE ||
app/androidApp/src/main/kotlin/com/example/aiapp/Highlighter.kt
+318
View File
@@ -0,0 +1,318 @@
package com.example.aiapp
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.text.AnnotatedString
import androidx.compose.ui.text.SpanStyle
import androidx.compose.ui.text.buildAnnotatedString
/** What a span of code is, in the terms the palette has a colour for. */
enum class Kind {
KEYWORD,
STRING,
LITERAL,
COMMENT,
METADATA,
PUNCTUATION,
MARK,
}
/** A run of [Kind] in the code, as a half-open range. */
data class Span(val start: Int, val end: Int, val kind: Kind)
/**
* The colours the highlighter draws with, ours rather than a library's; [catppuccinSyntax] is the
* one instance and lives with the rest of the palette.
*/
data class SyntaxPalette(
val keyword: Color,
val string: Color,
val literal: Color,
val comment: Color,
val metadata: Color,
val punctuation: Color,
val mark: Color,
) {
fun of(kind: Kind): Color =
when (kind) {
Kind.KEYWORD -> keyword
Kind.STRING -> string
Kind.LITERAL -> literal
Kind.COMMENT -> comment
Kind.METADATA -> metadata
Kind.PUNCTUATION -> punctuation
Kind.MARK -> mark
}
}
/**
* [code] with its keywords, strings and comments coloured, or plain if there is no language for it.
*
* Shared by a tool call's input ([ToolInputView]) and a reply's fences ([CodeFence]), so the same
* code is the same colours wherever it appears.
*
* Not a composable, and it takes no colour from the theme, because that is what lets [warm] run it
* off the drawing thread: the syntax palette is fixed, and a fence with no language is plain text
* which needs no colour of its own -- the style the caller draws it with carries that.
*
* The timing is the number the highlighter is judged by: the library this replaced took **174ms**
* on the emulator for a two-hundred-line Kotlin fence, which is why [ParsedReplies.highlighted]
* caches the answer rather than a `remember` inside the fence recomputing it on every scroll back.
*/
fun highlight(code: String, language: Language?): AnnotatedString {
if (language == null) return AnnotatedString(code)
val spans = DebugStats.timed("code highlighted") { scan(code, rulesOf(language)) }
val palette = catppuccinSyntax()
return buildAnnotatedString {
append(code)
spans.forEach { addStyle(SpanStyle(color = palette.of(it.kind)), it.start, it.end) }
}
}
/**
* [code] read once, left to right, into the spans that carry a colour.
*
* One pass with a small state -- in a comment, in a string, or in ordinary code -- rather than a
* locator per token kind over the whole text, which is what the library did and is why it found
* comments before it knew the language: a `#` inside a shell string, a `//` inside a URL and a
* block-comment opener inside a shell glob each commented out the rest of a line that was nothing
* of the sort.
*
* Every span is produced by advancing an index forward, so the result is ordered, non-overlapping
* and inside the code by construction. Nothing here throws: an unterminated string or comment runs
* to the end of the code, which is also what it looks like while a fence is still being written.
*
* In ordinary code the order of recognition is comment, string, attribute, number, word, and
* finally a single punctuation or mark character. Punctuation and marks are coloured only in
* ordinary code, never inside a string or a comment.
*/
fun scan(code: String, rules: Rules): List<Span> = Scanner(code, rules).run()
/** Characters coloured as punctuation, and as marks. Both sets are the ones the library used. */
private const val PUNCTUATION = ",.:;"
private const val MARKS = "()={}<>-+[]|&"
private class Scanner(private val code: String, private val rules: Rules) {
private val spans = ArrayList<Span>()
private var at = 0
fun run(): List<Span> {
while (at < code.length) {
// Every branch that answers true has advanced `at`, so this terminates.
val consumed =
blockComment() ||
lineComment() ||
rawString() ||
characterOrLifetime() ||
string() ||
attribute() ||
number() ||
word() ||
singleCharacter()
if (!consumed) at++
}
return spans
}
private fun emit(start: Int, kind: Kind) {
if (at > start) spans.add(Span(start, at, kind))
}
private fun starts(token: String) = code.startsWith(token, at)
/** Whether a line comment token here opens one; see [Rules.lineCommentsAtWordStart]. */
private fun atWordStart() = at == 0 || code[at - 1].isWhitespace() || code[at - 1] in ";|&("
/** Whether only whitespace stands between the start of this line and here. */
private fun atLineStart(): Boolean {
var back = at - 1
while (back >= 0 && code[back] != '\n') {
if (!code[back].isWhitespace()) return false
back--
}
return true
}
private fun toEndOfLine() {
while (at < code.length && code[at] != '\n') at++
}
/** From an open bracket through the one that matches it, or to the end if none does. */
private fun toMatchingBracket() {
var depth = 0
while (at < code.length) {
when (code[at]) {
'[' -> depth++
']' -> depth--
}
at++
if (depth == 0) return
}
}
private fun blockComment(): Boolean {
val comment = rules.blockComment ?: return false
if (!starts(comment.open)) return false
val start = at
at += comment.open.length
var depth = 1
while (at < code.length && depth > 0) {
// The closer is tried first so that a language whose two delimiters are the same
// string -- CoffeeScript's `###` -- closes rather than nesting forever.
if (starts(comment.close)) {
depth--
at += comment.close.length
} else if (comment.nests && starts(comment.open)) {
depth++
at += comment.open.length
} else {
at++
}
}
emit(start, Kind.COMMENT)
return true
}
private fun lineComment(): Boolean {
if (rules.lineComments.none { starts(it) }) return false
if (rules.lineCommentsAtWordStart && !atWordStart()) return false
val start = at
toEndOfLine()
emit(start, Kind.COMMENT)
return true
}
/** Rust and RON: `b`? `r` `#`* `"` … `"` `#`*, with no escapes inside. */
private fun rawString(): Boolean {
if (!rules.rawStrings) return false
var ahead = at
if (code.getOrNull(ahead) == 'b') ahead++
if (code.getOrNull(ahead) != 'r') return false
ahead++
var hashes = 0
while (code.getOrNull(ahead) == '#') {
ahead++
hashes++
}
if (code.getOrNull(ahead) != '"') return false
val start = at
val closer = "\"" + "#".repeat(hashes)
val closed = code.indexOf(closer, ahead + 1)
at = if (closed < 0) code.length else closed + closer.length
emit(start, Kind.STRING)
return true
}
/** See [Rules.lifetimes]: an apostrophe that is not a character literal opens nothing. */
private fun characterOrLifetime(): Boolean {
if (!rules.lifetimes || code[at] != '\'') return false
val next = code.getOrNull(at + 1) ?: return false
if (next == '\\' || code.getOrNull(at + 2) == '\'') {
quoted(Quote("'", "'", escapes = true))
} else {
at++
}
return true
}
private fun string(): Boolean {
// Longest opener wins, so Kotlin's `"""` is one delimiter rather than an empty string
// followed by a quote. A loop rather than filter/maxBy: this runs at every character of
// ordinary code, and the pair of lists that would allocate is the whole cost of the scan.
var quote: Quote? = null
for (candidate in rules.quotes) {
if (starts(candidate.open) && candidate.open.length > (quote?.open?.length ?: 0)) {
quote = candidate
}
}
quoted(quote ?: return false)
return true
}
private fun quoted(quote: Quote) {
val start = at
at += quote.open.length
while (at < code.length) {
if (quote.escapes && code[at] == '\\' && at + 1 < code.length) {
at += 2
continue
}
if (starts(quote.close)) {
at += quote.close.length
break
}
at++
}
at = at.coerceAtMost(code.length)
emit(start, Kind.STRING)
}
private fun attribute(): Boolean {
val start = at
when (rules.attributes) {
Attributes.NONE -> return false
Attributes.AT_WORD -> {
if (code[at] != '@' || !isWordStart(code.getOrNull(at + 1))) return false
at++
while (at < code.length && isWordPart(code[at])) at++
}
Attributes.HASH_BRACKET -> {
if (code[at] != '#') return false
var ahead = at + 1
if (code.getOrNull(ahead) == '!') ahead++
if (code.getOrNull(ahead) != '[') return false
at = ahead
toMatchingBracket()
}
Attributes.HASH_LINE -> {
if (code[at] != '#' || !atLineStart()) return false
toEndOfLine()
}
Attributes.LINE_BRACKET -> {
if (code[at] != '[' || !atLineStart()) return false
toMatchingBracket()
}
}
emit(start, Kind.METADATA)
return true
}
/**
* A number is a run starting with a digit and carrying on through letters, digits, `_` and `.`
* -- which covers `0xFF`, `1_000`, `1u32` and `3.14` without a grammar for any of them.
*/
private fun number(): Boolean {
if (!code[at].isDigit()) return false
val start = at
while (
at < code.length && (code[at].isLetterOrDigit() || code[at] == '_' || code[at] == '.')
) {
at++
}
emit(start, Kind.LITERAL)
return true
}
private fun word(): Boolean {
if (!isWordStart(code[at])) return false
val start = at
while (at < code.length && isWordPart(code[at])) at++
if (code.substring(start, at) in rules.keywords) emit(start, Kind.KEYWORD)
return true
}
private fun singleCharacter(): Boolean {
val kind =
when (code[at]) {
in PUNCTUATION -> Kind.PUNCTUATION
in MARKS -> Kind.MARK
else -> return false
}
at++
emit(at - 1, kind)
return true
}
}
private fun isWordStart(c: Char?) = c != null && (c.isLetter() || c == '_')
private fun isWordPart(c: Char) = c.isLetterOrDigit() || c == '_'
app/androidApp/src/main/kotlin/com/example/aiapp/Languages.kt
+437
View File
@@ -0,0 +1,437 @@
package com.example.aiapp
/**
* A language the highlighter has rules for.
*
* The names the reader writes after the backticks are aliases onto these; [fenceLanguage] holds
* that table. A word with no entry there is null, and null is drawn plain, because a fence coloured
* by another language's rules looks highlighted and is wrong in a way the reader cannot see.
*/
enum class Language {
C,
COFFEESCRIPT,
CPP,
CSHARP,
DART,
FISH,
GO,
JAVA,
JAVASCRIPT,
JSON,
KOTLIN,
PERL,
PHP,
PYTHON,
RON,
RUBY,
RUST,
SHELL,
SWIFT,
TOML,
TYPESCRIPT,
}
/**
* What [scan] needs to know about one language -- data, not code, so that adding a language is a
* row in [RULES] rather than a branch anywhere.
*
* The two forms that could not be expressed as data are flags here and a few lines in the scanner:
* [rawStrings], because the closing delimiter depends on how many hashes the opener had, and
* [lifetimes], because whether `'` opens anything at all depends on what follows it.
*/
data class Rules(
/** Words drawn as keywords. Only plain words; the scanner cannot reach anything else. */
val keywords: Set<String>,
/** Tokens that open a comment running to the end of the line. */
val lineComments: List<String> = emptyList(),
/**
* Whether [lineComments] count only at the start of a word.
*
* The shells need it: `$#`, `${#x}` and `a#b` are not comments, and greying the rest of those
* lines is one of the mistakes this scanner exists to stop.
*/
val lineCommentsAtWordStart: Boolean = false,
val blockComment: BlockComment? = null,
/** The string forms. The longest opener that matches wins, so `"""` is tried before `"`. */
val quotes: List<Quote> = emptyList(),
val attributes: Attributes = Attributes.NONE,
/** Rust and RON: an optional `b`, `r`, n hashes, `"`, closing at `"` and n hashes. */
val rawStrings: Boolean = false,
/**
* Rust: `'` opens a character literal only when a backslash or one character and a `'` follow.
* Otherwise it is a lifetime or a label and no string starts -- without this, `'a` opens a
* string that runs to the next apostrophe in the block.
*/
val lifetimes: Boolean = false,
)
data class BlockComment(val open: String, val close: String, val nests: Boolean)
/** One string form. [escapes] is whether a backslash escapes the closer (and itself). */
data class Quote(val open: String, val close: String, val escapes: Boolean)
/** What opens a metadata span, of the shapes that exist across these languages. */
enum class Attributes {
NONE,
/** `@` and a word: Kotlin and Java annotations, Python decorators. */
AT_WORD,
/** `#[` or `#![` through the matching `]`: Rust and RON attributes. */
HASH_BRACKET,
/** `#` at the start of a line, to the end of it: the C preprocessor. */
HASH_LINE,
/** `[` at the start of a line through the matching `]`: a TOML table header. */
LINE_BRACKET,
}
/** The rules for [language]. */
fun rulesOf(language: Language): Rules = RULES.getValue(language)
private val C_STYLE = BlockComment("/*", "*/", nests = false)
private val NESTING = BlockComment("/*", "*/", nests = true)
private val DOUBLE = Quote("\"", "\"", escapes = true)
private val SINGLE = Quote("'", "'", escapes = true)
private val TRIPLE_DOUBLE = Quote("\"\"\"", "\"\"\"", escapes = true)
private val TRIPLE_SINGLE = Quote("'''", "'''", escapes = true)
// Lazy because the keyword sets below are top-level properties too, and a file's properties
// initialize in the order they are written: read eagerly here, every set would be null.
private val RULES: Map<Language, Rules> by lazy {
mapOf(
Language.C to
Rules(
keywords = KEYWORDS_C,
lineComments = listOf("//"),
blockComment = C_STYLE,
quotes = listOf(DOUBLE, SINGLE),
attributes = Attributes.HASH_LINE,
),
Language.CPP to
Rules(
keywords = KEYWORDS_CPP,
lineComments = listOf("//"),
blockComment = C_STYLE,
quotes = listOf(DOUBLE, SINGLE),
attributes = Attributes.HASH_LINE,
),
Language.CSHARP to
Rules(
keywords = KEYWORDS_CSHARP,
lineComments = listOf("//"),
blockComment = C_STYLE,
quotes = listOf(DOUBLE, SINGLE),
),
// `###` opens and closes a block comment and `#` opens a line one, which is why the
// scanner tries the block opener first.
Language.COFFEESCRIPT to
Rules(
keywords = KEYWORDS_COFFEESCRIPT,
lineComments = listOf("#"),
blockComment = BlockComment("###", "###", nests = false),
quotes = listOf(TRIPLE_DOUBLE, TRIPLE_SINGLE, DOUBLE, SINGLE),
),
Language.DART to
Rules(
keywords = KEYWORDS_DART,
lineComments = listOf("//"),
blockComment = NESTING,
quotes = listOf(TRIPLE_DOUBLE, TRIPLE_SINGLE, DOUBLE, SINGLE),
attributes = Attributes.AT_WORD,
),
Language.FISH to
Rules(
keywords = KEYWORDS_FISH,
lineComments = listOf("#"),
lineCommentsAtWordStart = true,
// fish's single quotes escape only `\'` and `\\`, which is what "skip the
// character after a backslash" already does.
quotes = listOf(DOUBLE, SINGLE),
),
Language.GO to
Rules(
keywords = KEYWORDS_GO,
lineComments = listOf("//"),
blockComment = C_STYLE,
quotes = listOf(DOUBLE, SINGLE, Quote("`", "`", escapes = false)),
),
Language.JAVA to
Rules(
keywords = KEYWORDS_JAVA,
lineComments = listOf("//"),
blockComment = C_STYLE,
quotes = listOf(DOUBLE, SINGLE),
attributes = Attributes.AT_WORD,
),
Language.JAVASCRIPT to
Rules(
keywords = KEYWORDS_JAVASCRIPT,
lineComments = listOf("//"),
blockComment = C_STYLE,
quotes = listOf(DOUBLE, SINGLE, Quote("`", "`", escapes = true)),
),
Language.JSON to Rules(keywords = KEYWORDS_JSON, quotes = listOf(DOUBLE)),
Language.KOTLIN to
Rules(
keywords = KEYWORDS_KOTLIN,
lineComments = listOf("//"),
blockComment = NESTING,
quotes = listOf(Quote("\"\"\"", "\"\"\"", escapes = false), DOUBLE, SINGLE),
attributes = Attributes.AT_WORD,
),
Language.PERL to
Rules(
keywords = KEYWORDS_PERL,
lineComments = listOf("#"),
quotes = listOf(DOUBLE, SINGLE),
),
Language.PHP to
Rules(
keywords = KEYWORDS_PHP,
lineComments = listOf("//", "#"),
blockComment = C_STYLE,
quotes = listOf(DOUBLE, SINGLE),
attributes = Attributes.AT_WORD,
),
Language.PYTHON to
Rules(
keywords = KEYWORDS_PYTHON,
lineComments = listOf("#"),
quotes = listOf(TRIPLE_DOUBLE, TRIPLE_SINGLE, DOUBLE, SINGLE),
attributes = Attributes.AT_WORD,
),
Language.RON to
Rules(
keywords = KEYWORDS_RON,
lineComments = listOf("//"),
blockComment = NESTING,
quotes = listOf(DOUBLE, SINGLE),
attributes = Attributes.HASH_BRACKET,
rawStrings = true,
),
Language.RUBY to
Rules(
keywords = KEYWORDS_RUBY,
lineComments = listOf("#"),
quotes = listOf(DOUBLE, SINGLE),
),
Language.RUST to
Rules(
keywords = KEYWORDS_RUST,
lineComments = listOf("//"),
blockComment = NESTING,
// No `'` here: [Rules.lifetimes] decides when one opens a character literal.
quotes = listOf(DOUBLE),
attributes = Attributes.HASH_BRACKET,
rawStrings = true,
lifetimes = true,
),
Language.SHELL to
Rules(
keywords = KEYWORDS_SHELL,
lineComments = listOf("#"),
lineCommentsAtWordStart = true,
// A shell's single quotes are literal: `'a\'` is not one string.
quotes = listOf(DOUBLE, Quote("'", "'", escapes = false)),
),
Language.SWIFT to
Rules(
keywords = KEYWORDS_SWIFT,
lineComments = listOf("//"),
blockComment = NESTING,
quotes = listOf(TRIPLE_DOUBLE, DOUBLE),
attributes = Attributes.AT_WORD,
),
Language.TOML to
Rules(
keywords = KEYWORDS_TOML,
lineComments = listOf("#"),
quotes =
listOf(
TRIPLE_DOUBLE,
Quote("'''", "'''", escapes = false),
DOUBLE,
Quote("'", "'", escapes = false),
),
attributes = Attributes.LINE_BRACKET,
),
Language.TYPESCRIPT to
Rules(
keywords = KEYWORDS_TYPESCRIPT,
lineComments = listOf("//"),
blockComment = C_STYLE,
quotes = listOf(DOUBLE, SINGLE, Quote("`", "`", escapes = true)),
attributes = Attributes.AT_WORD,
),
)
}
/**
* The keyword sets.
*
* Every list below other than RON, TOML, fish and JSON came from dev.snipme:highlights 1.1.0
* (`SyntaxTokens.kt`, Apache-2.0), the library this scanner replaced, so that no fence which is
* coloured today turns plain. Entries that are not plain words were dropped -- Kotlin's `as?`,
* `!in` and `!is`, Swift's `#if` family, Ruby's `defined?`, CoffeeScript's `=` and `->` -- because
* the word scanner cannot reach them and the library only matched them by luck.
*/
private fun words(list: String): Set<String> =
list.split(Regex("\\s+")).filterNot(String::isEmpty).toSet()
private val KEYWORDS_C =
words(
"""auto break case char const continue default do double else enum extern float for goto if
int long register return short signed sizeof static struct switch typedef union unsigned
void volatile while"""
)
private val KEYWORDS_CPP =
words(
"""asm auto bool break case catch char class const const_cast continue default delete do
double dynamic_cast else enum explicit export extern false float for friend goto if inline
int long mutable namespace new operator private protected public register reinterpret_cast
return short signed sizeof static static_cast struct switch template this throw true try
typedef typeid typename union unsigned using virtual void volatile wchar_t while"""
)
private val KEYWORDS_CSHARP =
words(
"""abstract as base bool break byte case catch char checked class const continue decimal
default delegate do double else enum event explicit extern false finally fixed float for
foreach goto if implicit in int interface internal is lock long namespace new null object
operator out override params private protected public readonly ref return sbyte sealed short
sizeof stackalloc static string struct switch this throw true try typeof uint ulong unchecked
unsafe ushort using virtual void volatile while"""
)
private val KEYWORDS_COFFEESCRIPT =
words(
"""Infinity NaN and arguments await break by case catch class continue debugger delete defer
default do else export extends false finally for function if import in instanceof is isnt
let loop new no not null of on or package return super switch this throw true try typeof
unless undefined var wait when with yield"""
)
private val KEYWORDS_DART =
words(
"""abstract as assert async await base break case catch class const continue covariant
default deferred do dynamic else enum export extends external factory false final finally
for get if implements import in interface is late library mixin new null on operator part
required rethrow return sealed set show static super switch this throw true try var void
when with while yield"""
)
/**
* fish is not in the library at all, so its fences are drawn plain today. The list is the shell's
* own words, which is what a fish fence is mostly made of.
*/
private val KEYWORDS_FISH =
words(
"""and begin break builtin case command continue else end exec for function if in not or
return switch while set echo test string math read source"""
)
private val KEYWORDS_GO =
words(
"""break case chan const continue default defer else fallthrough false for func go goto if
import interface map package range return select struct switch true type var"""
)
private val KEYWORDS_JAVA =
words(
"""abstract assert boolean break byte case catch char class const continue default do double
else enum extends final finally float for goto if implements import instanceof int interface
long native new null package private protected public return short static strictfp super
switch synchronized this throw throws transient try void volatile while"""
)
private val KEYWORDS_JAVASCRIPT =
words(
"""async await boolean break case catch class const continue debugger default delete do else
enum export extends false finally for function if implements import in instanceof interface
let new null package private protected public return super switch this throw true try typeof
var void while with yield"""
)
private val KEYWORDS_JSON = words("true false null")
private val KEYWORDS_KOTLIN =
words(
"""actual abstract annotation as break by catch class companion const constructor continue
coroutine crossinline data delegate dynamic do else enum expect external false final finally
for fun get if import in infix inline interface internal is lazy lateinit native null object
open operator out override package private protected public reified return sealed set super
suspend tailrec this throw true try typealias typeof val var vararg when while yield"""
)
private val KEYWORDS_PERL =
words(
"""__DATA__ __END__ __FILE__ __LINE__ __PACKAGE__ and cmp continue do else elsif eq eval for
foreach goto gt if last le lt my ne next no not or package redo ref return sub unless until
use while xor"""
)
private val KEYWORDS_PHP =
words(
"""__halt_compiler abstract and array as break callable case catch class clone const continue
declare default die do echo else elseif empty enddeclare endfor endforeach endif endswitch
endwhile eval exit extends final finally fn for foreach function global goto if implements
include include_once instanceof insteadof interface isset list match new or print private
protected public require require_once return static switch throw trait try unset use var
while xor yield"""
)
private val KEYWORDS_PYTHON =
words(
"""False True and as assert async await break class continue def del elif else except finally
for from global if import in is lambda nonlocal not or pass raise return try while with
yield"""
)
/** RON is not in the library either; these are the words a RON file can hold. */
private val KEYWORDS_RON = words("true false Some None inf NaN")
private val KEYWORDS_RUBY =
words(
"""__ENCODING__ __END__ __FILE__ __LINE__ BEGIN END alias and begin break case class def do
else elsif end ensure false for if in module next nil not or redo rescue retry return self
super then true undef unless until when while yield"""
)
private val KEYWORDS_RUST =
words(
"""as async await break const continue crate dyn else enum extern false fn for if impl in
let loop match mod move mut pub ref return Self self static struct super trait true type
union unsafe use where while abstract become box do final macro override priv try typeof
unsized virtual yield"""
)
private val KEYWORDS_SHELL =
words(
"""alias bg bind break builtin caller cd command compgen complete compopt continue declare
dirs disown echo enable eval exec exit export fc fg getopts hash help history jobs kill let
local logout popd printf pushd pwd read readonly return set shift shopt source suspend
test"""
)
private val KEYWORDS_SWIFT =
words(
"""_ associatedtype class deinit enum extension fileprivate func import init inout internal
let open operator private precedencegroup protocol public rethrows static struct subscript
typealias var break case catch continue default defer do else fallthrough for guard if in
repeat return throw switch where while Any as await false is nil self Self super throws true
try associativity convenience didSet dynamic final get indirect infix lazy left mutating none
nonmutating optional override postfix precedence prefix Protocol required right set some Type
unowned weak willSet"""
)
/** TOML is not in the library; `inf` and `nan` are values rather than names, like the booleans. */
private val KEYWORDS_TOML = words("true false inf nan")
private val KEYWORDS_TYPESCRIPT =
words(
"""abstract as asserts await break case catch class const constructor continue debugger
default delete do else enum export extends false finally for from function get if implements
import in infer instanceof interface is keyof let module namespace new null number object
package private protected public readonly require global return set static string super
switch this throw true try type typeof undefined unique unknown var void while with yield"""
)
app/androidApp/src/main/kotlin/com/example/aiapp/Markdown.kt
+1 -2
View File
@@ -55,7 +55,6 @@ import com.mikepenz.markdown.model.markdownAnimations
import com.mikepenz.markdown.model.markdownDimens
import com.mikepenz.markdown.model.markdownPadding
import com.mikepenz.markdown.model.parseMarkdown
import dev.snipme.highlights.model.SyntaxLanguage
import java.util.concurrent.ConcurrentHashMap
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
@@ -670,7 +669,7 @@ class ParsedReplies {
*
* The key carries the language, because the same code lexes differently under two of them.
*/
fun highlighted(code: String, language: SyntaxLanguage?): AnnotatedString =
fun highlighted(code: String, language: Language?): AnnotatedString =
if (language == null) AnnotatedString(code)
else highlights.computeIfAbsent("$language\n$code") { highlight(code, language) }
app/androidApp/src/main/kotlin/com/example/aiapp/Theme.kt
+16 -18
View File
@@ -6,8 +6,6 @@ import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.darkColorScheme
import androidx.compose.runtime.Composable
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.graphics.toArgb
import dev.snipme.highlights.model.SyntaxTheme
/**
* Catppuccin Mocha, as published in `catppuccin/palette`.
@@ -198,24 +196,24 @@ val rawSurface: Color
@Composable get() = Mocha.Crust
/**
* Catppuccin Mocha as a syntax theme, for the highlighter used on a tool call's input.
* Catppuccin Mocha as the highlighter's palette; see [SyntaxPalette].
*
* Here with the rest of the palette rather than beside the code that highlights: a library's own
* theme would otherwise be the one surface in the app whose colours came from somewhere else, and
* the accents below are the same ones every other coloured thing already uses.
* Here with the rest of the palette rather than beside the code that highlights: the colours a
* fence is drawn in are the same accents every other coloured thing in the app already uses, and
* splitting them out would make code the one surface whose palette came from somewhere else.
*
* Not a composable, because [highlight] runs off the drawing thread; these colours never vary with
* the theme.
*/
fun catppuccinSyntax(): SyntaxTheme =
SyntaxTheme(
key = "catppuccin-mocha",
code = Mocha.Text.toArgb(),
keyword = Mocha.Mauve.toArgb(),
string = Mocha.Green.toArgb(),
literal = Mocha.Peach.toArgb(),
comment = Mocha.Overlay0.toArgb(),
metadata = Mocha.Yellow.toArgb(),
multilineComment = Mocha.Overlay0.toArgb(),
punctuation = Mocha.Subtext0.toArgb(),
mark = Mocha.Sky.toArgb(),
fun catppuccinSyntax(): SyntaxPalette =
SyntaxPalette(
keyword = Mocha.Mauve,
string = Mocha.Green,
literal = Mocha.Peach,
comment = Mocha.Overlay0,
metadata = Mocha.Yellow,
punctuation = Mocha.Subtext0,
mark = Mocha.Sky,
)
/**
app/androidApp/src/main/kotlin/com/example/aiapp/ToolInput.kt
+3 -4
View File
@@ -11,7 +11,6 @@ import androidx.compose.runtime.remember
import androidx.compose.ui.Modifier
import androidx.compose.ui.text.font.FontFamily
import androidx.compose.ui.unit.dp
import dev.snipme.highlights.model.SyntaxLanguage
import org.json.JSONObject
/**
@@ -27,7 +26,7 @@ data class ToolInput(
/** The thing that will actually be run or read, if this tool has one. */
val subject: String?,
/** The language [subject] is written in, for highlighting. */
val language: SyntaxLanguage?,
val language: Language?,
/** The tool's own one-line summary, when it wrote one. */
val description: String?,
/**
@@ -50,9 +49,9 @@ data class ToolInput(
* from being the special case that gets its own code path. Unknown tools fall through to "no
* subject, everything is rest", which is what the card always did.
*/
private val SUBJECTS: Map<String, Pair<String, SyntaxLanguage?>> =
private val SUBJECTS: Map<String, Pair<String, Language?>> =
mapOf(
"Bash" to ("command" to SyntaxLanguage.SHELL),
"Bash" to ("command" to Language.SHELL),
"Read" to ("file_path" to null),
"Write" to ("file_path" to null),
"Edit" to ("file_path" to null),
app/androidApp/src/test/kotlin/com/example/aiapp/HighlighterTest.kt
+240
View File
@@ -0,0 +1,240 @@
package com.example.aiapp
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
/**
* What the scanner colours, asserted as the text under each span rather than as offsets, so a
* failure prints the code that was got wrong instead of a pair of numbers.
*
* Most of these are the mistakes dev.snipme:highlights 1.1.0 made -- the library this scanner
* replaced -- measured against it directly before it was removed. They are here rather than in the
* `highlights-repro.sh` script they came from because a case that only a script can ask about is a
* case nobody asks about.
*/
class HighlighterTest {
/** Every span of [kind] in [code], as the text it covers. */
private fun spans(code: String, language: Language, kind: Kind): List<String> =
scan(code, rulesOf(language))
.filter { it.kind == kind }
.map { code.substring(it.start, it.end) }
private fun assertSpans(
code: String,
language: Language,
kind: Kind,
vararg expected: String,
) {
assertEquals(expected.toList(), spans(code, language, kind), "$kind in: $code")
}
// The four inputs the repro script asked the library about, and what it answered.
@Test
fun `a quoted glob is one string, not a comment`() {
// The library answered a span whose end preceded its start here, which crashed the app.
assertSpans("x '*/a/*'", Language.SHELL, Kind.STRING, "'*/a/*'")
assertSpans("x '*/a/*'", Language.SHELL, Kind.COMMENT)
}
@Test
fun `a find with globs has no comment in it`() {
val code = "find . -path '*/.git/*' -prune -o -name '*.kt' -print"
assertSpans(code, Language.SHELL, Kind.STRING, "'*/.git/*'", "'*.kt'")
assertSpans(code, Language.SHELL, Kind.COMMENT)
}
@Test
fun `a URL does not comment out the rest of a shell line`() {
val code = "curl https://example.com/x && echo done"
assertSpans(code, Language.SHELL, Kind.COMMENT)
assertSpans(code, Language.SHELL, Kind.KEYWORD, "echo")
}
@Test
fun `a URL inside a Kotlin string stays a string`() {
val code = "val url = \"https://example.com\"\nfun f() = 1"
assertSpans(code, Language.KOTLIN, Kind.COMMENT)
assertSpans(code, Language.KOTLIN, Kind.STRING, "\"https://example.com\"")
assertSpans(code, Language.KOTLIN, Kind.KEYWORD, "val", "fun")
}
// Attributes, which the library greyed out as comments.
@Test
fun `a Rust attribute is metadata and the struct after it still colours`() {
val code = "#[derive(Debug)]\nstruct A { b: u8 }"
assertSpans(code, Language.RUST, Kind.METADATA, "#[derive(Debug)]")
assertSpans(code, Language.RUST, Kind.COMMENT)
assertSpans(code, Language.RUST, Kind.KEYWORD, "struct")
}
@Test
fun `an inner Rust attribute closes at its own bracket`() {
val code = "#![allow(dead_code)]\nfn f() {}"
assertSpans(code, Language.RUST, Kind.METADATA, "#![allow(dead_code)]")
assertSpans(code, Language.RUST, Kind.KEYWORD, "fn")
}
@Test
fun `a C preprocessor line is metadata rather than a comment`() {
val code = "#include <stdio.h>\nint main() { return 0; }"
assertSpans(code, Language.C, Kind.METADATA, "#include <stdio.h>")
assertSpans(code, Language.C, Kind.COMMENT)
assertSpans(code, Language.C, Kind.KEYWORD, "int", "return")
}
@Test
fun `a Kotlin annotation is metadata`() {
assertSpans("@Composable fun f() {}", Language.KOTLIN, Kind.METADATA, "@Composable")
}
// Strings whose contents the library read as code.
@Test
fun `a hash inside a Kotlin string is not a comment`() {
val code = "val c = \"#FF0000\"\nval d = 1"
assertSpans(code, Language.KOTLIN, Kind.COMMENT)
assertSpans(code, Language.KOTLIN, Kind.STRING, "\"#FF0000\"")
}
@Test
fun `an apostrophe inside a Kotlin string does not open one`() {
val code = "val a = \"don't\"\nval b = \"x\""
assertSpans(code, Language.KOTLIN, Kind.STRING, "\"don't\"", "\"x\"")
}
@Test
fun `a Rust lifetime does not open a string but a character literal does`() {
val code = "fn f<'a>(x: &'a str) { let c = 'x'; }"
assertSpans(code, Language.RUST, Kind.STRING, "'x'")
}
@Test
fun `an escaped quote is inside the Rust character literal`() {
assertSpans("let c = '\\'';", Language.RUST, Kind.STRING, "'\\''")
}
@Test
fun `a Rust raw string keeps its inner quotes`() {
val code = "let s = r#\"a \"quoted\" b\"#;"
assertSpans(code, Language.RUST, Kind.STRING, "r#\"a \"quoted\" b\"#")
}
@Test
fun `a Kotlin triple quoted string is one string`() {
assertSpans(
"val s = \"\"\"a \"b\" c\"\"\"",
Language.KOTLIN,
Kind.STRING,
"\"\"\"a \"b\" c\"\"\"",
)
}
@Test
fun `a shell single quoted string takes no escapes`() {
// `\` is literal inside shell single quotes, so the string ends at the next apostrophe.
assertSpans("echo 'a\\' b", Language.SHELL, Kind.STRING, "'a\\'")
}
// Comments.
@Test
fun `Rust and Kotlin nest block comments`() {
val code = "/* a /* b */ c */ x"
assertSpans(code, Language.RUST, Kind.COMMENT, "/* a /* b */ c */")
assertSpans(code, Language.KOTLIN, Kind.COMMENT, "/* a /* b */ c */")
}
@Test
fun `C ends a block comment at the first close`() {
assertSpans("/* a /* b */ c */ x", Language.C, Kind.COMMENT, "/* a /* b */")
}
@Test
fun `a shell comment starts only at a word boundary`() {
val code = "\${#x} \$# a#b # real"
assertSpans(code, Language.SHELL, Kind.COMMENT, "# real")
}
@Test
fun `a hash anywhere is a Python comment`() {
assertSpans("x = 1 # note", Language.PYTHON, Kind.COMMENT, "# note")
}
// TOML, which the library has no rules for at all.
@Test
fun `a TOML table header is metadata and a hash in a value is not a comment`() {
val code = "[server]\ncolour = \"#FF0000\"\nport = 8080 # the real one"
assertSpans(code, Language.TOML, Kind.METADATA, "[server]")
assertSpans(code, Language.TOML, Kind.STRING, "\"#FF0000\"")
assertSpans(code, Language.TOML, Kind.COMMENT, "# the real one")
assertSpans(code, Language.TOML, Kind.LITERAL, "8080")
}
@Test
fun `a RON attribute and its values colour`() {
val code = "#![enable(implicit_some)]\n(count: 3, on: true)"
assertSpans(code, Language.RON, Kind.METADATA, "#![enable(implicit_some)]")
assertSpans(code, Language.RON, Kind.KEYWORD, "true")
assertSpans(code, Language.RON, Kind.LITERAL, "3")
}
@Test
fun `an unknown fence language is drawn plain`() {
assertEquals(null, fenceLanguage("brainfuck"))
assertEquals("+[-]", highlight("+[-]", fenceLanguage("brainfuck")).text)
}
@Test
fun `every alias the fence table knows has rules`() {
Language.entries.forEach { rulesOf(it) }
}
/**
* The scanner must never throw and must never answer a span the code does not contain: the
* library's reversed range is exactly the shape that crashed a card, and a fence still being
* written is an unterminated string or comment on every keystroke.
*/
@Test
fun `spans stay inside the code for every language and every nasty input`() {
val nasty =
listOf(
"",
"'",
"\"",
"\"unterminated",
"/* unterminated",
"###",
"#",
"#![",
"[",
"r#\"",
"\\",
"'''",
"\"\"\"",
"0x",
"1.2.3",
"a#b//c/*d*/'e\"f",
"\n\n \n",
)
for (language in Language.entries) {
for (code in nasty) {
val spans = scan(code, rulesOf(language))
spans.forEach {
assertTrue(
it.start in 0..it.end && it.end <= code.length,
"$language answered $it for ${code.replace("\n", "\\n")}",
)
}
assertEquals(
spans.sortedBy { it.start },
spans,
"$language answered spans out of order for ${code.replace("\n", "\\n")}",
)
}
}
}
}