Let a machine have more than one llama.cpp
A model whose kernels are not upstream needs the fork that has them, and the ordinary models still want the ordinary build. Anything under `~/.local/share/ai-app/llama/<name>/` -- `llama-server`, or the `bin/llama-server` a `cmake --install --prefix` leaves -- is now discovered beside the one on PATH and becomes a provider called `llama-cpp-<name>`, with its own router, preset and model settings. That keeps the module's security property rather than bending it: the phone still names no command, because what runs is still decided by what somebody put on the machine. Each probe answer is tagged with what was asked for, since two of these are now the same program under different paths. Flash attention joins the model settings (`flash-attn` in the preset). llama.cpp's `auto` stays the default; the control is for a model whose publisher asks for `on` outright, which Prism ML's ternary Bonsai does. Verified against the fork built into that directory: discovery answers `llama-cpp-prism`, the child server is started with `--flash-attn on`, and Ternary-Bonsai-2-27B PTQ1_0 loads and answers through a session.
This commit is contained in:
1 parent
df48a334f7
commit
7e7910083c
6 files changed
+220
-16
No files matched your search
@@ -675,6 +675,7 @@ fn section(found: &Model, settings: &BTreeMap<String, String>) -> String {
|
||||
("contextSize", "ctx-size"),
|
||||
("gpuLayers", "n-gpu-layers"),
|
||||
("threads", "threads"),
|
||||
("flashAttention", "flash-attn"),
|
||||
// How far ahead the draft head guesses. Not defaulted: 2 measured 7%
|
||||
// faster than llama.cpp's 3 on this machine's GPU, once, which is a
|
||||
// reason to make the knob reachable and not a reason to move it for
|
||||
|
||||
Reference in new issue
Block a user