Let a machine have more than one llama.cpp

A model whose kernels are not upstream needs the fork that has them, and
the ordinary models still want the ordinary build. Anything under
`~/.local/share/ai-app/llama/<name>/` -- `llama-server`, or the
`bin/llama-server` a `cmake --install --prefix` leaves -- is now
discovered beside the one on PATH and becomes a provider called
`llama-cpp-<name>`, with its own router, preset and model settings.

That keeps the module's security property rather than bending it: the
phone still names no command, because what runs is still decided by what
somebody put on the machine. Each probe answer is tagged with what was
asked for, since two of these are now the same program under different
paths.

Flash attention joins the model settings (`flash-attn` in the preset).
llama.cpp's `auto` stays the default; the control is for a model whose
publisher asks for `on` outright, which Prism ML's ternary Bonsai does.

Verified against the fork built into that directory: discovery answers
`llama-cpp-prism`, the child server is started with `--flash-attn on`,
and Ternary-Bonsai-2-27B PTQ1_0 loads and answers through a session.
This commit is contained in:
iris-ai committed 2026-09-21 01:37:36 -04:00
1 parent df48a334f7
commit 7e7910083c
6 files changed
+220 -16

No files matched your search

+32
View File
@@ -174,6 +174,38 @@ Two models are downloaded under `~/.local/share/ai-app/models`:
projector being kept out of the models a provider offers. Qwen3-0.6B beside
it is the other half of that rig -- the model that answers `refused`.
**A second llama.cpp is installed here, and it is the rig for a custom
build.** `~/.local/share/ai-app/llama/prism/` is Prism ML's fork
(`prism` branch, `~/repos/llama.cpp-prism`, Vulkan, `cmake --install
--prefix`), so discovery finds it as a provider called `llama-cpp-prism`
beside the ordinary `llama-cpp`. It is what exercises that mechanism at all,
and it serves `prism-ml/Ternary-Bonsai-2-27B-gguf` -- ternary packings stock
llama.cpp rejects as unknown types. Rebuild it with
`cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON
-DCMAKE_INSTALL_RPATH='$ORIGIN/../lib'`, about six minutes at `-j8`.
**Which Bonsai packing runs on the GPU is the backend's question, not the
model's.** Measured 2026-09-21 with `llama-bench -p 512 -n 64 -r 2 -fa 1
-ngl 99` on the free card:
| packing | backend | pp512 | tg64 |
| --- | --- | ---: | ---: |
| `PTQ1_0`, 5.53 GiB | Vulkan | 519 t/s | 7.5 t/s |
| `PQ2_0`, 7.21 GiB | **CPU**, 8 cores | unfinished after 9 min | -- |
The fork's Vulkan port covers `PTQ1_0` only -- shaders, a `mul_mat_vec` and
the FWHT included -- so `PQ2_0` has no kernel there and every matmul falls
back to the CPU. That reads exactly like a stuck load: the process sits at
700% CPU for minutes with the card idle. On CUDA and HIP it is the other way
round, since `mmq.cu` guards `PTQ1_0` out of the HIP build (it wants Turing
MMA) and leaves `PQ2_0` in. **ROCm cannot be tested in this VM**: there is no
`/dev/kfd`, because the GPU here is virtio-gpu rather than a passed-through
card.
7.5 tok/s is the honest speed of that Vulkan kernel, against 42 for the
IQ3_S 27B beside it -- smaller weights, slower decode. Nothing is
misconfigured; the fork's fast kernels are CUDA and Metal.
**Do not test with a 2-bit quant**: the IQ2_XXS of the 0.6B produces fluent
nonsense, which reads exactly like a broken driver — `llama-cli` produces the
same from the file directly, which is how to tell the two apart in a hurry.