Let a machine have more than one llama.cpp
A model whose kernels are not upstream needs the fork that has them, and the ordinary models still want the ordinary build. Anything under `~/.local/share/ai-app/llama/<name>/` -- `llama-server`, or the `bin/llama-server` a `cmake --install --prefix` leaves -- is now discovered beside the one on PATH and becomes a provider called `llama-cpp-<name>`, with its own router, preset and model settings. That keeps the module's security property rather than bending it: the phone still names no command, because what runs is still decided by what somebody put on the machine. Each probe answer is tagged with what was asked for, since two of these are now the same program under different paths. Flash attention joins the model settings (`flash-attn` in the preset). llama.cpp's `auto` stays the default; the control is for a model whose publisher asks for `on` outright, which Prism ML's ternary Bonsai does. Verified against the fork built into that directory: discovery answers `llama-cpp-prism`, the child server is started with `--flash-attn on`, and Ternary-Bonsai-2-27B PTQ1_0 loads and answers through a session.
This commit is contained in:
1 parent
df48a334f7
commit
7e7910083c
6 files changed
+220
-16
No files matched your search
@@ -174,6 +174,38 @@ Two models are downloaded under `~/.local/share/ai-app/models`:
|
||||
projector being kept out of the models a provider offers. Qwen3-0.6B beside
|
||||
it is the other half of that rig -- the model that answers `refused`.
|
||||
|
||||
**A second llama.cpp is installed here, and it is the rig for a custom
|
||||
build.** `~/.local/share/ai-app/llama/prism/` is Prism ML's fork
|
||||
(`prism` branch, `~/repos/llama.cpp-prism`, Vulkan, `cmake --install
|
||||
--prefix`), so discovery finds it as a provider called `llama-cpp-prism`
|
||||
beside the ordinary `llama-cpp`. It is what exercises that mechanism at all,
|
||||
and it serves `prism-ml/Ternary-Bonsai-2-27B-gguf` -- ternary packings stock
|
||||
llama.cpp rejects as unknown types. Rebuild it with
|
||||
`cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON
|
||||
-DCMAKE_INSTALL_RPATH='$ORIGIN/../lib'`, about six minutes at `-j8`.
|
||||
|
||||
**Which Bonsai packing runs on the GPU is the backend's question, not the
|
||||
model's.** Measured 2026-09-21 with `llama-bench -p 512 -n 64 -r 2 -fa 1
|
||||
-ngl 99` on the free card:
|
||||
|
||||
| packing | backend | pp512 | tg64 |
|
||||
| --- | --- | ---: | ---: |
|
||||
| `PTQ1_0`, 5.53 GiB | Vulkan | 519 t/s | 7.5 t/s |
|
||||
| `PQ2_0`, 7.21 GiB | **CPU**, 8 cores | unfinished after 9 min | -- |
|
||||
|
||||
The fork's Vulkan port covers `PTQ1_0` only -- shaders, a `mul_mat_vec` and
|
||||
the FWHT included -- so `PQ2_0` has no kernel there and every matmul falls
|
||||
back to the CPU. That reads exactly like a stuck load: the process sits at
|
||||
700% CPU for minutes with the card idle. On CUDA and HIP it is the other way
|
||||
round, since `mmq.cu` guards `PTQ1_0` out of the HIP build (it wants Turing
|
||||
MMA) and leaves `PQ2_0` in. **ROCm cannot be tested in this VM**: there is no
|
||||
`/dev/kfd`, because the GPU here is virtio-gpu rather than a passed-through
|
||||
card.
|
||||
|
||||
7.5 tok/s is the honest speed of that Vulkan kernel, against 42 for the
|
||||
IQ3_S 27B beside it -- smaller weights, slower decode. Nothing is
|
||||
misconfigured; the fork's fast kernels are CUDA and Metal.
|
||||
|
||||
**Do not test with a 2-bit quant**: the IQ2_XXS of the 0.6B produces fluent
|
||||
nonsense, which reads exactly like a broken driver — `llama-cli` produces the
|
||||
same from the file directly, which is how to tell the two apart in a hurry.
|
||||
|
||||
Reference in new issue
Block a user