Download GGUF models from HuggingFace, watchably
The first half of the llama.cpp work Bryan asked for: browse HuggingFace, fetch a model, and see how far it has got from any device. The design is dev-updater's build-progress shape with the four changes its author recommended after living with it, since a model download is an hour where a build is two minutes: - **A run has an id.** Without one "not downloading" means three different things -- finished, never started, or someone else's run ended while you were away -- and over an hour that ambiguity is certain rather than theoretical. A device compares the run it was watching to the run reported now. - **Outcomes outlive their run**, so a phone that was asleep at the moment of completion can still find out what happened. - **Cancel exists.** Retrofitting cancellation into a blocking loop is miserable, and several gigabytes over someone's data plan is not something to have no answer for. - **Progress is bytes, not a parsed marker.** We own the loop, so it counts directly; `total` is whatever Content-Length said and nothing else, and stays absent when the server sends none rather than becoming a bar drawn from a guess. The download owns its own thread rather than the blocking pool, which exists for short work. It resumes through HTTP Range, and trusts the 206 rather than the request -- a server that ignores Range answers 200 with the whole file, and appending to that would corrupt it. `truncate(false)` on the open is load-bearing for the same reason and says so. Searching is proxied through the server rather than done from the phone, because the app trusts exactly one certificate -- this one -- and the machine that must do the downloading is also the one whose view of what exists matters. Verified against the real HuggingFace, not a mock: searched, listed a repository's GGUFs, downloaded 234 MB with live byte progress, cancelled mid-flight, confirmed the partial survived, restarted and watched it resume at 162 MB rather than 0, and let it finish. The result's sha256 matches the one HuggingFace publishes for that file, so the resume is byte-correct and not merely the right length. llama.cpp then loaded it and ran inference. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017xn8nHw1tw1R6PtiY1eEtw
This commit is contained in:
1 parent
4004183acf
commit
9d29776f02
3 files changed
+658
No files matched your search
@@ -456,3 +456,112 @@ async fn send_event(
|
||||
tx.send(SseEvent::default().id(entry.seq.to_string()).data(data))
|
||||
.await
|
||||
}
|
||||
|
||||
/// Separate router because its state is the model store, like `usage`'s.
|
||||
///
|
||||
/// Keys are `owner/repo/file.gguf` and so contain slashes, which is why
|
||||
/// nothing here puts one in the path: a key travels in the body or a query
|
||||
/// string, and the routes stay addressable without escaping rules nobody
|
||||
/// would get right from a phone.
|
||||
pub fn models_router(store: Arc<crate::models::ModelStore>) -> Router {
|
||||
Router::new()
|
||||
.route("/models", get(list_models))
|
||||
.route("/models/search", get(search_models))
|
||||
.route("/models/files", get(repo_files))
|
||||
.route("/models/download", post(start_download))
|
||||
.route("/models/cancel", post(cancel_download))
|
||||
.route("/models/delete", post(delete_model))
|
||||
.with_state(store)
|
||||
}
|
||||
|
||||
/// What this machine has and what it is fetching, in one answer.
|
||||
///
|
||||
/// Both together deliberately: a phone showing the model list needs both
|
||||
/// to draw one screen, and two routes would let it render a model as
|
||||
/// absent while its download sits at 99%.
|
||||
#[derive(serde::Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
struct ModelsResponse {
|
||||
local: Vec<crate::models::LocalModel>,
|
||||
downloads: Vec<crate::models::DownloadStatus>,
|
||||
}
|
||||
|
||||
async fn list_models(
|
||||
State(store): State<Arc<crate::models::ModelStore>>,
|
||||
) -> Result<axum::Json<ModelsResponse>, ApiError> {
|
||||
let listing = tokio::task::spawn_blocking(move || ModelsResponse {
|
||||
local: store.list(),
|
||||
downloads: store.downloads(),
|
||||
})
|
||||
.await
|
||||
.context("listing models panicked")?;
|
||||
Ok(axum::Json(listing))
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct SearchQuery {
|
||||
q: String,
|
||||
}
|
||||
|
||||
async fn search_models(
|
||||
Query(query): Query<SearchQuery>,
|
||||
) -> Result<axum::Json<Vec<crate::models::RemoteRepo>>, ApiError> {
|
||||
// Blocking HTTP, like the usage fetch: off the request workers.
|
||||
let found = tokio::task::spawn_blocking(move || crate::models::search(&query.q))
|
||||
.await
|
||||
.context("model search panicked")?
|
||||
.map_err(bad_request)?;
|
||||
Ok(axum::Json(found))
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct RepoQuery {
|
||||
repo: String,
|
||||
}
|
||||
|
||||
async fn repo_files(
|
||||
State(store): State<Arc<crate::models::ModelStore>>,
|
||||
Query(query): Query<RepoQuery>,
|
||||
) -> Result<axum::Json<Vec<crate::models::RemoteFile>>, ApiError> {
|
||||
let files = tokio::task::spawn_blocking(move || crate::models::files(&query.repo, &store))
|
||||
.await
|
||||
.context("listing repository files panicked")?
|
||||
.map_err(bad_request)?;
|
||||
Ok(axum::Json(files))
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct DownloadRequest {
|
||||
repo: String,
|
||||
file: String,
|
||||
}
|
||||
|
||||
/// Starts a download, or rejoins the one already running for that model.
|
||||
async fn start_download(
|
||||
State(store): State<Arc<crate::models::ModelStore>>,
|
||||
axum::Json(body): axum::Json<DownloadRequest>,
|
||||
) -> Result<axum::Json<crate::models::DownloadStatus>, ApiError> {
|
||||
let status = store.start(&body.repo, &body.file).map_err(bad_request)?;
|
||||
Ok(axum::Json(status))
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct KeyRequest {
|
||||
key: String,
|
||||
}
|
||||
|
||||
async fn cancel_download(
|
||||
State(store): State<Arc<crate::models::ModelStore>>,
|
||||
axum::Json(body): axum::Json<KeyRequest>,
|
||||
) -> Result<axum::Json<crate::models::DownloadStatus>, ApiError> {
|
||||
let status = store.cancel(&body.key).map_err(bad_request)?;
|
||||
Ok(axum::Json(status))
|
||||
}
|
||||
|
||||
async fn delete_model(
|
||||
State(store): State<Arc<crate::models::ModelStore>>,
|
||||
axum::Json(body): axum::Json<KeyRequest>,
|
||||
) -> Result<StatusCode, ApiError> {
|
||||
store.delete(&body.key).map_err(bad_request)?;
|
||||
Ok(StatusCode::NO_CONTENT)
|
||||
}
|
||||
Reference in new issue
Block a user