A resume splices: it appends bytes from wherever the server is now onto
whatever is already on disk. If the file changed upstream in between, the
result is exactly the failure that survives every cheap check -- the right
number of bytes, the wrong contents, and no error anywhere. HuggingFace
files do get updated, so this is a real path rather than a theoretical one.
`If-Range` is the header for this and would have been the tidy answer, but
HuggingFace's CDN ignores it: probed today, a deliberately stale validator
still answers 206 with the ranged bytes rather than 200 with the whole
body. So the check is done here instead. A partial now has an identity file
beside it holding the ETag it was written against, written before the body
so an interrupted download still knows what it is a piece of. On resume,
the response's ETag is compared against it, and a mismatch throws the
partial away and asks again from zero. A partial with no identity at all is
not resumed either -- it could be a fragment of anything.
The sha256 HuggingFace publishes is now also checked before the file gets
its real name, so a bad one is never offered to be run. That is
belt-and-braces after the above rather than the primary defence, which is
the right order: detecting corruption after downloading gigabytes is worth
far less than not creating it.
Verified by planting one: a 60 MB partial of random bytes with an
identity file naming a revision that does not exist. The server logged
"changed upstream since the partial was written -- starting again",
restarted from zero rather than appending, and the finished file's sha256
matches the published one. Repeated the honest resume too -- cancel at
145 MB, restart, resume at 162 MB, correct hash.
Thanks to dev-updater's session for the If-Range idea and for saying to
confirm the CDN honours it rather than assume, which is exactly what it
turned out not to do.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017xn8nHw1tw1R6PtiY1eEtw
irisandClaude Opus 5
6f149398d0