kvstore: stamp checkpoints with a behavioral tokenizer fingerprint - #1020
Open
nazerim wants to merge 2 commits into
Open
kvstore: stamp checkpoints with a behavioral tokenizer fingerprint#1020nazerim wants to merge 2 commits into
nazerim wants to merge 2 commits into
Conversation
Persisted KV checkpoints encode their conversation as a token history whose bytes, on future requests, are re-tokenized by whatever tokenizer the running engine has. When tokenizer code or data changes upstream (the JoyAI pre-tokenizer has seen several rule fixes), existing checkpoint token histories become unreproducible: every later request diverges from the restored session at the first changed merge boundary, reloads the same stale file by text key, and re-imports the stale tokenization. The visible symptom is a permanent per-turn live-miss + disk-reload + partial-re-prefill loop for every session resumed with old checkpoints (observed in production: 29 consecutive turns, ~38k tokens re-prefilled each turn, session permanently pinned to a stale frontier). A text-equality check at load cannot detect this class without also rejecting legitimate preserved-reasoning continuations, whose stored sampled tokens intentionally differ from a whole-text re-tokenization of their bytes - the bridge's premise. The correct discriminator is the engine identity that produced the tokens. This commit adds a behavioral fingerprint: an append-only set of probe strings (apostrophe-punctuation seams, digit runs, indentation, newline joins, UTF-8 letters, special-token edges) is run through the real tokenizer once, and the resulting ids hashed together with the token table. It covers tokenizer data AND code changes with no human-maintained version constant. Checkpoint trailers now begin with a TOKFP section carrying the fingerprint; the loader pre-checks it by a 16-byte seek to the trailer, before touching the payload (session untouched), and mismatches fall back to the normal cold path, which rewrites a fresh stamped file under the same key. Unstamped legacy files keep today's trusted behavior. Verified in a live agent session: stamped files restore at full depth (451k tokens, 38-token re-prefill); a synthetic probe change (simulated upstream tokenizer edit) makes every old stamped file reject in microseconds with a clear warning and the session self-heals on the next store.
ds4_str holds a pointer and a length; hashing the raw array therefore hashed process mapping state, not tokenizer data. Same-binary restarts can reproduce the allocator layout by luck, but any different mapping would silently change the fingerprint and reject valid stamped checkpoints after an ordinary restart - the opposite of the feature's purpose. Hash content instead: n_vocab, then every token's length + bytes in id order, then the merge-rank table (used/cap plus each used slot's key bytes and rank; open-addressed slot order is a deterministic function of the GGUF merge list). The behavioral probes still ride on top, so code changes are covered even when data is not. Validated live (Apple M5 Max): fingerprint e467e62ceeadf47e identical across separate process launches of the unchanged model, and a stamped checkpoint restored at full depth (32768-token grid, 69 ms load) across a stop/start cycle with zero mismatch lines. A single-byte token mutation on an APFS clone of the model (TABLE->TASLE, same length, outside every probe string) moves the fingerprint to 5f83c7a25a3b184e, proving data sensitivity independent of probe coverage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #983 (GitHub refuses reopen after the rebase force-push; same branch, rebased onto current main - merge cost was one trivial hunk).
Problem. A checkpoint saved by a different tokenizer build (or a different gguf vocab - both happened to me in production: a model-file migration produced four foreign-vocab restores that the text-keyed tiers trusted, each one a multi-thousand-token silent corruption or a pinned re-prefill loop) restores without complaint and then poisons every following turn.
Fix. The engine computes
ds4_engine_tokenizer_fingerprint()once: a hash over n_vocab, every token's length+bytes in id order, the merge-rank table contents, and a small append-only behavioral probe set run through the real tokenizer - covering data and pre-tokenizer code change with no human-maintained version constant. Every save writes it as the first trailer section (TOKFP, auxiliary ext bit 1<<5, so key_kind and existing consumers are untouched); the loader pre-checks it with a 16-byte seek before touching the payload and rejects mismatched files, leaving the session untouched so the ordinary fallback chain walks to the next candidate or a cold rebuild, which rewrites fresh under the key. Unstamped legacy files remain trusted (status quo ante).Design note carried from #983's review: @JordiPosthumus caught that the first version hashed the raw
ds4_strarray (pointers) - restart-stable on my machine by luck, wrong on any other allocator layout. The shipped version hashes content only; regressions: two fresh processes print identical fp (e467e62ceeadf47e), a single-byte flip in a clone of the model file changes it (5f83c7a25a3b184e).Validation (M5 Max, optimized build clean, ds4-server unit suite green on the rebased head): production soak since #983: the stamp paid for itself during a model-file migration - one bounded cold rebuild, zero silent corruption after. Cross-restart restore of a 32768-token grid: 69 ms, zero mismatches.