Repository navigation
Fix/sanitize non utf8 source bytes - #571
Open
seanmaher-scrape wants to merge 2 commits into
Open
seanmaher-scrape wants to merge 2 commits into
seanmaher-scrape wants to merge 2 commits into
Conversation
/healthz always returns 200 with no DB check, so it can't serve as a real Kubernetes readiness probe -- a pod could be marked ready and receive traffic while unable to reach Postgres. /readyz pings the database (via a new exported Client.Ping, reusing the existing GORM connection pool rather than opening a new connection) with a 2-second timeout, and returns 503 if unreachable. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes CBST-2721: source files saved in a non-UTF-8 encoding (e.g. Windows-1252/Latin-1, common in older codebases with characters like a-umlaut/o-umlaut/u-umlaut) passed through extraction unmodified, then failed the Postgres INSERT outright when persisted (client/server encoding is UTF-8, SQLSTATE 22021). Because persistChunks propagates that error with a hard return (no continue), a single bad file aborted the rest of that commit's chunk extraction entirely. SourceText.Text() now runs the extracted content through strings.ToValidUTF8, replacing invalid byte sequences with U+FFFD instead of passing them through raw. Confirmed no existing UTF-8 validation exists anywhere in the extraction/persistence path before this change (grepped the whole module for unicode/utf8 usage).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
SourceText.Text() (the default plain-text extractor in infrastructure/extraction/source.go) only checks for null bytes to detect binary content — it does no encoding validation. string(content) reinterprets raw bytes without transcoding, so a source file saved in a non-UTF-8 encoding (e.g. Windows-1252/Latin-1 — common in older codebases with characters like ä/ö/ü) passes through extraction and chunking untouched.
This fails downstream at persistence: the enrichment store's INSERT hits Postgres with invalid UTF-8, which Postgres's UTF-8 client/server encoding rejects outright (SQLSTATE 22021, invalid byte sequence for encoding "UTF8").
The impact is worse than a single dropped file: ChunkFiles.persistChunks propagates that error with a hard return, and its caller's file loop has no continue around it — so one non-UTF-8 file aborts the rest of that commit's chunk extraction entirely, not just its own enrichment.
Confirmed there's no existing UTF-8 handling anywhere in the extraction/persistence path (grepped the whole module for unicode/utf8 usage — the only hit is unrelated query-token counting in domain/search/token_budget.go).
Fix
SourceText.Text() now runs the extracted content through strings.ToValidUTF8, replacing invalid byte sequences with U+FFFD instead of passing them through raw. Minimal, no charset auto-detection needed — just stops the hard failure.
Testing
go build ./... and go vet ./... clean.
New tests in source_test.go: plain UTF-8 content unchanged, binary detection still works, and a real non-UTF-8 byte sequence (0xe4 0xf6 0xfc) gets sanitized to valid UTF-8.
Ran the full infrastructure/extraction package suite — 4 pre-existing failures in pdfium_text_renderer_test.go (Windows/WebAssembly path handling, unrelated) confirmed present with this change reverted too, so not a regression introduced here.