Files counted as added without verifying they linked, causing re-upload and re-embed every cycle

[ArthurVardevanyan]

Issue

oikb uploads files to Open WebUI with fire-and-forget semantics — it POSTs to /files/ and treats the HTTP 200 response as success, even though the actual KB linkage (chunking + embedding) happens in the background and can fail silently.

Symptom

Files are reported as added every sync cycle, even though they never land in the KB. This causes:

• The same files to be re-uploaded and re-embedded every hour
• SyncResult reports N added when zero files were actually added
• Orphaned file rows, orphaned vector collections, and wasted embedding cost

Root cause

OikbClient.upload_file does not pass process_in_background=false. Open WebUI returns 200 OK immediately, then the background task fails (e.g. 400: Duplicate content detected). Since oikb never verifies whether the file actually linked, it counts the upload as success and the next sync sees the file is still missing → adds it again.

[tobru]

Confirming this at scale on oikb 0.4.0 + Open WebUI 0.11.3, syncing ~1,380 Nextcloud notes hourly into one KB.

Over ~19 hours the Open WebUI file table grew from 1,125 to 16,874 rows, ~800 new rows per sync, ~80% of all rows in state failed. Every failed row carries the same error:

Duplicate content detected. Please provide unique content to proceed.

The loop is self-sustaining: the re-uploaded file is rejected because its content is already in the KB, so it never links, so the next sync's diff still doesn't see it, so it uploads again. oikb's own counters show the split, the most recent sync reported files_added=796, unmodified=567 against ~1,363 source notes, i.e. dedupe works for ~40% of files and fails permanently for the rest.

Secondary effect worth noting: the Open WebUI container eventually died with OSError: [Errno 24] Too many open files while serving a static asset, and has been restart-looping under the accumulated load.

Cleanup required deleting 15,486 orphaned file rows via the API. With sync stopped the instance is stable.

[gbr-lat]

Confirming on oikb 0.5.0 + Open WebUI 0.11.4 (Docker), with server-side evidence.

Repro: KB reset, file A synced alone and indexed; then A' added (different bytes,
same extracted text) and oikb sync -> "Sync complete: 1 added, 1 unchanged".
The server logs then show Duplicate content detected, and
Failed to link file <id> to knowledge <kb>: 400.

Server state afterwards (file table): the rejected upload has data.status = failed,
data.error = "Duplicate content detected...", hash = NULL, and is never linked
to the KB. --dry-run keeps listing it under + Added. Running oikb sync once
more created a new failed row for the same file, so one orphan accumulates per
sync (same mechanism as #106).