Skip to content

performance.md

Latest commit

 

History

History
132 lines (98 loc) · 14.6 KB

performance.md

File metadata and controls

132 lines (98 loc) · 14.6 KB
Raw

Codexa performance

GPU acceleration, indexer throughput knobs, and the ramp-up benchmark used to tune a new host.

← README · Decision records: ADR 0007 (the concurrency shape these knobs tune), ADR 0010 (the cache contract behind every cache listed here)

Table of contents

  1. GPU acceleration
  2. Performance knobs
  3. Benchmarking

GPU acceleration

cfg.embeddings.device: "auto" (the default) probes cuda → mps → xpu → cpu in priority order and picks the first backend that's available. Override with an explicit value when needed:

Hardware device value install hint
NVIDIA "cuda" (or "cuda:0" etc) pip install torch --index-url https://download.pytorch.org/whl/cu121
AMD (ROCm) "cuda" (yes — ROCm masquerades under the CUDA API) pip install torch --index-url https://download.pytorch.org/whl/rocm6.0
Apple Silicon "mps" default pip install torch already includes MPS
Intel Arc / iGPU "xpu" pip install intel-extension-for-pytorch
Anything else "cpu" n/a

The Diagnostics tab reports which backend is currently active. For the local LLM provider, llama-cpp-python needs to be rebuilt with the matching backend flag (LLAMA_CUDA=1, LLAMA_HIPBLAS=1, LLAMA_METAL=1, LLAMA_VULKAN=1); the default wheel is CPU-only.

Performance knobs

Indexer-side

These knobs tune the two-lane producer/consumer design recorded in ADR 0007 — parallel chunk workers feeding one serial main-process embed consumer.

  • Auto-batch — embeddings.auto_batch: false to disable. Halves the encoding batch at ≥60 % RAM (or ≥80 % VRAM on CUDA), doubles back to the configured ceiling at ≤40 % RAM (and ≤55 % VRAM), hard floor of 1. Hysteresis keeps the batch from oscillating turn-to-turn; RAM is re-polled every 8 batches so the per-batch cost stays negligible.
  • Cache stats — every run logs embedding_cache_summary (hits, misses, hit rate, total entries/bytes); codexa-cache-stats prints it on demand.
  • Cross-file chunk dedup — within a single embed() call, identical strings (boilerplate / license headers / EPUB nav chunks) are encoded once and the result is reused across every position. Layered on top of the on-disk shard cache.
  • mtime-first change detection — the manifest records each file's (mtime_ns, size). On the next run, files whose stat is unchanged skip both the rehash and the comparison entirely; the change_detection log event reports rehash_skipped so you can see the win. For restored backups, rsync -a / cp -p, network/shared-FS corpora, or cross-host workflows where content can change while mtime and size are preserved, set indexer.trust_mtime: false to rehash known files every run.
  • SQLite manifest — set metadata.manifest_file to a .db / .sqlite path on corpora bigger than ~50k files. Saves become a single transaction of REPLACE INTOs instead of an O(n) full-file rewrite. The connection pool (manifest_sqlite_pool, Phase 82) keeps WAL-mode handles warm across the partial-save path so per-row UPSERTs don't reopen the DB.
  • Cores-aware worker caps — chunker / OCR process counts come from cpu_worker_count(kind, configured=…) which floors to 1 core reserved for the OS, applies a per-kind cpu-fraction cap, and a per-kind hard ceiling. Chunkers scale as cores // 4 (floor 2, ceiling 8); OCR is capped at 2. Setting a higher workers: in config doesn't break anything — it's silently clamped, with a workers_downscaled WARNING emitted so the downscale is visible.
  • CPU-only chunkers — the chunking workers are spawned with CUDA_VISIBLE_DEVICES="" and 1-thread BLAS caps so they don't fight the main process embedder for the GPU or saturate every core. The embedder runs in the main process and respects embeddings.main_thread_cap independently.
  • OOM backoff (CUDA) — model.encode() is wrapped in a single-retry path: on a CUDA RuntimeError whose message looks OOM-shaped, the embedder halves its batch (embed_oom_backoff WARNING), calls torch.cuda.empty_cache(), and retries the same texts split in halves. Floors at size 1 so a single oversized chunk still surfaces the failure instead of infinite-halving.
  • Effective-budget pre-flight log — every run emits an effective_budget event with cores, chunker_workers, ocr_workers, embedder_main_thread_cap, embedder_batch_size, and embedder_device. Pair it with the optional ramp-up benchmark in scripts/bench_indexer.py when tuning a new host.

Search-side

Search concurrency stays at named orchestration and pipeline-stage seams (ADR 0003); the entries below are those bounded parallel points plus per-surface memos.

  • A/B rewrite parallel retrieval — when cfg.search.query_rewrite.ab_fuse is enabled, original and rewritten queries run concurrently through separate pipeline instances on a dedicated two-worker pool before their deterministic RRF merge (PERF-56). The pool is deliberately separate from nested sub-query retrieval so two outer branches cannot consume the workers their own expansion futures need.
  • HyDE parallel fan-out — cfg.search.hyde.n controls how many hypothetical paragraphs the dispatcher draws for the dense-embedding RRF merge. The draws fire concurrently via ThreadPoolExecutor (Phase 88 PERF-31) so n=3 no longer pays 3× cold-Ollama round-trip on the latency-critical path. n<=1 skips the executor (no pool-spawn overhead).
  • ChatbotObjective memo — ChatbotObjective.from_cfg is memoised on cfg_fingerprint(cfg.chatbot) (PERF-33). The chat-thread render + escalation render + sidebar all hit this per-paint; the memo keeps the cost to one walk per distinct cfg snapshot. Cap=8 FIFO + reset_chatbot_objective_cache wired into AppContext.reset.
  • Jaccard scorer query-hoist — HistoryCompressor.relevance_compressed (BOT-9) hoists the query tokenisation outside the per-turn loop when the default Jaccard scorer is used (PERF-34). Custom scorers keep the legacy (query, body) signature so their contract is unchanged.
  • _last_user_queries — walks reversed(history) directly instead of reversed(list(history)) (PERF-35). Native Turn lists carry __reversed__ so no copy is needed; the early-break bound (len(prior) >= k) caps the walk.
  • Pilot store tombstone overlay — PilotRunStore.update (PERF-32, Phase 96) appends an _op="update" overlay line instead of re-reading + re-writing the whole JSONL. Lazy compaction (drop overlays + rewrite consolidated base records via utils.io.atomic_write_text) fires once the overlay-to-base ratio crosses 2.0 AND base_n ≥ 4. Operator UI status toggles drop from O(n) read + O(n) write to O(1) append.
  • Sub-query parallel — search.semantic_retrieve._retrieve_candidates batches HyDE / query-decomp / multilingual / synonym sub-query embeddings into one embedder.embed([...]) call, then dispatches each store.query concurrently with order-preserved results before the RRF merge (PERF-30).
  • Matryoshka view — cfg.embeddings.matryoshka_target_dim truncates and re-normalizes supported catalog models (for example BAAI/bge-m3) on both the index-write path and the query path, so stored vectors and query vectors keep the same reduced dimension (WIRE-21 / ML-7 / VEC-1). Changing matryoshka_target_dim requires a codexa-index --force-reindex: the value is baked into the persisted vectors, so querying an index built at one target with a different target compares mismatched dimensionalities and corrupts retrieval. The embedding fingerprint records the effective stored dim (ML-30), so a mismatched query now trips an embedding_fingerprint_mismatch warning + BM25 fallback rather than returning silent garbage — but the correct fix is always a force-reindex.

Cache contract

The contract and the AppContext lifecycle behind it are recorded in ADR 0010. Wired caches track the six-point AGENTS.md contract (key shape · size cap · TTL or invalidation · lock · reset hook · hit/miss counters + <name>_cache_stats()). The ONNX vision-session cache and the ColBERT scorer cache (CACHE-33: maxsize=2 + close-on-evict, mirroring the reranker cache) both use bounded close-on-evict caches. LazyKeyedCache (codexa.utils.caching, Phase 91) ships maxsize= LRU eviction + ttl_s= per-entry expiry + on_evict= close hook for the multi-MB / multi-GB cases:

  • search._query_llm_cache — TtlCache(maxsize=256, ttl_s=600) shared by rewrite, decomposition, and HyDE. Keys include query/history/locale plus the effective category-specific transform settings and full LLM provider chain; a model or transform edit misses immediately, while unrelated search tuning preserves the warm entry.
  • search.fusion._BM25_SEARCHER_CACHE — LazyKeyedCache("bm25_corpus", maxsize=8, on_evict=close_searcher). Releases the postings mmap on LRU eviction so A/B operator sweeps don't leak (SCAL-15 / CACHE-8).
  • metadata.entities._BACKEND_CACHE — LazyKeyedCache("entities_backend", maxsize=2, on_evict=close_pipeline). Caps at one warm primary + one warm-on-swap secondary so hot-swapping cfg.metadata.entities.backend / model doesn't pin multi-GB pipelines per swap (SCAL-16 / CACHE-9).
  • search.multilingual._CORPUS_LANGUAGE_CACHE — BoundedLRU(maxsize=16) keyed by (resolved_manifest_path, mtime_ns). Manifest replacement or generation publication changes the key; a load lock provides one parse per identity across concurrent queries, AppContext.reset() clears it, and registry stats expose hits, misses, size, cap, and zero TTL (PERF-59).
  • ingestion.ocr_cache — one recipe-aware OcrCache per OCR worker/process, shared by image and whole-PDF calls. Keys combine the streamed content hash with output-changing OCR parameters; ocr.cache.max_entries defaults to 100,000, oldest rows from the largest shard are pruned every prune_every_writes writes (default 256) and at deterministic teardown, and the cache registry exposes aggregate process hits, misses, size, cap, prune count, and reset (CACHE-23).
  • ui.sessions_base._RETENTION_PURGE_CACHE — TtlCache(maxsize=128, ttl_s=300) keyed by (session_store_path, retention_days). A module lock makes lookup + claim atomic across Streamlit reruns; TTL expiry, capacity eviction, or AppContext.reset() invalidates claims, and registry stats expose hits, misses, size, cap, and TTL (CACHE-22).

Every cache hooks into AppContext.reset() so a --reset CLI sweep + every test fixture drains them through one entry point.

Benchmarking

scripts/bench_scan_handoff.py exercises the startup scan/change handoff with 200,000 synthetic 100-byte paths and no filesystem fixture. It reports incremental peak RSS and exits non-zero above the 80 MiB budget; the test suite also caps growth from 100,000 to 200,000 paths at 45 MiB. Production keeps one path/stat/scope inventory, skips duplicate full-corpus stat, seen-path, and unchanged-hash structures, and releases the inventory before normal queue construction:

python scripts/bench_scan_handoff.py
python scripts/bench_scan_handoff.py --files 100000 --max-peak-mib 80

scripts/bench_run_ledger.py extends that probe through physical file/directory identity tracking, change outcomes, and lane/format routing. These current-run structures live in one temporary SQLite ledger with a 2 MiB page cache; query-backed sequences feed the chunk pools and the ledger is closed and unlinked on every normal/error exit. The authoritative manifest remains the sole in-memory O(N) corpus map:

python scripts/bench_run_ledger.py --mode fresh
python scripts/bench_run_ledger.py --mode unchanged

The CI gate runs fresh and 200,000-row-manifest overlap probes at 100,000 and 200,000 files. Each mode permits at most 25 MiB incremental handoff RSS and 12 MiB growth across that doubling; ledger bytes are reported separately because bounded memory deliberately trades RAM for temporary disk.

scripts/bench_generation_clone.py reproduces the SD-12 copy/hardlink bookkeeping evidence without an operator corpus. It reports logical, copied, hardlinked, and fallback bytes plus median clone and publish/GC time. File/byte accounting is deterministic; timing is host-specific and informational:

uv run python scripts/bench_generation_clone.py --runs 5 --buckets 32

The backend decision and reopening thresholds are recorded in ADR 0001.

scripts/bench_indexer.py ramps the indexer through a series of (workers, main_thread_cap, batch_size, sample) rungs against a small symlinked sample of the corpus. Each rung runs codexa-index as a subprocess inside its own watchdog:

  • kill if available RAM drops below --floor-pct × total (or --floor-min-mb, whichever is higher) — default 10 % / 2 GB,
  • kill if system pressure stays above --pressure-max for 3 s — default 0.92,
  • kill if 1-min loadavg exceeds ncpu × --loadavg-mult for --loadavg-dwell-s — default 1.2 / 10 s, the "host responsiveness" gate that fires before the desktop UI gets laggy,
  • kill on --rung-timeout-s (default 600 s).

Per-rung metrics land in logs/bench/<utc>/results.json (peak RSS, peak system pressure, mean / peak CPU %, child-tree CPU %, peak 1-min loadavg, files-and-chunks/sec, run duration, kill reason). The default ladder bumps one knob at a time so when a rung is killed you know exactly which lever crossed the line.

Benchmark directories also contain per-rung configs, logs, and sample_paths.txt. Treat logs/bench/<utc>/ as operator-private output when the source corpus path is private; the current writer records sampled absolute paths for reproducibility.

python scripts/bench_indexer.py --help            # all knobs
python scripts/bench_indexer.py                   # default 14-rung text ramp
python scripts/bench_indexer.py --ocr-mode        # gentler 8-rung OCR ramp
python scripts/bench_indexer.py --rungs my.json   # custom ladder

--ocr-mode swaps in an OCR-specific ladder (sweeps engine, pdf_dpi, ocr_workers, sample size), restricts the fixture to image extensions (.png/.jpg/.tiff/…), and bumps the per-rung timeout to 20 min since OCR is seconds-per-page. The default engine is tesseract; paddle / both rungs need paddlepaddle>=3.0 plus a working paddleocr install before they can be exercised.