Skip to content

testing.md

Latest commit

 

History

History
66 lines (46 loc) · 6.53 KB

testing.md

File metadata and controls

66 lines (46 loc) · 6.53 KB
Raw

Codexa testing

Local pytest invocations, the bench opt-in marker, and the GitHub Actions / SonarCloud wiring.

← README · Decision records: ADR 0004 (why the suite is offline-only), ADR 0005 (the seam modules tests patch), ADR 0010 (the reset entry point fixtures drain)

Table of contents

  1. Testing
  2. Continuous integration

Testing

pytest                              # full suite (~4-5 min, ~6.6k collected cases)
pytest --cov=codexa --cov-report=term --cov-fail-under=90
HYPOTHESIS_PROFILE=deep pytest $(ls tests/*fuzz*.py tests/security/test_*.py)  # deep fuzz
pytest -m bench                     # opt-in indexer ramp-up benchmark
ruff check codexa tests             # lint
pyright codexa tests                # static type check

The default suite is offline-only by design (mirroring the product's no-default-egress invariant, ADR 0004): CODEXA_DISABLE_HF=1 and TOKENIZERS_PARALLELISM=false are set in tests/conftest.py, NLTK calls in metadata.entities are mocked, and SentenceTransformer / Chroma / Ollama / paddleocr / tesseract are all replaced with deterministic stand-ins.

Hypothesis property tests run inside the normal suite with the ci profile (deadline=None, cheap example budget). The current suite shape is roughly 5.5k def test_* functions plus 180+ @given properties; pytest's expanded collection is about 6.6k executed cases. For adversarial sweeps, run HYPOTHESIS_PROFILE=deep pytest ...; the scheduled/manual .github/workflows/deep-fuzz.yml lane applies that profile without slowing every pull request. That lane discovers its targets at run time — every tests/**/*fuzz*.py plus tests/security/test_*.py — so a new fuzz file is picked up automatically (no YAML edit needed). The ls glob above mirrors that discovery for a local run.

The bench marker is deselected by default. Tests under it spawn the real indexer subprocess against BENCH_CORPUS_DIR (default /path/to/corpus) and measure throughput + system pressure; they are the test-runner counterpart of scripts/bench_indexer.py. See tests/test_bench_indexer.py for the full env-var override list.

⚠️ Do not run two pytest processes in parallel against this repo. Each pulls in chromadb + pymupdf + sentence-transformers and a second concurrent run can OOM the host.

Conventions

  • Phase regression tests — every shipped phase lands one regression file under tests/test_phase{N}_{slug}_{YYYY_MM_DD}.py covering the closed TODO row(s). The file's docstring lists the closed IDs + the contract pinned. Phase 85+ examples: ROB-* atomic writes, WIRE-18/19/20/21/23/24/25, PROMPT-5..10, FAITH-5/7+COH-8, PERF-30/31/33/34/35, FAITH-8/9, RETQ-7/8/9/10, COH-9..12+BOT-28/29/30, CACHE-2/3/8/9/12+SCAL-15/16.
  • AppContext.reset between tests — when a test mutates a wired cache, drain the chain via AppContext.current().reset() (or the per-cache clear_<name>_cache() helper) in a fixture teardown so subsequent tests start cold (ADR 0010).
  • Type ignores on duck-typed test stubs — SimpleNamespace / Mock objects swapped in for Session / Turn / BM25Searcher etc. carry a # type: ignore[arg-type] (or similar) marker; pyright's stub-vs-real-type mismatch is intentional.

Continuous integration

GitHub Actions runs on every push and pull request to develop / main. The workflow has five jobs:

  • lock — uv lock --check, so a pyproject.toml edit that was never re-locked fails before anything is installed and every later job resolves from the committed uv.lock.
  • lint — ruff check codexa tests on Python 3.12.
  • typecheck — uv run --frozen pyright on Python 3.12. This is a hard gate; a type regression blocks merge.
  • test — pytest with --cov-fail-under=90 on Python 3.12 (XML coverage uploaded as an artifact). The same job then runs the bundled repo-doc retrieval quality gate: codexa eval docs-baseline --save … --min-recall 0.5 --min-ndcg-at-5 0.3 --max-regression 0.02 against the repo docs (deterministic bundled cases, no operator gold file needed) and uploads the retrieval-quality-baseline summary artifact. Fixed floors catch absolute quality drops; --max-regression compares against the committed .codexa-quality/retrieval_baseline_summary.json. The bundled relevance lists are explicitly open (closed_gold: false), so precision is neither published nor gated. A second, optional retrieval quality eval gate runs only when an operator copies tests/data/eval_qrel.example.yml to tests/data/eval_qrel.yml and provides a matching indexed corpus (skipped otherwise). Set closed_gold: true and add --min-precision only when expected_sources exhaustively judges that corpus; omitted closed_gold defaults to true for legacy files.
  • sonar — SonarQube Cloud analysis (needs test), ingesting the coverage XML. Skipped on fork PRs (no SONAR_TOKEN).

codexa eval tune-hybrid --cases … is not a CI job — it is an operator-run command. It selects the fusion grid winner by mean NDCG@5, using mean MRR@5 as the tie-breaker; the cutoff stays fixed while rrf_k is swept. Open and closed qrels both contribute ranking metrics; only closed qrels contribute the retained precision diagnostic.

A second workflow, deep-fuzz.yml, runs the Hypothesis Deep Fuzz job on a schedule and on manual dispatch — it is not part of the per-push gate.

Concurrency is set so a new push to a branch cancels any in-flight run, and the same OMP_NUM_THREADS/MKL_NUM_THREADS/TOKENIZERS_PARALLELISM discipline used locally is applied as workflow-level env vars to keep CI minutes predictable. See .github/workflows/ci.yml.

sonar-project.properties configures SonarQube / SonarCloud: production code is codexa/; tests/ and examples/ are declared as test sources but excluded from the main quality analysis so test idioms (long parametrize lists, repeated fixture scaffolding) don't surface as code smells. The coverage XML emitted by the CI pytest run feeds Sonar via sonar.python.coverage.reportPaths.