Skip to content

refactor: make corpus operations functional [AAI-766] - #1359

Closed
ypicard wants to merge 7 commits into
Future-House:mainfrom
ypicard:yann/functional-corpus
Closed

ypicard wants to merge 7 commits into
Future-House:mainfrom
ypicard:yann/functional-corpus

Conversation

@ypicard

@ypicard ypicard commented Sep 29, 2026 •

Copy link
Copy Markdown

Superseded by platform PR #5898, which keeps input preservation in e14c and uses released PaperQA. The platform change requires no PaperQA or server API migration.

Avoid whole-corpus copies by returning updated documents without changing inputs.

Breaking API change: add, retrieve, evidence, and query operations return (docs, result); clear, delete, and merge_docs return docs. Callers retain the returned documents to reuse embeddings and indexes. Query operations also leave input sessions unchanged. Use the standalone merge_docs(*corpora) function to combine independent results and retain the first corpus’s retrieval state.

Qdrant corpus retrieval raises NotImplementedError; use NumPy. Direct Qdrant operations remain available. Custom vector stores must implement fork().

Corpus benchmark: 2,000 chunks, 256-dimensional embeddings, median of 5 runs. Baseline includes the old deep copy. Memory means peak traced allocations, not RSS; excludes network and LLM calls.

Operation Before (ms) After (ms) Before (MiB) After (MiB)
Insert 143.100 0.034 10.074 0.023
Cold retrieval 153.023 13.032 13.403 8.090
Warm retrieval 152.004 3.378 13.363 4.053

Comment thread src/paperqa/docs.py
return updated, session


def merge_docs(*corpora: Docs) -> Docs:

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👀

@ypicard ypicard closed this Sep 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant