Fix #1273: bug: Thread leak in searcher._retrieve_paths causes container thread exhaustion#1969
Open
Memtensor-AI wants to merge 1 commit into
Open
Fix #1273: bug: Thread leak in searcher._retrieve_paths causes container thread exhaustion#1969Memtensor-AI wants to merge 1 commit into
Memtensor-AI wants to merge 1 commit into
Conversation
Replace per-request ContextThreadPoolExecutor instantiation across the tree-text search pipeline with two shared class-level executors plus a 30 s per-future timeout. The previous code opened a fresh pool inside `with` blocks at four call sites (`_retrieve_paths`, `_retrieve_from_long_term_and_user`, `_retrieve_from_tool_memory`, `_deduplicate_rawfile_results`); whenever a worker hung on a slow Neo4j / embedding / HTTP call, `shutdown(wait=True)` never returned and the worker threads accumulated unboundedly across requests, eventually exhausting the container's pthread limit (reporter observed 8744 threads; /search returned HTTP 200 with empty results, /chat HTTP 503, even docker exec failing). Searcher now owns `_search_executor` (max_workers=10, prefix `search`) for outer A-F retrieval paths and `_search_subtask_executor` (max_workers=10, prefix `search-sub`) for the nested sub-paths and the rawfile-dedup. The two-pool split prevents nested-submission deadlock; total threads are bounded at 20 per Searcher regardless of request volume or downstream latency. Every `Future.result(...)` and the `as_completed(...)` iterator now carry `timeout=SEARCH_TASK_TIMEOUT_SECONDS` (30 s default); on TimeoutError the call site logs a warning and skips that contribution so the caller can still return. Closes #1273
Memtensor-AI
requested review from
CarltonXiang,
MatthewZhuang,
World-controller and
syzsunshine219
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixed unbounded thread accumulation in the tree-text search pipeline (issue #1273) by replacing four per-request
with ContextThreadPoolExecutor(...)blocks insrc/memos/memories/textual/tree_text_memory/retrieve/searcher.pywith two shared, class-level executors created once inSearcher.__init__:_search_executor(max_workers=10, prefixsearch) for the outer A–F retrieval paths and_search_subtask_executor(max_workers=10, prefixsearch-sub) for the nested sub-paths in_retrieve_from_long_term_and_user,_retrieve_from_tool_memory, and_deduplicate_rawfile_results. The two-pool split eliminates nested-submission deadlock; total worker threads are now bounded at 20 per Searcher regardless of request volume or downstream latency.Every
Future.result(...)and theas_completed(...)iterator in the four affected methods now carrytimeout=SEARCH_TASK_TIMEOUT_SECONDS(30 s default). Onconcurrent.futures.TimeoutErrorthe call site logs a warning identifying the path and skips that contribution, so a single stuck Neo4j/embedding/HTTP call can no longer block the caller indefinitely while threads accumulate.Tests: added 12 regression tests in
tests/memories/textual/test_tree_searcher_thread_pool.pycovering executor singleton identity, pool sizing/prefixes, no-per-request-construction guarantee, bounded thread count under 16 sequential searches, timeout propagation toFuture.result, hanging-subtask recovery, and per-method use of the correct pool. Real pytest output: 12 new + 4 pre-existing searcher tests pass; the wider tests/memories/textual suite is green (70 passed); all other observed failures across the repo are pre-existing ondev-20260624-v2.0.22(confirmed viagit stash). Ruff lint and format both clean.Related Issue (Required): Fixes #1273
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Automated tests are pending.
Checklist
@MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller please review this PR.
Reviewer Checklist