fix(search_packages): relax nix package search constraints and score on the tool side

[Scott McMaster (scottmcmaster)]

Summary

This is for issue 617 -- another try at making the search_packages tool return good results with less churn (my white whale).

The ^ and $ anchors overconstrain the nix search so you can see this kind of churn:

With this change:

1. We no longer anchor but just let the query term tokens pass through to nix search. This gets us a lot more candidate results. So...
2. Then we collect those across all channels (no longer just each channel individually).
3. And do some heuristics to score them. We can improve these heuristics later, do Manhattan distances or other kinds of spelling/sounds-like correction, let an LLM do the scoring, or whatever -- but I think what's here is pretty effective and also fast/easy.

Then you can see much happier results more like this (note the scoring implied in the agent's evolution logs in the UI):

In addition to not missing pretty obvious things, by return just MORE results, we give the agent the opportunity to make better decisions in future search and edit steps.

Test Plan

Some new unit tests, plus manual testing.

• No test plan needed

Docs

• Docs updated (companion PR in darkmatter/nixmac-web: #___)
• No docs update needed

[Scott McMaster (scottmcmaster)]

• fix(search_packages): relax nix package search constraints and score on the tool side #647 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[prelint]

Caution

The process_results sort closure calls relevance_score(a, &a.name) — scoring each result against its own name rather than the user's search query.

^{apps/native/src-tauri/src/evolve/search_packages.rs:265}

Caution

When use_regex = true, the new code calls regex::escape(query) before passing it to nix search, which escapes all regex metacharacters and turns the query into a literal string search.

^{apps/native/src-tauri/src/evolve/search_packages.rs:84}

2 finding(s) posted as inline comments.

[prelint]

Search relaxation pools all channels and scores by heuristic

Product decisions in this change

1. Search terms are no longer anchored to exact package name boundaries, so each search returns a wider candidate set instead of failing when the query does not match a package name exactly.

The anchored query was the root cause of the search-churn problem shown in the PR screenshots. A user asking for "spotify" got zero results because the real attribute name includes a prefix or suffix, forcing the agent to retry with variant spellings. Removing anchors and escaping the terms instead is the correct fix. The terms are still regex-escaped, so a query like python3.10 does not accidentally match python3X10 via a regex dot. This decision is clearly correct and the fix is narrow.

2. Non-regex query terms are regex-escaped before passing to nix search, replacing the old approach of adding `^` and `$` anchors.

Package names regularly contain ., -, and +. Without escaping, those characters act as regex wildcards and return unrelated matches. Escaping preserves the user's intent while still allowing the broader substring match the PR wants. The test build_search_queries_no_regex confirms the terms now pass through without anchors and without treating special characters as operators. This is the right tradeoff.

3. Package search results are ranked by a heuristic scoring function before the agent sees them, with weights that favor exact name matches over prefix matches over description matches.

The prior scoring bug (every result scored identically because the function used each result's own name as the query) is now fixed. The user query flows correctly from the call site into relevance_score, and the new test process_results_ranks_by_user_query verifies query-driven ordering. The remaining concern is that the weights (1000 for exact name, 900 for normalized, 500 for prefix, 300 for all-terms-in-name, 5 per description term) were chosen by inspection and have not been validated against the specific eval cases that showed search churn. A weight inversion -- for example, a description-heavy package outranking the correct exact match -- would send the agent in the wrong direction silently. The approach is sound; the weights need a validation pass against the eval suite before they are treated as reliable.

4. Results from all channels are pooled into a single ranked set before the agent sees them, so any channel can surface the best match regardless of its position in the channel list.

The old behavior stopped collecting once the result quota filled, which meant later channels never contributed even when they held a better match. Pooling all channels before scoring fixes that structural bias. The concern is for users who register a pinned or corporate channel and expect that channel's version of a package to win when the query is unambiguous. The scoring function has no channel-preference signal, so a nixpkgs result with a slightly higher name score beats the pinned channel's result. This behavior change has not been acknowledged in the PR, and the prior open question about whether any user segment relies on channel-ordering priority remains unanswered.

Option	What users get	What it costs	Effort to change later
Current (pooled, score-ordered)	Best name match from any channel wins	Pinned-channel priority silently lost	Add channel-weight field to scoring
Prior (channel-ordered, first-fills)	Pinned channel always surfaces first	Correct package from later channel never appears	Already removed
Scored with channel-weight bonus	Pinned channel wins on ties	Slightly more complex scoring	Low

5. Each channel can contribute up to 100 candidates to the pool, and the agent receives no results until every channel search completes.

The ceiling of 100 per channel prevents a generic query from producing thousands of candidates for the scoring pass. However, the prior early-exit that stopped collecting once the agent's result quota filled is now gone. Before this change, a two-channel search on a common term stopped mid-second-channel once the limit was reached. Now both channels run to 100 results regardless, and the agent waits for all channels to finish. Three channels at 100 results each is a meaningful latency increase for every search call. The agent-improvements plan (PR-10) assigns per-tool deadlines to search_packages, search_docs, and ensure_secret, but that work has not landed. Until it does, there is no bound on how long a search blocks the agent loop. The prior open question about whether this latency increase is acceptable until PR-10 lands remains unanswered.

Since the last review

• Addressed in code: Scoring bug fixed, but weights were chosen by inspection and not validated against eval cases. (The process_results_ranks_by_user_query test in the diff confirms the user query now flows correctly into scoring and drives the result order. The weights themselves are still inspection-chosen, but the structural bug is gone.)
• Still open: Channel pooling removes any channel-ordering priority guarantee for users with pinned or corporate channels. (No code change adds a channel-preference signal to scoring. No PR conversation comment acknowledges this behavior change or its impact on pinned-channel users.)
• Still open: Removing the early-exit increases latency because all channels run to 100 results before the agent sees anything, with no per-tool deadline to bound the wait. (CHANNEL_LIMIT remains 100 per channel, the early-exit is still removed, and no per-tool deadline has landed. No PR conversation comment addresses the latency increase.)
• Still open: Heuristic score weights were not validated against the specific eval cases showing search churn, and no eval suite run against this branch was planned or reported. (No eval results appear in the PR description or conversation. The weights in relevance_score match the prior version exactly.)
• Still open: Does any user segment rely on channel-ordering priority to ensure a pinned channel's package surfaces first? (No PR conversation comment addresses this question. No code adds channel-weight to the scoring function.)
• Still open: Is the per-tool deadline for search_packages queued immediately after this PR, or is the latency increase acceptable until that work lands? (No PR conversation comment addresses this. PR-10 in the agent-improvements plan covers per-tool deadlines but has not landed.)

Open questions

• Were the heuristic score weights validated against the eval cases that showed search churn, such as the spotify case in the PR screenshots? Has the eval suite been run against this branch?

• Does any user segment register a corporate or pinned channel and rely on that channel's result surfacing first? The pooling change removes that guarantee with no acknowledgment in the PR or its conversation.

• Is the per-tool deadline for search_packages (planned in PR-10) queued immediately after this PR ships? Until it lands, three-channel searches on common terms have no wall-clock bound.

Recommendation

Discuss first
All five decisions are defensible in isolation, but three prior open questions from the last review cycle remain completely unanswered after this revision. The channel-ordering behavior change could silently break a user segment that depends on pinned-channel priority, and no one has confirmed that segment does not exist. The latency increase from removing the early-exit has no per-tool deadline to bound it, and no timeline for that fix has been stated. At minimum the author should answer the channel-priority and latency questions in the PR before this ships.

[darkmatter]

🎨 Storybook preview

Open Storybook preview

Updated for b4b7d9e

---

⚠️ Detected UI changes (5)

These stories' HTML snapshots changed. I've added screenshots + links to the changed stories below. Review them carefully then accept the changes to regenerate baselines and include them in this PR:

Flows/Evolve › Playground

Flows/Evolve › 1. Begin (idle)

Flows/Evolve › 2. Evolving (progress)

Flows/Evolve › Evolving With Error Event

Flows/Evolve › 3. Review (changes generated)

---

Accept UI changes

• Click here to accept these changes

Alternatively, you can run bun run test:update-snapshots locally to re-generate the baselines and then push the changes to this PR.

What does this do?

The screenshots above show UI changes detected by the Storybook
snapshot tests run on this PR. Each image is the rendered output of
a Storybook story from the code in this PR branch; the snapshot
test compared it against the committed baseline in
__snapshots__/ and flagged the difference.

Checking the box tells the darkmatter[bot] to regenerate the
baselines from this PR's current code and commit them directly to
this branch. The new baselines become the source of truth for
future runs — only accept after confirming the visual changes are
intentional.

Comparison baseline: the committed __snapshots__/ files on this
PR branch (carried forward from develop). Accept updates them in
place on this branch.

[github-actions]

	Warnings
⚠️	No Linear issue ID found in this PR's title, description, or branch name (expected something like ENG-123). Add one so this work is traceable in Linear, or add #no-linear to the PR description to acknowledge it's intentionally untracked.

	Messages
📖	No docs update needed — acknowledged.

📋 PR Overview

Lines changed	360 (+279 / -81)
Files	0 added, 2 modified, 0 deleted
Draft / WIP	no
Has Test Plan	yes
Linear issue	no
No Test Plan Needed	no
New UI components	no
New Storybook stories	no
New Rust modules	no
New TS source files	no
New tests	no
package.json touched	no
Cargo.toml touched	no
Infra / CI touched	no

🔬 Coverage

Report	Lines	Statements	Functions	Branches
apps/native/coverage/coverage-summary.json	35.6%	35.2%	30.5%	29.5%

Generated by 🚫 dangerJS against b4b7d9e

[linear-code]

ENG-617