Doctor: no detection or reclamation for pre-#1451 shadow-$HOME trees — upgraded installs silently carry orphaned MEMORY/LEARNING data (112 ratings + 22 learning artefacts stranded on ours)

[vanvonlj]

TL;DR

#1451 fixed prevention. Nothing fixes the installed base.

The consumer-side normalization from #1451 works — verified on a fresh 7.1.1 deploy, our shadow trees stopped growing the moment 7.1.1 landed. But every install that ran ≤6.0.5 still has the shadow trees it already accumulated, and the MEMORY/LEARNING data inside them is real, orphaned, and invisible to MemoryRetriever, the statusline ratings display, and the learning loop.

Nothing detects this. Doctor.ts — the tool whose entire job is "what's broken, here's the fix command" — has zero checks for it. So the user's memory system silently under-reports forever, and they have no way to know.

What we found on one upgraded install

Upgraded 6.0.5 → 7.1.1. Then swept for directories literally named $HOME:

find "$HOME" -maxdepth 6 -type d -name '$HOME'

22 shadow trees — one per project directory a session had ever run in, plus ~, plus ~/.claude/LIFEOS/PULSE. Contents, versus what was in the real MEMORY:

stranded in $HOME/…	actually in MEMORY
112 satisfaction ratings (MEMORY/LEARNING/SIGNALS/ratings.jsonl)	2
117 MEMORY/OBSERVABILITY/success-claim-gate.jsonl records	2
117 MEMORY/OBSERVABILITY/format-gate.jsonl records	112
15 MEMORY/STATE/instruction-integrity.jsonl records	1
22 learning artefacts — incl. 3 complete LEARNING/FAILURES captures (CONTEXT.md + sentiment.json + tool-calls.json + full transcript.jsonl) and 9 LEARNING/ALGORITHM sentiment notes	—

This is the exact failure mode #1451's description predicted ("FailureCapture wrote a full failure record … invisible to MemoryRetriever, the statusline ratings display, and the learning loop") — it just also persists after the fix, for anyone who already had it.

The memory system wasn't thin. It was writing to the wrong address, and every freshness: no data reading was simultaneously true and deeply misleading.

Why it stays invisible

• Doctor.ts --reconcile reconciles hooks (declared vs registered). It does not look at env values or the filesystem.
• A plain Doctor.ts run reports capability status (codex / browser / voice / cloudflare). Nothing about paths.
• The shadow trees sit in project directories, not under ~/.claude, so nothing in a LifeOS sweep would ever traverse them.
• The data is valid, parseable, and complete — it's simply at the wrong path. No error is ever raised, nothing is corrupt, nothing crashes. It just quietly isn't there.

Repro

On any install that ran ≤6.0.5 for a while, then upgraded:

# 1. the shadow trees (should print nothing on a healthy box)
find "$HOME" -maxdepth 6 -type d -name '$HOME'

# 2. the poisoned source, if the install never re-ran InstallSettings
jq '.env | with_entries(select(.value | tostring | test("\\$HOME")))' \
   ~/.claude/settings.system.json
# { "LIFEOS_DIR": "$HOME/.claude/LIFEOS",
#   "PROJECTS_DIR": "$HOME/Projects",
#   "LIFEOS_CONFIG_DIR": "$HOME/.claude/LIFEOS" }

# 3. count what's stranded
find "$HOME" -maxdepth 6 -type d -name '$HOME' \
  -exec sh -c 'cat "$1/.claude/LIFEOS/MEMORY/LEARNING/SIGNALS/ratings.jsonl" 2>/dev/null' _ {} \; \
  | grep -c .

Note that #1451 is consumer-side by design (settings.system.json deliberately keeps the portable $HOME template), so an upgraded install legitimately still has the literal in settings.system.json and is nonetheless safe. Detection therefore cannot key on the settings value alone — it has to look for the directories.

Proposed fix: a Doctor capability + --reclaim

This is squarely Doctor's shape: probe, report, hand back a copy-paste fix.

1. Detect — add a capability check:

// Doctor.ts — shadow-$HOME trees (#1404 / #1451)
// #1451 stops NEW ones. Installs upgraded from <=6.0.5 keep the ones they already
// grew, and the MEMORY/LEARNING data inside is orphaned — valid, complete, and
// unreachable. Nothing else in the system will ever notice.
const shadows = findShadowHomeDirs(process.env.HOME!);      // find -type d -name '$HOME'
const orphaned = countOrphanedRecords(shadows);             // ratings + gates + learning artefacts

if (shadows.length) {
  report({
    name: "Orphaned memory (shadow-$HOME trees)",
    state: "broken",
    powers: "the learning loop, satisfaction ratings, failure capture",
    detail: `${shadows.length} shadow tree(s) holding ${orphaned} orphaned record(s) — ` +
            `written by a pre-#1451 install and invisible to MemoryRetriever`,
    fix: "bun LIFEOS/TOOLS/Doctor.ts --reclaim",
  });
}

2. Reclaim — --reclaim merges and removes. The merge must be a union keyed on the exact line, not a copy: the real files already hold records, and .jsonl here is append-only, so concatenate → dedupe exact lines → sort by timestamp. Non-JSONL artefacts (LEARNING/FAILURES/**, LEARNING/ALGORITHM/*.md) copy only where the destination does not already exist. Caches (STATE/*-cache.*, instruction-hashes.json, USER/CACHE/freshness.json, last-response.txt) are disposable — drop them. Back up before touching anything.

We implemented exactly this and it recovered all 112 ratings + 117 + 15 + 22 artefacts cleanly; happy to open a PR with it if useful.

One trap worth calling out for whoever implements this: it is tempting to "fix" the settings file by expanding every $HOME in it. Do not. Hook command and statusLine strings are shell-evaluated, so $HOME/.claude/hooks/X.hook.ts is correct there and must survive; only env values are injected verbatim. A blanket expansion breaks every hook path — a close cousin of the corruption already reported in #1484. Our regression test asserts both halves: env free of the literal and hook commands still carrying $HOME.

Environment

• LifeOS 7.1.1 (03a969d), upgraded from 6.0.5
• Linux (Ubuntu), Bun 1.3.14, Claude Code harness
• Related: settings.system.json env values ship a literal, unexpanded $HOME — breaks USER scaffold on fresh installs (regression of the #1124 / #1270 class) #1404 (root cause), fix: expand literal $HOME in injected env path vars (LIFEOS_DIR/LIFEOS_CONFIG_DIR/PROJECTS_DIR) — fixes #1404 shadow-$HOME directories and orphaned learning data #1451 (prevention, merged — works, confirmed here), Statusline/tools write to a literal $HOME/ dir in CWD when spawned with unset/unexpanded $HOME #1463 (statusline variant), Installer writes literal "${HOME}" into settings.json, creating a bogus "${HOME}/" directory #1124 / fix: resolve path env vars (PAI_DIR/PROJECTS_DIR/CLAUDE_CONFIG_DIR) through paths.ts — settings.json env values are not shell-expanded #1270 (earlier instances of the same class). This issue is specifically about the remediation gap for existing installs, which none of the above address.

[danielmiessler]

Shipped your proposal in source: Doctor.ts now detects shadow-$HOME trees and --reclaim merges JSONL by exact-line union and copies artefacts non-destructively — and the settings-file trap you called out is honored (env values only; hook commands keep their $HOME). The credit comment in the file cites you. Ships in the next release.