Sweep proxy, data-file reads, and one-step setup (0.2.0)

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "compactio",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "System 1 for your coding agent. Cuts tool output the task does not need, before it enters the context. Powered by Jev.",
   "author": { "name": "Rama Aditya", "url": "https://github.com/RamaAditya49" },
   "homepage": "https://github.com/RamaAditya49/compactio",

BLUEPRINT.md

@@ -112,10 +112,11 @@ It runs after the tool finishes and before the model sees the output.
 4. **Low confidence → `full`.** Keeping too much costs less than dropping too much.
 5. The original output stays on disk. The model can get it back with `compactio show <id>`.
 
-Code files from `Read` are never cut, because the agent may edit them. Small outputs (< 2 KB)
+Code files from `Read` are never cut, because the agent may edit them. Data files (logs, CSV,
+JSONL, lockfiles, minified bundles) take the filter path. Images, PDFs, and notebooks are skipped. Small outputs (< 2 KB)
 pass untouched. The saving comes from big outputs.
 
-### 4.2 Sweep (old context): v0.3
+### 4.2 Sweep (old context): v0.3, shipped as an opt-in proxy (`compactio proxy`)
 
 It runs every N turns, not every turn.
 

README.md

@@ -48,6 +48,8 @@ compactio stops the waste where it starts: at the tool output, before it enters
 - **Safe by default.** Low confidence, a timeout, or an API error returns the full output. compactio never blocks your agent.
 - **Nothing is lost.** Every cut output stays on your disk. The note in the output tells the agent how to get it back.
 - **Code is never cut.** A `Read` of a source file always passes in full. compactio only skips an exact re-read of an unchanged file.
+- **Data files are cut.** A `Read` of a log, a CSV, a JSONL dump, a lockfile, or a minified bundle takes the same path as `Bash` output.
+- **Old output can go too (opt-in).** The Sweep proxy removes old tool results that the current goal no longer needs. See [Sweep](#sweep-opt-in).
 - **Secrets stay local.** API keys, tokens, private keys, and `KEY=value` pairs are masked before any request.
 - **Two providers.** Use a TypeSafe key or an OpenRouter key.
 - **Works without a key.** Local mode uses lossless filters, the re-read skip, and a head-and-tail cut for very large output.
@@ -64,23 +66,21 @@ claude plugin install compactio@compactio
 
 Node 22.18 or later must be on your `PATH`.
 
-**2. Add one API key** in `~/.claude/settings.json`. Pick one provider:
+**2. Add one API key.** Get a key from [TypeSafe](https://console.typesafe.ai/keys) (the maker of Jev) or [OpenRouter](https://openrouter.ai/keys). Then run this in a terminal and paste the key. The key does not show on the screen.
 
-```jsonc
-{
-  "env": {
-    // Option A: TypeSafe, the maker of Jev (https://console.typesafe.ai/keys)
-    "TYPESAFE_API_KEY": "your-typesafe-key"
-
-    // Option B: OpenRouter (https://openrouter.ai/keys)
-    // "OPENROUTER_API_KEY": "sk-or-..."
-  }
-}
+```bash
+npx compactio key
 ```
 
-Without a key, compactio runs in local mode.
+Without a key, compactio runs in local mode. You can also put `TYPESAFE_API_KEY` or `OPENROUTER_API_KEY` in the `env` block of `~/.claude/settings.json` yourself.
+
+**3. Run setup in Claude Code**
 
-**3. Restart Claude Code.** compactio now works on every session.
+```text
+/compactio:setup
+```
+
+Setup checks the key and turns on the [Sweep](#sweep-opt-in). Then restart Claude Code. compactio now works on every session.
 
 **4. See the savings**
 
@@ -132,6 +132,46 @@ compactio uses the Claude Code `PostToolUse` hook. The hook runs after a tool fi
 | `headtail` | The first 40 and the last 30 lines | A long build or install log |
 | `stub` | One line | Output that is not related to the goal |
 
+### Sweep (opt-in)
+
+The hook can only cut **new** output. Old tool results stay in the history, and the agent sends them again on every turn. The Sweep removes them.
+
+The Sweep is a local proxy between Claude Code and the Anthropic API. On each request it does three steps:
+
+| Step | Who | What |
+|---|---|---|
+| 1 | Code | Put back every earlier tombstone, so that the prompt prefix does not change between turns. |
+| 2 | Jev | When old results hold 40,000 characters or more, rate up to 16 of them in one request: `keep` or `drop`. |
+| 3 | Code | Replace the `drop` results with a tombstone, but only when the saving pays for the prompt-cache rewrite. |
+
+A tombstone looks like this:
+
+```text
+[compactio: the output of Bash npm test was removed because the current goal no longer needs it. Full output: node ".../cli.js" show 1a2b3c4d. Or run the tool again.]
+```
+
+Rules:
+
+- The last 10 messages are never swept. They are the work in progress.
+- Results below 4,000 characters and results with images are never swept.
+- Jev must answer `drop` with a confidence of 0.8 or more. Otherwise the result stays.
+- **Cache gate.** A change in the middle of the history makes the next request write the cache again after that point. The Sweep drops only when `dropped × 0.1 × turns ≥ rest-of-history × 1.15`. `turns` is `COMPACTIO_SWEEP_TURNS` (default 30).
+
+`/compactio:setup` turns it on when a Jev key is present. You can also turn it on and off yourself: `/compactio:sweep on`, `/compactio:sweep off`, or `npx compactio sweep on` in a terminal. Then restart your Claude Code sessions.
+
+`sweep on` does four steps:
+
+1. Copy compactio to `~/.compactio/bin`, so that a plugin update does not break the proxy.
+2. Start the proxy. On Linux, it is the systemd user service `compactio-proxy`, with `Restart=always`: it starts again after a crash and after a reboot. On macOS and Windows, it is a background process.
+3. Wait until the proxy answers.
+4. Add two keys to the `env` block of `~/.claude/settings.json` (backup: `settings.json.compactio-bak`):
+   - `ANTHROPIC_BASE_URL=http://127.0.0.1:8787`
+   - `ANTHROPIC_DEFAULT_OPUS_MODEL=claude-opus-5-5[1m]`. Behind a custom base URL, Claude Code does not detect the 1M window, and autocompact runs in a loop. The `[1m]` suffix fixes it.
+
+**Session guard.** At the start of each Claude Code session, compactio checks the proxy. If the proxy is down, compactio starts it before the first request. If it still does not start, Claude Code shows a message with the command that turns the Sweep off.
+
+`compactio sweep status` shows the state. `compactio sweep off` removes the keys first, then stops the proxy.
+
 ### What leaves your machine
 
 Only a bounded and redacted summary goes to the provider:
@@ -153,15 +193,23 @@ Set these variables in the `env` block of `~/.claude/settings.json`.
 | `COMPACTIO_JEV_MODEL` | `jev-1.13.0` (TypeSafe), `~typesafe/jev-latest` (OpenRouter) | Jev model. |
 | `COMPACTIO_JEV_URL` | the provider endpoint | Custom endpoint, for example a proxy. |
 | `COMPACTIO_TIMEOUT_MS` | `1500` | Time limit for one decision. After it, the full output passes. |
-| `COMPACTIO_HOME` | `~/.compactio` | Folder for stored outputs, read hashes, and the decision log. |
+| `COMPACTIO_HOME` | `~/.compactio` | Folder for stored outputs, read hashes, Sweep decisions, and the decision log. |
+| `COMPACTIO_UPSTREAM` | `https://api.anthropic.com` | Sweep proxy: the API that receives the requests. |
+| `COMPACTIO_PORT` | `8787` | Sweep proxy: the local port, when `proxy` gets no port. |
+| `COMPACTIO_SWEEP_TIMEOUT_MS` | `3000` | Sweep proxy: time limit for one Jev request. After it, the history passes unchanged. |
+| `COMPACTIO_SWEEP_TURNS` | `30` | Sweep proxy: expected turns left in a session. A higher value drops more often. |
 
 ## Commands
 
 | Command | Description |
 |---|---|
+| `/compactio:setup` | Check the Jev key and turn on the Sweep. |
+| `npx compactio key` | Save a Jev key in the Claude Code settings. The key does not show on the screen. |
 | `/compactio:gain` | Show the savings scoreboard in Claude Code. |
 | `npx compactio gain` | Show the scoreboard in a terminal. |
 | `npx compactio show <id>` | Print a stored original output. The agent runs this itself when it needs the full output. |
+| `npx compactio sweep on\|off\|status` or `/compactio:sweep on\|off\|status` | Install, remove, or check the Sweep proxy service. |
+| `npx compactio proxy [port]` | Run the Sweep proxy in the foreground on `127.0.0.1`. |
 | `claude plugin disable compactio@compactio` | Turn compactio off. |
 
 ## FAQ
@@ -209,19 +257,55 @@ Claude Code today. Codex CLI, OpenCode, Gemini CLI, Cursor, and Trae are next. S
 - Claude Code already limits Bash output to 30,000 characters. On one Bash call, compactio saves at most about 7,500 tokens. The agent then carries that saving through every later turn.
 - We make no claim about the total bill until the public benchmark (task success, tokens, and cost) exists.
 
+## Limitations
+
+Read these before you use compactio. They are the limits of the design, not bugs.
+
+**Where compactio has no effect**
+
+- **Your messages, the agent's replies, the system prompt, and MCP tool schemas.** compactio only touches tool output.
+- **Source code from `Read`.** It always passes in full, because the agent may edit it. In a session that mostly reads code, the saving is small.
+- **Images, PDFs, and notebooks from `Read`.** They pass untouched and do not count in the scoreboard.
+- **Old context, without the Sweep.** The hook only cuts new output. The history that is already in the context stays until you run the Sweep, `/compact`, or `/clear`.
+- **Other hosts.** v0.1 works in Claude Code only.
+
+**Limits of the decision**
+
+- **Jev sees a preview, not the full output.** The preview is the head, the error lines, and the tail, about 6,000 characters (2,400 for each Sweep result). Jev can misjudge an output whose important part is in the middle.
+- **Jev request limits.** State and questions must fit in 64,000 tokens, and state plus the longest question in 32,000 tokens. For this reason, one Sweep request rates 16 results at most. The others wait for a later request.
+- **The goal is the last prompt.** A short prompt such as "continue" gives Jev little to work with.
+- **The data-file list is fixed.** compactio knows a data file by its name: `.log`, `.out`, `.csv`, `.tsv`, `.jsonl`, `.ndjson`, `.map`, `.min.js`, `.min.css`, and common lockfiles. If the agent must edit such a file, it gets a cut view. It must run `compactio show <id>` to see all of it.
+- **Latency.** A large output waits up to 1.5 s for Jev. A Sweep request waits up to 3 s. The time limit then lets the output pass unchanged.
+
+**Limits of the Sweep proxy**
+
+- **Claude Code cannot reach the API when the proxy is down.** The proxy fails open for its own errors, but not for a stopped process. The session guard starts it at the start of a session, and systemd starts it after a crash on Linux. A proxy that stops in the middle of a session on macOS or Windows stays down until the next session. Use `compactio sweep off`, not `systemctl stop`, to turn it off.
+- **macOS and Windows are not tested.** The background-process path is tested on Linux only.
+- **The key step needs a terminal.** A key typed into the Claude Code chat goes into the conversation, so `/compactio:setup` does not take a key.
+- **Each sweep costs one cache rewrite.** The cache gate estimates the cost with a fixed number of turns left (`COMPACTIO_SWEEP_TURNS`). If the session ends sooner, the sweep costs more than it saves.
+- **A tombstone is permanent.** A dropped result stays dropped for the whole session. The agent must run the tool again or run `compactio show <id>`.
+- **The proxy sees all API traffic,** including the auth header. It forwards the header and does not store it. It stores the dropped outputs on disk under `COMPACTIO_HOME`.
+- **The context window is set by name.** Behind a custom `ANTHROPIC_BASE_URL`, Claude Code does not detect the 1M window. `sweep on` maps the `opus` alias to `claude-opus-5-5[1m]`. If you pick a model by its full id, or pick Sonnet or Haiku, add `[1m]` yourself where the model supports it. When a new Opus ships, update `ANTHROPIC_DEFAULT_OPUS_MODEL`. `CLAUDE_CODE_MAX_CONTEXT_TOKENS` and `CLAUDE_CODE_DISABLE_UNKNOWN_MODEL_WINDOW_ENFORCEMENT` did not stop the loop in Claude Code 2.1.282.
+- **Test coverage.** Unit tests use a fake API and a fake Jev. One real Claude Code session (Opus 5.5, subscription login) ran through the proxy with 3 reads and 8 shell calls. Jev rated the old reads in 0.7 s. Longer real sessions are not tested yet.
+
+**Limits of the numbers**
+
+- Token counts are estimates: characters ÷ 4.
+- The state files (`store/`, `sessions/`, `sweep.json`) are never pruned.
+
 ## How compactio compares
 
 | | compactio | [rtk](https://github.com/rtk-ai/rtk) | [fast-jev-compaction](https://github.com/tamaratran/fast-jev-compaction) |
 |---|---|---|---|
 | When it acts | After each tool call | Before each shell command | At compaction |
 | How it decides | Jev, from the current goal and a content preview | Fixed rules per command | Jev, from a size note |
-| Tools covered | Bash, Grep, Web, MCP, re-reads | Shell commands | All, at compaction |
+| Tools covered | Bash, Grep, Web, MCP, data-file reads, re-reads, old results (Sweep) | Shell commands | All, at compaction |
 
 ## Roadmap
 
 - [x] **v0.1** Claude Code: tool output filter, re-read skip, scoreboard, redaction, local mode, TypeSafe and OpenRouter
 - [ ] **v0.2** Codex CLI, OpenCode, Gemini CLI, Cursor, Trae. Replay evaluation on real sessions.
-- [ ] **v0.3** Sweep: remove stale context in long sessions (opt-in local proxy), with prompt-cache protection
+- [x] **v0.3** Sweep: remove stale context in long sessions (opt-in local proxy), with prompt-cache protection. Data-file reads.
 - [ ] **v0.4** Gate: route each prompt to the cheapest model that can do the task
 - [ ] **v1.0** Public benchmark
 

commands/setup.md

@@ -0,0 +1,8 @@
+---
+description: Set up compactio in one step (filter, Jev key check, Sweep)
+allowed-tools: Bash(node:*)
+---
+
+!`node "${CLAUDE_PLUGIN_ROOT}/src/cli.ts" setup`
+
+Show the output above to the user as it is. Do not add commentary.

commands/sweep.md

@@ -0,0 +1,9 @@
+---
+description: Turn the Sweep proxy on or off, or show its status (on | off | status)
+argument-hint: on | off | status
+allowed-tools: Bash(node:*)
+---
+
+!`node "${CLAUDE_PLUGIN_ROOT}/src/cli.ts" sweep $ARGUMENTS`
+
+Show the output above to the user as it is. Do not add commentary.

hooks/hooks.json

@@ -3,16 +3,45 @@
     "PostToolUse": [
       {
         "matcher": "Bash|Read|Grep|WebFetch|WebSearch|mcp__.*",
-        "hooks": [{ "type": "command", "command": "node \"${CLAUDE_PLUGIN_ROOT}/src/hook.ts\" post-tool", "timeout": 5 }]
+        "hooks": [
+          {
+            "type": "command",
+            "command": "node \"${CLAUDE_PLUGIN_ROOT}/src/hook.ts\" post-tool",
+            "timeout": 5
+          }
+        ]
       }
     ],
     "UserPromptSubmit": [
-      { "hooks": [{ "type": "command", "command": "node \"${CLAUDE_PLUGIN_ROOT}/src/hook.ts\" prompt", "timeout": 5 }] }
+      {
+        "hooks": [
+          {
+            "type": "command",
+            "command": "node \"${CLAUDE_PLUGIN_ROOT}/src/hook.ts\" prompt",
+            "timeout": 5
+          }
+        ]
+      }
     ],
     "SessionStart": [
       {
         "matcher": "compact|clear",
-        "hooks": [{ "type": "command", "command": "node \"${CLAUDE_PLUGIN_ROOT}/src/hook.ts\" reset", "timeout": 5 }]
+        "hooks": [
+          {
+            "type": "command",
+            "command": "node \"${CLAUDE_PLUGIN_ROOT}/src/hook.ts\" reset",
+            "timeout": 5
+          }
+        ]
+      },
+      {
+        "hooks": [
+          {
+            "type": "command",
+            "command": "node \"${CLAUDE_PLUGIN_ROOT}/src/hook.ts\" start",
+            "timeout": 10
+          }
+        ]
       }
     ]
   }

package.json

@@ -1,6 +1,6 @@
 {
   "name": "compactio",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "System 1 for your coding agent. Cuts tool output the task does not need, before it enters the context. Powered by Jev.",
   "type": "module",
   "engines": {

src/cli.ts

@@ -1,8 +1,11 @@
 #!/usr/bin/env node
-// compactio CLI: `show <id>` prints a stored original output, `gain` prints the scoreboard.
+// compactio CLI: `show <id>` prints a stored original output, `gain` prints the scoreboard,
+// `proxy [port]` runs the Sweep proxy, `sweep on|off|status` installs it as a service.
 import { fileURLToPath } from "node:url";
 import { JEV_PRICE_PER_TOKEN } from "./jev.ts";
 import * as store from "./store.ts";
+import * as install from "./install.ts";
+import { serve } from "./sweep.ts";
 
 const tok = (chars: number) => Math.round(chars / 4); // estimate: ~4 characters per token
 const fmt = (n: number) => n.toLocaleString("en-US");
@@ -11,12 +14,14 @@ const k = (chars: number) => (chars >= 1000 ? `${(chars / 1000).toFixed(1)}k` :
 const bar = (part: number, width = 20) => "█".repeat(Math.round(part * width)).padEnd(width, "░");
 
 export function gain(entries: store.LogEntry[], sid?: string): string {
-  const rows = sid ? entries.filter((e) => e.sid === sid) : entries;
+  // Sweep "judge" rows are Jev calls, not outputs: count their cost, not their size.
+  const all = sid ? entries.filter((e) => e.sid === sid) : entries;
+  const rows = all.filter((e) => e.level !== "judge");
   const cut = rows.filter((e) => e.after < e.before);
   const before = rows.reduce((a, e) => a + e.before, 0);
   const saved = rows.reduce((a, e) => a + e.before - e.after, 0);
   const cutBefore = cut.reduce((a, e) => a + e.before, 0);
-  const jev = rows.filter((e) => e.engine === "jev");
+  const jev = all.filter((e) => e.engine === "jev");
   const cost = jev.reduce((a, e) => a + (e.jevTokens ?? 0), 0) * JEV_PRICE_PER_TOKEN;
   const share = before ? saved / before : 0;
   const avgCut = cutBefore ? saved / cutBefore : 0;
@@ -28,8 +33,9 @@ export function gain(entries: store.LogEntry[], sid?: string): string {
     `  Share of tool output cut     ${bar(share)} ${Math.round(share * 100)}%`,
     `  Outputs cut                  ${cut.length} of ${rows.length}  (avg cut ${Math.round(avgCut * 100)}%)`,
     `  Jev decisions                ${jev.length}  ·  $${cost.toFixed(4)}`,
+    `  Old outputs swept            ${rows.filter((e) => e.engine === "sweep").length}`,
     `  Unchanged re-reads skipped   ${rows.filter((e) => e.level === "unchanged").length}`,
-    `  Fail-open                    ${rows.filter((e) => e.engine === "fail-open").length}`,
+    `  Fail-open                    ${all.filter((e) => e.engine === "fail-open").length}`,
   ];
   const top = [...cut].sort((a, b) => b.before - b.after - (a.before - a.after)).slice(0, 3);
   if (top.length) {
@@ -52,7 +58,21 @@ if (process.argv[1] !== fileURLToPath(import.meta.url)) {
   process.stdout.write(text);
 } else if (cmd === "gain") {
   console.log(gain(store.readLog(), arg));
+} else if (cmd === "proxy") {
+  const port = Number(arg ?? process.env.COMPACTIO_PORT ?? 8787);
+  serve(port).on("listening", () => console.log(`compactio sweep proxy on http://127.0.0.1:${port}`));
+} else if (cmd === "sweep" && (arg === "on" || arg === "off" || arg === "status" || !arg)) {
+  const run = arg === "on" ? install.on() : arg === "off" ? install.off() : install.status();
+  Promise.resolve(run).then(console.log, (e) => {
+    console.error(`compactio: ${e.message ?? e}`);
+    process.exit(1);
+  });
+} else if (cmd === "setup" || cmd === "key") {
+  (cmd === "setup" ? install.setup() : install.key()).then(console.log, (e) => {
+    console.error(`compactio: ${e.message ?? e}`);
+    process.exit(1);
+  });
 } else if (cmd) {
-  process.stderr.write("usage: compactio show <id> | compactio gain [session-id]\n");
+  process.stderr.write("usage: compactio show <id> | compactio gain [session-id] | compactio proxy [port] | compactio sweep on|off|status | compactio setup | compactio key\n");
   process.exit(1);
 }