Derive chunk spans from byte offsets behind a chunker contract

[Ranhiru]

First feature PR of Phase 2. Lands the seam every later chunking change plugs into, and fixes two live citation bugs on the way.

Bugs fixed

_line_window_for_piece recovered a chunk's position by searching for the chunk's own text:

1. full_text.find(piece, offset_hint) locks onto the wrong occurrence in files with repeated blocks — JSON, XML, .csproj, generated code.
2. _chunk_plain did "\n".join(text.splitlines()).strip() and then passed base_line=1, so every line number in a file starting with blank lines was shifted.

Reproduced on a 200-line JSON file with three leading blank lines. Old code reports the first chunk as lines 1-38; the content actually lives at 4-41, and asserting content in "\n".join(lines[0:38]) fails. New code reports 4-41 and the assertion holds.

Both disappear once lines derive from byte offsets.

The contract

A chunker's only job is to choose byte cut points. It never produces content, line numbers, IDs, or token counts.

chunkers/base.py   contract + shared span math
chunkers/text.py   fixed-size overlapping chunker (~35 lines)

ChunkDraft is three fields: start_byte, end_byte, section. A single shared finalize snaps both offsets to UTF-8 codepoint boundaries, trims whitespace, drops whitespace-only spans, and computes line_end = line_of(end - 1) so a span ending at column 0 reports the previous line — the off-by-one tree-sitter's end-exclusive end_point would otherwise introduce later.

chunkers/* import nothing from config.py, so they are unit-testable without a workspace.yaml (config.py:20 calls load_config() at import).

CRLF is normalised once, at read. Lone \r and \x0b deliberately are not: splitlines() treating them as breaks is exactly the divergence being removed.

Behaviour held constant

• Text cut positions are unchanged (step = max(1, target - overlap), same loop), so .json / .yaml / .xml boundaries do not move.
• _build_chunk is untouched — same stable string, same sha1[:16], same tokens_est.
• Chunk stays at nine fields, so chunk_types.py, keyword_index.py, vector_index.py, storage.py and app.py are zero-diff. Cost: no filtering by strategy.
• One deliberate change: ordinal is now file-global rather than per-section, closing a chunk-ID collision when two sections share a heading and line window.

Config surface

chunking: in workspace.yaml, entirely optional — existing workspaces need no edit. Profiles merge per-name (setting only target_size on code keeps its defaulted strategy); rules: replaces wholesale; hard_max_size defaults to 2 * target_size. Rules flatten to by_extension at load, so routing is one dict lookup.

markdown and code strategies route to text until their chunkers land. The surface ships first; the strategies fill in behind it.

_coerce_chunking rejects: non-mapping sections, unknown strategy (catches tree_sitter pasted from the design doc), non-positive sizes, hard_max_size < target_size, overlap >= target_size (today masked into a chunk explosion by max(1, target - overlap)), unknown profile references, duplicate extensions, a code rule without language, language on a non-code rule, and unknown keys in a profile body.

Not validated at load: whether language is supported by the parser pack. That would drag tree_sitter_language_pack into a module app.py, cli.py and setup.sh all import — validate it lazily when the code chunker lands.

INDEX_VERSION 1.1.0 -> 1.2.0: corrected line numbers rotate chunk IDs, so existing indexes must rebuild. A real chunking fingerprint replaces this blunt instrument next.

Tests

79 passing. Two new modules, both runnable without a workspace.yaml:

• test_chunk_spans.py — test_line_numbers_exact (for every chunk, the source lines it claims contain its content) plus leading blank lines, repeated blocks, UTF-8 multibyte with no U+FFFD, CRLF vs LF, no trailing newline, span ending at column 0, line_of boundaries, snap_forward never splitting a codepoint.
• test_chunking_config.py — defaults, partial profile override, wholesale rule replacement, and each validation error.

Verified

• make test green.
• Chunked this repo's own CLAUDE.md, index.py, cli.py, workspace.example.yaml and a synthetic repeated-block JSON file through chunk_file: 35 chunks, every reported path:line_start-line_end confirmed to contain its content, no duplicate chunk IDs.
• _schema_invalidation_reason({"index_version": "1.1.0"}) returns index_version 1.1.0 -> 1.2.0.

Not verified: a full make build end to end — this checkout has no cloned repos and no embeddings endpoint running.