fix(sections): one item header is not an item structure, and TwentyF stops reading the parser we are deleting (dt1f, 07lk.3)

[dgunning]

Two commits, split because their blast radii differ: the first changes section extraction for every form, the second changes 20-F only. The second's case rests on the first having landed, so they ship together but stay separable — 933edc1d was verified to stand alone in a worktree.

1. 933edc1d — one item header is not an item structure

The section extractor finds headers in layers: semantic headings first, then bold paragraphs, table cells and plain paragraphs as fallbacks. Every fallback was gated on whether any header collected so far mentioned an item.

On 0001144204-10-017467, a 2010 20-F in EDGARizer HTML, header detection promoted 86 nodes of which three named an item — and one of those three was a sentence reading "Please refer to Item 6.E, Directors, Senior Management and Employees". Those three suppressed the strategies that find the filing's fifteen real item headers. doc.sections returned four items where the legacy ChunkedDocument found twenty-six.

Two defects in one predicate: the item test was unanchored (re.search, so prose counted), and one item was treated as proof of a structure. The gate now asks for coverage — have the headers named at least half the items the form defines? The denominator comes from each form's own FormSchema.

8-K and title-based forms keep the presence test. An 8-K reports only the items it has, so a two-item 8-K is complete; a ratio against the 33 the form allows would put every 8-K below any floor. Same reason the parity benchmark gives 8-K no coverage rate.

The 0.5 floor is measured, not chosen. It sits in the middle of a gap that exists in every form's distribution over the parity corpus:

20-F   10%  |  61% 84% 90% 90%
10-K   17% 22% 39% 48%  |  65% 78% 83% 87% 96% 100%
10-Q   43%  |  71% 86% 100%

Any cut in 49–60% separates these identically. Strategies can now run in the same pass rather than only when every earlier one came up empty, so 3, 4 and 5 dedup by position as 3b already did.

Measurement — all 115 corpus fixtures, before vs after, diffed per filing

form	legacy_only before → after
20-F	26 → 12
10-K	59 → 59
10-Q	3 → 3
8-K	1 → 1

Zero new gaps; zero items the new parser stopped finding. Exactly one filing changed, gaining 14 items. 20-F coverage delta moved −8.4% → −3.7%.

Parse timings are unchanged on the filings where the gate newly fires (pfe, wmt, xom 10-Ks; gbdc, nflx 10-Qs) — those reach sections through the TOC path, so the extra tree walks never happen.

2. d60f2885 — TwentyF reads the parser we are keeping

TwentyF was the last class where legacy ChunkedDocument was primary; TenK, TenQ and CurrentReport already used it only as an empty-result fallback. Deleting edgar.files in 6.0 means deleting ChunkedDocument, so every remaining primary use has to go first.

Its in-code justification — "the pattern-based extractor doesn't handle the Table of Contents format well" — did not survive measurement. On the one 20-F where legacy clearly won, TOC detection returned no sections at all, so the TOC path was not what was winning. The cause was the gate above.

Item numbers come from Section.item, not the section key. The key is not stable across detection strategies — the TOC engine emits part_i_item_1, its anchor path emits Item 1, the pattern extractor emits item_1, and all three are live on the corpus. Keying off it would have returned section keys on most filings and raw titles (Part I, Signatures) on sections that are not items at all. Section.item is '1' in every case and None on non-items.

.items also now deduplicates and sorts canonically on both paths (Item 4 < Item 4A < Item 5, Item 16A < Item 19); the legacy path previously returned document order, which put Item 17 first on several filings. This also settles a disagreement — __getitem__ has read the new parser for some time while .items read legacy, so the two could describe the same filing differently.

What users see, across the 15-filing 20-F corpus

	filings
Identical	10
Better (gains Item 3C)	1
Legacy fallback fires, unchanged	4
Fewer items	2

The two that lose are 0001062993-16-008650 (20 → 17) and 0001144204-10-017467 (26 → 18). Those 11 items are exactly the 20-F entries already in BASELINE_GAPS — no new ground lost. Without the gate fix first, the second would have been 26 → 4, which is why these are ordered this way.

Verification

• Fast lane: 4986 passed (was 4976 — the 18 new tests)
• tests/issues/regression/ + company reports + filing: 2486 passed, 4 xfailed
• Parity ratchet: 6 passed, including test_closed_gaps_are_recorded — the win is banked, 22 → 8
• 933edc1d alone in a detached worktree: 66 passed
• Ruff clean on everything touched (the 13 pre-existing S110 in to_context are untouched)

Notes for review

The dt1f regression fixture is synthetic on purpose. The real filing lives in tests/fixtures/text_boundary_corpus, which is gitignored (91 MB), so a test reading it would silently skip in CI — the failure mode test_section_parity_ratchet.py documents at length. The document in test_dt1f_item_coverage_gate.py reproduces the shape and fails the same way (2 sections, not 11) against the old gate. It carries its own guard test, so it cannot quietly stop reproducing the defect.

item_sort_key moved to _structures.py where the other cross-form helpers live. ten_k._item_sort_key stays bound to it because tests/test_company_reports.py imports that name; 8-K's dotted-item ordering in current_report is a genuinely different key and is untouched.

Still open on 07lk.3. 10-K is now the whole critical path at 59 gaps, against 20-F 12, 10-Q 3, 8-K 1. Deleting the TenK/TenQ/CurrentReport fallbacks needs each form at zero. wfc/10k is the sharpest of those — heading coverage there is 0, so the gates already fire and it is a different failure from this one.

🤖 Generated with Claude Code