DOCS-3014: Fix three defects in the llms.txt Markdown conversion - #2956
Merged
Conversation
The converter turns rendered HTML into Markdown for llms-full.txt. Three defects made that output worse than it needed to be, and all three would be far more visible once we publish per-page Markdown. Heading anchors leaked into headings. Docusaurus appends a hash-link anchor containing a zero-width space to every heading, which the converter kept: "## Before you begin[<zero-width space>](#before-you-begin)". There were 2,559 of these in calico/llms-full.txt alone. Strip a.hash-link during extraction. Code fences were double-spaced and lost their indentation. Prism renders each source line as a block-level token-line div that also ends in a <br>, so the line break is represented twice and hast-util-to-text honoured both. Demote those divs to spans so the <br> is the only break, and pass whitespace: 'pre' so leading indentation and column alignment survive. This turns unusable YAML samples back into valid YAML and restores the columns in kubectl output. Heading hierarchy was inverted. The page <h1> lives in a <header> we strip, so bodies start at h2, but they were nested under an h3 doc title. Shift body headings to sit below their doc title. Fixing the hierarchy meant reducing the nesting depth. Under the previous section/title structure the shift needed to be two levels, which pushed 69% of headings onto h6 and flattened them together. Each doc now gets a single h2 in llms-full.txt, with its section carried as a metadata line instead of a heading, so only one shift level is needed. That also gives every doc a Source line, so a reader can tell which page a passage came from. Verified against a full build: 2,559 anchor artifacts to 0, exactly one h1 per file, and h6 headings down from 825 to 428 in calico/llms-full.txt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
✅ Deploy Preview for calico-docs-preview-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview succeeded!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves the docusaurus-plugin-llms-txt HTML→Markdown conversion pipeline so the generated llms-full.txt output is cleaner and structurally correct ahead of follow-on PRs that will emit per-page .md files.
Changes:
- Strip Docusaurus heading “hash-link” anchors during HTML extraction.
- Fix Prism-rendered code block conversion to preserve indentation and avoid double-spaced lines (
hast-util-to-textwithwhitespace: 'pre'+ preprocessing). - Restructure
llms-full.txtgeneration to add per-docSource:metadata and adjust body heading hierarchy viashiftHeadings(), plus add Jest unit tests and update Jest transforms for ESM deps.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/plugins/docusaurus-plugin-llms-txt/index.js | Passes siteUrl into full-output generation to support per-doc Source: lines. |
| src/plugins/docusaurus-plugin-llms-txt/generate.js | Adds shiftHeadings() and changes llms-full.txt layout to doc-title + metadata + shifted body headings. |
| src/plugins/docusaurus-plugin-llms-txt/extract.js | Removes a.hash-link from extracted HTML and preprocesses Prism token lines before conversion. |
| src/plugins/docusaurus-plugin-llms-txt/convert.js | Preserves code whitespace by using toText(..., { whitespace: 'pre' }) for code blocks. |
| src/plugins/docusaurus-plugin-llms-txt/test/conversion.test.js | Adds unit tests covering anchor stripping, code fence formatting/indentation, and heading shifting. |
| jest.config.mjs | Configures Jest to transform ESM-only dependencies by not ignoring node_modules. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+37
to
+42
| const marker = fenceMatch[1]; | ||
| if (openFence === null) { | ||
| openFence = marker[0]; | ||
| } else if (marker[0] === openFence) { | ||
| openFence = null; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of seven PRs on DOCS-3014, adding .md twins for every docs URL. This one fixes defects in the existing HTML-to-Markdown converter so that later PRs build on correct output. It changes no doc content.
The converter reads rendered HTML and produces the Markdown in llms-full.txt. Three defects made that output worse than it needed to be, and each would be far more visible on a per-page .md file than it is buried in a 5 MB blob.
Heading anchors leaked into the text. Docusaurus appends a hash-link anchor containing a zero-width space to every heading, and the converter kept it, producing headings like "## Before you begin". There were 2,559 of these in calico/llms-full.txt alone. Fixed by stripping a.hash-link during extraction.
Code fences were double-spaced and lost their indentation. Prism renders each source line as a block-level token-line div that also ends in a br, so the line break is represented twice and hast-util-to-text honoured both. Fixed by demoting those divs to spans, so the br is the only break, and by passing whitespace: 'pre' so leading indentation survives. Two visible consequences:
Heading hierarchy was inverted. The page h1 lives in a header element that we strip, so page bodies start at h2, but they were then nested under an h3 doc title. Body headings are now shifted to sit below their doc title.
Fixing the hierarchy meant reducing the nesting depth, which is the one change here that goes beyond a straight bug fix. Under the old section-then-title structure the shift had to be two levels, and 69% of headings in these docs are already at h4 or deeper, so a two-level shift pushed nearly all of them onto h6 and flattened them into a single indistinguishable level. Each doc now gets one h2 in llms-full.txt and carries its section as a metadata line rather than a heading, so a one-level shift is enough and nothing collapses. A side benefit is that every doc now has a Source line, so a reader can tell which page a passage came from. Previously there was no way to know.
Verified against a full local build of all three products, 1,036 pages:
The remaining heading-level jumps in the output come from source pages that skip a level themselves, mostly DocCardLink rendering an h5 under an h3. That is component markup, not conversion, and is out of scope here.
There is nothing to see on the deploy preview. Generation is still gated behind GENERATE_LLMS, which Netlify never sets, so the preview serves the unchanged copies committed under static/. That gate is removed in the next PR in the stack. To see the effect locally:
Tests: adds 15 unit tests covering all three fixes, at src/plugins/docusaurus-plugin-llms-txt/test/conversion.test.js. These need Jest to transform the plugin's ESM-only dependencies (cheerio, unified, rehype, remark), so jest.config.mjs now sets transformIgnorePatterns to an empty array. That costs about two seconds on the full suite and avoids maintaining a brittle allowlist of the whole unified ecosystem. All 9 suites pass.