refactor: separate document processing from import workflow

documentation/CONTRIBUTION_WORKFLOW.md

@@ -53,7 +53,7 @@ flowchart TD
 | --- | --- |
 | Create the issue | [Bug report](../.github/ISSUE_TEMPLATE/bug_report.md) and [feature request](../.github/ISSUE_TEMPLATE/feature_request.md) templates (GitHub offers them when you click **New issue**) |
 | Branch off `main` | [`DEPLOYMENT.md` § 1. Create a feature branch](../DEPLOYMENT.md#1-create-a-feature-branch) and [§ Branch Protection Rules](../DEPLOYMENT.md#branch-protection-rules) |
-| Changing import behavior | [`IMPORT_RULES.md`](IMPORT_RULES.md) and [`DEPLOYMENT.md` § Updating Import Rules](../DEPLOYMENT.md#updating-import-rules) |
+| Changing import behavior | [`IMPORT_RULES.md`](IMPORT_RULES.md), [`DOCUMENT_PROCESSING.md`](DOCUMENT_PROCESSING.md) for the shared processing boundary, and [`DEPLOYMENT.md` § Updating Import Rules](../DEPLOYMENT.md#updating-import-rules) |
 | Changing user-facing import errors | [`IMPORT_ERROR_CODES.md`](IMPORT_ERROR_CODES.md), and the "Copy rules" at the top of [`nofos/bloom_nofos/import_errors.py`](../nofos/bloom_nofos/import_errors.py) |
 | Adding or changing UI | [`UI_PATTERNS.md`](UI_PATTERNS.md): check this growing catalog for a relevant pattern and reuse its shared component. It is not a complete inventory of the site's UI; extend it when introducing another verified pattern. |
 | Updating docs | [`documentation/README.md`](README.md): docs go in `documentation/`, never a top-level `docs/`; ADRs go in [`adr/`](adr/README.md); PR screenshots go in `review-evidence/` |

documentation/DOCUMENT_PROCESSING.md

@@ -0,0 +1,65 @@
+# Shared document processing
+
+`nofos/document_processing.py::process_document_content` accepts HTML text and
+returns existing mutable soup, section/subsection dictionaries and extracted
+instruction tables. An internal caller can use it without a request, upload,
+user or document database record. It does not persist documents or return web
+responses. These parser-specific results are not a public interchange schema.
+
+## Responsibilities
+
+The shared sequence preserves character/link replacement, HTML parsing, the
+first Before you begin removal, heading-level resolution, ordered NOFO
+normalization, then section/subsection parsing. Builder opts into the existing
+introductory-page capture, scoped tagline extraction and Metadata consumption
+rules through explicit `builder_import` and `extract_scoped_tagline` arguments.
+The result carries the captured tagline. Section parsing uses the same copied
+source as the application previously did, retaining intact metadata and callout
+content for field inference and confirmation. Composer/Compare retain their
+existing defaults. No transformation rule or order changes.
+
+Builder, Composer and Compare use it through `BaseNofoImportView`. The view
+retains upload conversion, filename handling, warning counts, permissions,
+error presentation and logging. Word conversion still uses Mammoth, its style
+map, warning filtering and strict mode. HTML callers need no uploaded file.
+
+The optional section parser preserves the existing view extension point and
+receives only soup and heading level. Composer instruction attachment stays
+in the parsing error scope. Step 3 and key-date postprocessing stay outside it.
+Metadata inference, persistence, transactions, auditing, readability snapshots
+and reimport confirmation remain in their existing owners.
+
+Exceptions propagate unchanged, including `ValidationError` with `no_sections`.
+The application maps failures to existing pages and error codes. Conversion
+warnings remain separate from content results; HTML uploads still report zero
+Mammoth warnings. Results retain mutable identity for consumer hooks.
+
+## Portability limits
+
+This is an internal separation, not a package, service or framework migration.
+BeautifulSoup tags and dictionaries remain parser-specific. Normalization
+remains NOFO-specific, including Word-derived table/callout/heading conventions.
+It does not promise compatibility with every editor's HTML or form data.
+
+The imported `nofo.py` still loads Django models, settings, Constance and
+application utilities. Django initialization is required even though processing
+performs no document persistence. Existing validation exceptions and heading
+limits remain application dependencies. Later extraction would need to isolate
+these dependencies and translate errors at the application boundary.
+
+With DEBUG enabled, `process_nofo_html` still writes a debug HTML snapshot under
+the fixtures directory. Disable DEBUG for sensitive content or processing that
+must not write that snapshot. Tests run with DEBUG disabled. No new side effects
+or configuration switches are introduced.
+
+Dependencies include BeautifulSoup and existing HTML/Markdown helpers and
+configuration. The separate Word adapter also needs Mammoth, its style map and
+document transform. Output generation remains separate and is not duplicated.
+
+## Verification
+
+Direct synthetic tests need no request, upload or database records. They compare
+results against the pre-extraction sequence and verify metadata, instructions,
+extension-point identity and failure codes. Characterization tests retain
+transformation order and hook error ownership. Real synthetic imports exercise
+all three consumers. Full CI and a local UI smoke check complete delivery.

documentation/IMPORT_RULES.md

@@ -7,6 +7,7 @@ This document catalogs every automatic content rule the NOFO Builder applies whe
 **Not to be confused with import error *codes*.** The `IMPORT-NNN` IDs here are numbered rule identifiers for content transformations. The `IMPORT-NAME` strings a user sees when an import is blocked (`IMPORT-NO-SECTIONS`, `IMPORT-OPDIV-BLANK`, and so on) are a separate namespace, catalogued in [`IMPORT_ERROR_CODES.md`](IMPORT_ERROR_CODES.md). A rule here can be the *reason* an error code fires; they are not the same registry.
 
 **Source files covered by this document:**
+- `nofos/nofos/document_processing.py` — shared content orchestration; rule implementations remain in their existing modules, with unchanged order and behavior. See [DOCUMENT_PROCESSING.md](DOCUMENT_PROCESSING.md).
 - `nofos/nofos/nofo.py` — `process_nofo_html()` and the ~20 cleanup passes it runs, plus sectioning/subsectioning and metadata-suggestion logic
 - `nofos/nofos/utils.py` — `style_map_manager`, the Mammoth DOCX→HTML style-name map
 - `nofos/nofos/import_transforms.py` — pre-Mammoth DOCX document transform (`transform_word_document`)

documentation/README.md

@@ -19,6 +19,7 @@ decision records belong in `adr/`; PR-specific screenshots and validation notes
 - [WORD_METADATA_IMPORT.md](WORD_METADATA_IMPORT.md) — Legacy labels and trailing Metadata blocks, precedence, defaults and saved-record behavior
 - [INTRODUCTORY_PAGE_IMPORT.md](INTRODUCTORY_PAGE_IMPORT.md) — Supported Word introductory headings, destinations, maintained registration families and legacy preservation
 - [endnote-import.md](endnote-import.md) — How Word imports preserve native notes and link manually authored bracketed references
+- [DOCUMENT_PROCESSING.md](DOCUMENT_PROCESSING.md) — Shared content-processing boundary, application responsibilities and remaining portability limits
 - [NOFO_BUILDER_FAQ.md](NOFO_BUILDER_FAQ.md) — Answers to common questions about designing a NOFO in NOFO Builder: importing, Markdown, tables, page layout, endnotes, appendices, metadata, and status; links to the editor tasks for step-by-step instructions
 - [READABILITY_METRICS.md](READABILITY_METRICS.md) — The `hhs-nofo-metrics` integration and source contract; when readability snapshots are saved (by a person, or automatically on import, re-import and Download PDF); saved readability history, deletion, archiving, and duplication behavior
 - [UI_PATTERNS.md](UI_PATTERNS.md) — Reusable interface patterns, including shared file-upload error states, loading progress modals, readability reliability alerts, and subsection save-conflict recovery

documentation/review-evidence/document-processing-1058/README.md

@@ -0,0 +1,15 @@
+# Document processing smoke check
+
+Local Chrome check of the #1058 refactor, using an isolated SQLite database and
+synthetic HTML. No production document or credentials were used. DEBUG was off.
+
+The existing import form accepted the synthetic HTML, advanced to naming, saved
+the NOFO and displayed its edit page. Verified both step headings, Summary,
+applicant prose, the Funding amounts caption, the table's zero value, and the
+ordered-list content. Clicking Read eligibility note reached `#note-1`.
+
+![Imported synthetic content](import-smoke.jpg)
+
+This is a content-processing smoke check, not a new Word conversion test or PDF
+export check. Automated import/reimport, Composer and Compare tests cover the
+other consumers and the unchanged Word conversion path.

nofos/nofos/document_processing.py

@@ -0,0 +1,91 @@
+"""Internal content processing, separate from upload and persistence workflows.
+
+This reuses Builder's NOFO-specific rules and parser objects, not a portable
+interchange schema. See documentation/DOCUMENT_PROCESSING.md for dependencies.
+"""
+
+from copy import deepcopy
+from dataclasses import dataclass
+
+from bs4 import BeautifulSoup
+from django.core.exceptions import ValidationError
+
+from .nofo import (
+    builder_import_content,
+    consume_tagline_paragraph,
+    decompose_before_you_begin_section,
+    extract_basic_information_tagline,
+    get_sections_from_soup,
+    get_subsections_from_sections,
+    process_nofo_html,
+    replace_chars,
+    replace_links,
+    resolve_section_heading_level,
+    unwrap_heading_containers,
+)
+
+
+@dataclass
+class ProcessedDocumentContent:
+    """Existing parser results, with source retained for application inference."""
+
+    soup: BeautifulSoup
+    sections: list
+    instructions_tables: list
+    imported_tagline: str | None = None
+
+
+def get_sections_and_subsections_from_soup(
+    soup, top_heading_level, *, consume_scoped_tagline=False, consume_metadata=False
+):
+    """Extract content dictionaries, retaining the existing validation code."""
+    content = builder_import_content(soup) if consume_metadata else soup
+    sections = get_sections_from_soup(content, top_heading_level)
+    if not len(sections):
+        raise ValidationError("That file does not contain a NOFO.", code="no_sections")
+    return get_subsections_from_sections(
+        sections, top_heading_level, consume_scoped_tagline=consume_scoped_tagline
+    )
+
+
+def process_document_content(
+    html, *, section_parser=None, builder_import=False, extract_scoped_tagline=False
+):
+    """Normalize HTML and parse sections without a request, upload or ORM record.
+
+    ``section_parser`` preserves the application's existing overridable section
+    parser. Instruction attachment and consumer postprocessing stay with callers.
+    Exceptions propagate unchanged for the caller to present or handle.
+    """
+    cleaned_content = replace_links(replace_chars(html))
+    soup = BeautifulSoup(cleaned_content, "html.parser")
+    if builder_import:
+        from .introductory_page import capture_introductory_page
+
+        soup = capture_introductory_page(soup)
+    # Preserve the pre-pass before selecting the main section heading level.
+    decompose_before_you_begin_section(soup)
+    top_heading_level = resolve_section_heading_level(
+        builder_import_content(soup) if builder_import else soup
+    )
+    imported_tagline = None
+    if extract_scoped_tagline:
+        unwrap_heading_containers(soup)
+        imported_tagline = extract_basic_information_tagline(soup, top_heading_level)
+    soup, instructions_tables = process_nofo_html(soup, top_heading_level)
+    if extract_scoped_tagline:
+        imported_tagline = consume_tagline_paragraph(soup, imported_tagline)
+    if section_parser is None:
+        section_parser = get_sections_and_subsections_from_soup
+    if builder_import or extract_scoped_tagline:
+        sections = section_parser(
+            deepcopy(soup) if extract_scoped_tagline else soup,
+            top_heading_level,
+            consume_scoped_tagline=extract_scoped_tagline,
+            consume_metadata=builder_import,
+        )
+    else:
+        sections = section_parser(soup, top_heading_level)
+    return ProcessedDocumentContent(
+        soup, sections, instructions_tables, imported_tagline
+    )

nofos/nofos/tests_nofos/test_document_processing.py

@@ -0,0 +1,94 @@
+"""Direct content callers need no request, upload, user or document record."""
+
+from bs4 import BeautifulSoup
+from django.core.exceptions import ValidationError
+from django.test import SimpleTestCase, override_settings
+
+from nofos.document_processing import process_document_content
+from nofos.nofo import (
+    decompose_before_you_begin_section,
+    get_sections_from_soup,
+    get_subsections_from_sections,
+    process_nofo_html,
+    replace_chars,
+    replace_links,
+    resolve_section_heading_level,
+)
+
+
+@override_settings(DEBUG=False)
+class DocumentProcessingTests(SimpleTestCase):
+    def test_builder_options_retain_source_and_consume_metadata_and_tagline(self):
+        html = (
+            "<h1>Step 1: Review the opportunity</h1>"
+            "<h2>Basic information</h2><h3>Tagline</h3><p>Apply today.</p>"
+            "<h2>Summary</h2><p>Applicant prose.</p>"
+            "<h1>Metadata</h1><p>Opdiv: CDC</p>"
+            "<p>Opportunity number: SYNTHETIC-001</p>"
+        )
+        result = process_document_content(
+            html, builder_import=True, extract_scoped_tagline=True
+        )
+        self.assertIn("Opdiv: CDC", str(result.soup))
+        self.assertNotIn("Opdiv: CDC", str(result.sections))
+        self.assertNotIn("Metadata", [section["name"] for section in result.sections])
+        self.assertIn("Applicant prose.", str(result.sections))
+        self.assertNotIn("Apply today.", str(result.sections))
+        self.assertEqual(result.imported_tagline, "Apply today.")
+
+    def test_direct_content_preserves_metadata_and_excludes_redundant_section(self):
+        result = process_document_content(
+            "<h1>Before you begin</h1><p>Redundant text.</p>"
+            "<p>Opdiv: CDC</p><h1>Opportunity</h1>"
+            "<h2>Summary</h2><p>Public&nbsp;agencies may apply.</p>"
+        )
+        self.assertIn("Opdiv: CDC", str(result.soup))
+        self.assertNotIn("Redundant text.", str(result.soup))
+        self.assertEqual([s["name"] for s in result.sections], ["Opportunity"])
+        self.assertNotIn("Opdiv: CDC", str(result.sections))
+        self.assertIn("Public agencies may apply.", str(result.sections))
+
+    def test_existing_processing_sequence_has_identical_results(self):
+        # Pre-extraction sequence provides a concrete before/after oracle.
+        shapes = (
+            "<div><h1>Opportunity</h1><h2>Eligibility</h2><p>Public agencies.</p></div>",
+            "<h1>Opportunity</h1><h2>Awards</h2><table><caption>Funding</caption><tr><td>Program</td><td>Amount</td></tr><tr><td>Training</td><td>0</td></tr></table>",
+            '<h1>Opportunity</h1><h2>Summary</h2><img src="logo.png" alt="Agency logo"><p><a href="#note-1">Read note</a></p><ol><li id="note-1">Source</li></ol>',
+            "<p>Opdiv: CDC</p><h1>Opportunity</h1><h2>Summary</h2>"
+            "<table><tr><td><p>Instructions for NOFO writers</p><p>Keep this instruction.</p></td></tr></table><p>Applicant text.</p>",
+        )
+        for html in shapes:
+            with self.subTest(html=html):
+                before = BeautifulSoup(
+                    replace_links(replace_chars(html)), "html.parser"
+                )
+                decompose_before_you_begin_section(before)
+                level = resolve_section_heading_level(before)
+                before, instructions = process_nofo_html(before, level)
+                sections = get_subsections_from_sections(
+                    get_sections_from_soup(before, level), level
+                )
+                after = process_document_content(html)
+                self.assertEqual(str(after.soup), str(before))
+                self.assertEqual(str(after.sections), str(sections))
+                self.assertEqual(str(after.instructions_tables), str(instructions))
+
+    def test_section_parser_extension_keeps_mutable_identity(self):
+        sections = [{"name": "Consumer result", "subsections": []}]
+        calls = []
+
+        def section_parser(soup, level):
+            calls.append((soup, level))
+            return sections
+
+        result = process_document_content(
+            "<h1>Opportunity</h1>", section_parser=section_parser
+        )
+        self.assertIs(result.sections, sections)
+        self.assertIs(calls[0][0], result.soup)
+        self.assertEqual(calls[0][1], "h1")
+
+    def test_no_sections_failure_propagates_without_web_presentation(self):
+        with self.assertRaises(ValidationError) as error:
+            process_document_content("<p>Only body text.</p>")
+        self.assertEqual(error.exception.code, "no_sections")

nofos/nofos/tests_nofos/test_import_characterization.py

@@ -90,8 +90,18 @@ def instruction_hook(**kwargs):
         }
         with ExitStack() as stack:
             for function, (name, result) in transforms.items():
+                module = (
+                    "nofos.views."
+                    if function
+                    in {
+                        "parse_uploaded_file_as_html_string",
+                        "add_final_subsection_to_step_3",
+                        "add_line_breaks_to_key_dates_values",
+                    }
+                    else "nofos.document_processing."
+                )
                 stack.enter_context(
-                    patch("nofos.views." + function, side_effect=step(name, result))
+                    patch(module + function, side_effect=step(name, result))
                 )
             stack.enter_context(
                 patch.object(