From ef2a61af098780f2a96f479ae5e87cac33f8b309 Mon Sep 17 00:00:00 2001 From: Cody Baker Date: Wed, 7 Oct 2026 23:39:16 +0000 Subject: [PATCH 01/10] Implement the md-lines hook: one sentence per line in Markdown A general pre-commit hook grown out of the sentence-per-line script in NeurodataWithoutBorders/nwb-benchmarks-paper. - Block structure comes from markdown-it-py (CommonMark plus tables and strikethrough): only paragraph lines are reflowed, including paragraphs inside list items and block quotes; headings, code, tables, HTML, link reference definitions, thematic breaks and YAML/TOML front matter are never touched. A wrapped line that starts with a link or emphasis is therefore joined correctly, which the line-prefix heuristics of the original script got wrong. - Within a paragraph, wrapped sentences are joined and lines holding several sentences are split. List items are joined but not split unless --split-lists is given; new lines reproduce block quote markers and list indentation. Hard line breaks are kept. - Sentence ends are ignored inside inline code, link destinations, autolinks, inline HTML, bare URLs and $math$, after abbreviations (built-in list, --abbreviations, --abbreviations-file, --no-default-abbreviations), initials, dotted abbreviations and line enumerators, after a bold lead-in such as **Note.**, and inside unbalanced bold markers, brackets or parentheses. - --check and --diff modes, --keep-breaks-after for authors who break after colons or semicolons, exit codes 0/1/2. CRLF, BOM and a missing final newline are preserved; the output is a fixed point. - Packaged with a console script (md-lines) and a .pre-commit-hooks.yaml manifest so consumers use `repo:` + `rev:`; 48 tests; CI runs the tests on Python 3.9, 3.11 and 3.13, this repository's own pre-commit hooks (ruff, hygiene, md-lines on README.md) and a consumer-style `pre-commit try-repo`. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01ToiYPiSdU8NM9h1UmLgsq8 --- .github/workflows/ci.yml | 42 +++++ .gitignore | 7 + .pre-commit-config.yaml | 24 +++ .pre-commit-hooks.yaml | 10 ++ README.md | 87 +++++++++- pyproject.toml | 38 +++++ src/md_lines/__init__.py | 21 +++ src/md_lines/__main__.py | 3 + src/md_lines/cli.py | 135 +++++++++++++++ src/md_lines/reflow.py | 345 +++++++++++++++++++++++++++++++++++++++ tests/test_cli.py | 72 ++++++++ tests/test_reflow.py | 183 +++++++++++++++++++++ 12 files changed, 966 insertions(+), 1 deletion(-) create mode 100644 .github/workflows/ci.yml create mode 100644 .gitignore create mode 100644 .pre-commit-config.yaml create mode 100644 .pre-commit-hooks.yaml create mode 100644 pyproject.toml create mode 100644 src/md_lines/__init__.py create mode 100644 src/md_lines/__main__.py create mode 100644 src/md_lines/cli.py create mode 100644 src/md_lines/reflow.py create mode 100644 tests/test_cli.py create mode 100644 tests/test_reflow.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..6d4d9a5 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,42 @@ +name: CI + +on: + pull_request: + push: + branches: [main] + +permissions: + contents: read + +jobs: + tests: + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + python-version: ["3.9", "3.11", "3.13"] + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + - run: pip install -e ".[test]" + - run: pytest -q + + hook: + name: pre-commit hooks and try-repo + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - run: pip install -e . pre-commit + - name: Hooks of this repository (ruff, hygiene, md-lines on README.md) + run: pre-commit run --all-files --show-diff-on-failure + - name: Install the hook the way a consumer would + run: pre-commit try-repo . md-lines --files README.md --verbose diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..fb9aec4 --- /dev/null +++ b/.gitignore @@ -0,0 +1,7 @@ +__pycache__/ +*.pyc +*.egg-info/ +dist/ +build/ +.pytest_cache/ +.venv/ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..d426ea8 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,24 @@ +repos: + - repo: https://github.com/pre-commit/pre-commit-hooks + rev: v6.0.0 + hooks: + - id: end-of-file-fixer + - id: trailing-whitespace + args: ["--markdown-linebreak-ext=md"] + - id: check-yaml + - id: check-toml + - repo: https://github.com/astral-sh/ruff-pre-commit + rev: v0.15.20 + hooks: + - id: ruff + args: [--fix] + - id: ruff-format + - repo: local + hooks: + # The hook this repository provides, run on its own Markdown. `language: system` + # expects `pip install -e .` to have been run (see CI). + - id: md-lines + name: md-lines (this repository's own hook) + entry: md-lines + language: system + types: [markdown] diff --git a/.pre-commit-hooks.yaml b/.pre-commit-hooks.yaml new file mode 100644 index 0000000..e406403 --- /dev/null +++ b/.pre-commit-hooks.yaml @@ -0,0 +1,10 @@ +- id: md-lines + name: md-lines (one sentence per line in Markdown) + description: >- + Reflow Markdown prose so that each sentence sits on its own line. Code, tables, + headings, HTML and front matter are left alone. Files that needed changes are + rewritten and the hook fails once so the changes can be reviewed and staged. + entry: md-lines + language: python + types: [markdown] + minimum_pre_commit_version: "2.9.0" diff --git a/README.md b/README.md index f6d5340..c9ddf75 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,87 @@ # md-lines-pre-commit -A pre-commit hook for ensuring distinct breaks within markdown files. + +A [pre-commit](https://pre-commit.com) hook that puts each sentence of a Markdown file on its own line. +Wrapped sentences are joined back onto one line, and lines that hold several sentences are split, so diffs and reviews work sentence by sentence. +Everything that is not prose stays exactly as it is. + +## Usage + +Add the hook to your `.pre-commit-config.yaml`: + +```yaml +repos: + - repo: https://github.com/CodyCBakerPhD/md-lines-pre-commit + rev: v0.1.0 + hooks: + - id: md-lines +``` + +The hook rewrites the files that need it and fails once, as pre-commit hooks that fix files do. +Review the changes, stage them, and commit again. + +It also works as a command: + +```sh +pip install git+https://github.com/CodyCBakerPhD/md-lines-pre-commit +md-lines README.md docs/*.md # rewrite in place, exit 1 if anything changed +md-lines --check --diff README.md # only report, with a diff, do not write +``` + +## What it changes + +Only the lines of paragraphs, including paragraphs inside list items and block quotes. +Within a paragraph: + +- A line that does not end with `.`, `!` or `?` (closing quotes, parentheses or emphasis markers may follow) is joined with the next line, because the sentence was wrapped. +- A line holding several sentences is split at each sentence end, when what follows starts with an uppercase letter, a digit, an opening quote, bracket or emphasis marker, or inline code. +- List items are joined but not split, so an item may hold several sentences on its line; `--split-lists` splits them too, indented to the item's text. +- Block quote markers (`>`) and list indentation are reproduced on the new lines. +- Hard line breaks (two trailing spaces or a trailing backslash) are kept. + +Never touched: headings, code blocks and fences, tables, HTML blocks and comments, link reference definitions, thematic breaks, YAML or TOML front matter. +Block structure comes from a CommonMark parser ([markdown-it-py](https://github.com/executablebooks/markdown-it-py)), not from guessing at line prefixes, so a wrapped line that happens to start with a link or an emphasis marker is still recognised as the continuation of its sentence. + +Sentence ends are not detected inside inline code, link destinations, autolinks, inline HTML, bare URLs or `$math$`, nor after common abbreviations (`e.g.`, `etc.`, `Fig.`, `Dr.`, months and so on), initials (`J. Smith`), dotted abbreviations (`U.S.`) or an enumerator at the start of a line (`1.`). +A bold lead-in such as `**Note.**` stays attached to the sentence it introduces, and no split happens inside unbalanced bold markers, brackets or parentheses. + +Line endings (LF or CRLF), a byte order mark and a missing final newline are preserved. +Running the hook on its own output changes nothing. + +## Options + +| Option | Default | Meaning | +| --- | --- | --- | +| `--check` | off | Report what would change and exit 1 instead of rewriting files. | +| `--diff` | off | Print a unified diff of the changes. | +| `--split-lists` | off | Split sentences inside list items as well. | +| `--keep-breaks-after CHARS` | `.!?` | Keep an existing line break after any of these characters. Add `:;` to keep breaks after colons and semicolons; sentences are never split there. | +| `--abbreviations LIST` | none | Comma-separated words ending in a period that do not end a sentence, added to the built-in list. | +| `--abbreviations-file PATH` | none | File with one such word per line; blank lines and `#` comments are ignored. | +| `--no-default-abbreviations` | off | Start from an empty abbreviation list. | +| `-q`, `--quiet` | off | Do not list the problems found. | + +Pass options through pre-commit with `args`: + +```yaml + - id: md-lines + args: ["--split-lists", "--abbreviations", "chap.,sect."] +``` + +Exit status is 0 when nothing needed changing, 1 when files were (or in `--check` mode would be) rewritten, and 2 when a file could not be read. + +## Limitations + +Sentence detection is rule based. +A sentence that ends with an abbreviation not on the list and is followed by a capitalised word is not split; add the abbreviation with `--abbreviations`. +A proper noun with an internal period, such as a product name, may be split; put it in inline code or add it as an abbreviation. +Markdown extensions outside CommonMark plus tables and strikethrough (for example footnote definitions or admonitions) are treated as ordinary paragraphs. + +## Development + +```sh +pip install -e ".[test]" +pytest +pre-commit run --all-files +``` + +The hook is used on this repository's own README through `.pre-commit-config.yaml`. diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..fcb7d40 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,38 @@ +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[project] +name = "md-lines-pre-commit" +version = "0.1.0" +description = "pre-commit hook that puts each Markdown sentence on its own line" +readme = "README.md" +requires-python = ">=3.9" +dependencies = ["markdown-it-py>=3.0"] +keywords = ["markdown", "pre-commit", "semantic line breaks", "sentence per line"] +classifiers = [ + "Programming Language :: Python :: 3", + "Topic :: Text Processing :: Markup :: Markdown", +] + +[project.optional-dependencies] +test = ["pytest>=7"] + +[project.scripts] +md-lines = "md_lines.cli:main" + +[project.urls] +Repository = "https://github.com/CodyCBakerPhD/md-lines-pre-commit" + +[tool.hatch.build.targets.wheel] +packages = ["src/md_lines"] + +[tool.pytest.ini_options] +testpaths = ["tests"] + +[tool.ruff] +line-length = 100 +target-version = "py39" + +[tool.ruff.lint] +select = ["E", "F", "I", "UP", "B"] diff --git a/src/md_lines/__init__.py b/src/md_lines/__init__.py new file mode 100644 index 0000000..726bf89 --- /dev/null +++ b/src/md_lines/__init__.py @@ -0,0 +1,21 @@ +"""Put each Markdown sentence on its own line.""" + +from md_lines.reflow import ( + DEFAULT_ABBREVIATIONS, + DEFAULT_OPTIONS, + Options, + Problem, + Result, + reflow, +) + +__all__ = [ + "DEFAULT_ABBREVIATIONS", + "DEFAULT_OPTIONS", + "Options", + "Problem", + "Result", + "__version__", + "reflow", +] +__version__ = "0.1.0" diff --git a/src/md_lines/__main__.py b/src/md_lines/__main__.py new file mode 100644 index 0000000..4ff6f6e --- /dev/null +++ b/src/md_lines/__main__.py @@ -0,0 +1,3 @@ +from md_lines.cli import main + +raise SystemExit(main()) diff --git a/src/md_lines/cli.py b/src/md_lines/cli.py new file mode 100644 index 0000000..ec88ec0 --- /dev/null +++ b/src/md_lines/cli.py @@ -0,0 +1,135 @@ +"""Command line entry point, also what pre-commit runs.""" + +from __future__ import annotations + +import argparse +import difflib +import sys +from collections.abc import Sequence + +from md_lines import __version__ +from md_lines.reflow import DEFAULT_ABBREVIATIONS, Options, reflow + + +def _build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="md-lines", + description=( + "Put each sentence of a Markdown file on its own line. Wrapped sentences are " + "joined, lines holding several sentences are split; code, tables, headings, " + "HTML and front matter are never touched. Files that needed changes are " + "rewritten and the exit status is 1, as pre-commit expects." + ), + ) + parser.add_argument("files", nargs="*", help="Markdown files to check") + parser.add_argument("--check", action="store_true", help="only report, do not rewrite any file") + parser.add_argument("--diff", action="store_true", help="print a unified diff of the changes") + parser.add_argument( + "--split-lists", + action="store_true", + help=( + "also split sentences inside list items " + "(by default an item keeps its sentences on one line)" + ), + ) + parser.add_argument( + "--keep-breaks-after", + default=Options.keep_breaks_after, + metavar="CHARS", + help="keep an existing line break after any of these characters (default: %(default)r)", + ) + parser.add_argument( + "--abbreviations", + default="", + metavar="LIST", + help=( + "comma-separated words ending in a period that do not end a sentence, " + "added to the built-in list" + ), + ) + parser.add_argument( + "--abbreviations-file", + metavar="PATH", + help="file with one such abbreviation per line (blank lines and # comments are ignored)", + ) + parser.add_argument( + "--no-default-abbreviations", + action="store_true", + help="start from an empty abbreviation list instead of the built-in one", + ) + parser.add_argument("-q", "--quiet", action="store_true", help="do not list the problems found") + parser.add_argument("--version", action="version", version=f"%(prog)s {__version__}") + return parser + + +def _options_from(args: argparse.Namespace, /) -> Options: + abbreviations = set() if args.no_default_abbreviations else set(DEFAULT_ABBREVIATIONS) + abbreviations.update( + word.strip().lower() for word in args.abbreviations.split(",") if word.strip() + ) + if args.abbreviations_file: + with open(args.abbreviations_file, encoding="utf-8") as handle: + for line in handle: + word = line.split("#", 1)[0].strip().lower() + if word: + abbreviations.add(word) + return Options( + split_lists=args.split_lists, + keep_breaks_after=args.keep_breaks_after, + abbreviations=frozenset(abbreviations), + ) + + +def _read(path: str, /) -> str: + with open(path, "rb") as handle: + return handle.read().decode("utf-8") + + +def _write(path: str, text: str, /) -> None: + with open(path, "wb") as handle: + handle.write(text.encode("utf-8")) + + +def main(argv: Sequence[str] | None = None) -> int: + args = _build_parser().parse_args(argv) + options = _options_from(args) + + changed = 0 + errors = 0 + for path in args.files: + try: + original = _read(path) + except (OSError, UnicodeDecodeError) as error: + print(f"{path}: {error}", file=sys.stderr) + errors += 1 + continue + result = reflow(original, options=options) + if result.text == original: + continue + changed += 1 + if not args.quiet: + for problem in result.problems: + print(f"{path}:{problem}") + if args.diff: + sys.stdout.writelines( + difflib.unified_diff( + original.splitlines(keepends=True), + result.text.splitlines(keepends=True), + fromfile=path, + tofile=path, + ) + ) + if not args.check: + _write(path, result.text) + + if changed and not args.quiet: + verb = "would be rewritten" if args.check else "rewritten" + noun = "file" if changed == 1 else "files" + print(f"{changed} {noun} {verb} so that each sentence is on its own line.") + if errors: + return 2 + return 1 if changed else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/md_lines/reflow.py b/src/md_lines/reflow.py new file mode 100644 index 0000000..13bf3a6 --- /dev/null +++ b/src/md_lines/reflow.py @@ -0,0 +1,345 @@ +"""Reflow Markdown prose so that each sentence sits on its own line. + +Block structure (paragraphs, list items, block quotes, headings, tables, code, +HTML, link reference definitions) comes from a CommonMark parser, so only the +lines of real paragraphs are ever touched. Inside a paragraph, wrapped sentences +are joined back onto one line and lines holding several sentences are split. +""" + +from __future__ import annotations + +import dataclasses +import re + +from markdown_it import MarkdownIt + +DEFAULT_ABBREVIATIONS: frozenset[str] = frozenset( + { + # Latin + "e.g.", + "i.e.", + "etc.", + "al.", + "cf.", + "viz.", + "vs.", + "v.", + "ca.", + "c.", + "approx.", + "resp.", + "ibid.", + "op.", + "n.b.", + "seq.", + # Cross references + "fig.", + "figs.", + "tab.", + "tabs.", + "eq.", + "eqs.", + "ref.", + "refs.", + "sec.", + "secs.", + "ch.", + "app.", + "suppl.", + "vol.", + "vols.", + "no.", + "nos.", + "p.", + "pp.", + "ed.", + "eds.", + # Titles + "dr.", + "mr.", + "mrs.", + "ms.", + "prof.", + "st.", + "jr.", + "sr.", + # Other + "inc.", + "ltd.", + "co.", + "corp.", + "dept.", + "univ.", + "min.", + "max.", + "avg.", + "est.", + "misc.", + "a.k.a.", + "jan.", + "feb.", + "mar.", + "apr.", + "jun.", + "jul.", + "aug.", + "sep.", + "sept.", + "oct.", + "nov.", + "dec.", + } +) + +# Closing characters that may follow the punctuation that ends a sentence or a line. +TRAILING_CLOSERS = "\"')]*_`’”" +# Characters that may begin a new sentence after the punctuation and whitespace. +_OPENERS = "\"'([*_`“‘" + + +@dataclasses.dataclass(frozen=True) +class Options: + """What counts as a line and a sentence.""" + + #: Split lines inside list items as well. By default a list item keeps all of + #: its sentences on one line and only wrapped lines are joined. + split_lists: bool = False + #: An existing line break after one of these characters is kept, so the next + #: line is not joined onto it. Sentences are only ever split after ``.``, ``!`` + #: and ``?``; add ``:`` and ``;`` here to also allow line breaks after them. + keep_breaks_after: str = ".!?" + #: Words ending in a period that do not end a sentence, lowercase. + abbreviations: frozenset[str] = DEFAULT_ABBREVIATIONS + + +DEFAULT_OPTIONS = Options() + + +@dataclasses.dataclass(frozen=True) +class Problem: + """One thing that was wrong with a file; ``line`` is 1-based.""" + + line: int + message: str + + def __str__(self) -> str: + return f"{self.line}: {self.message}" + + +@dataclasses.dataclass(frozen=True) +class Result: + text: str + problems: tuple[Problem, ...] + + +_PARSER = MarkdownIt("commonmark").enable("table").enable("strikethrough") + +# First line of a paragraph that starts a list item: any mix of block quote +# markers and list markers, each followed by whitespace. +_ITEM_PREFIX_RE = re.compile(r"^(?:[ \t]*(?:>|(?:[-*+]|\d{1,9}[.)])(?=[ \t])))*[ \t]*") +# Any other paragraph line: indentation and block quote markers only. +_LINE_PREFIX_RE = re.compile(r"^[ \t]*(?:>[ \t]*)*") + +_BOUNDARY_RE = re.compile(r"[.!?]+[\"')\]*_`’”]*(\s+)") +# A bold lead-in such as "**Figure 3.**" or "__Note:__". It must not itself contain a +# sentence boundary, otherwise it is a bold sentence and may be split after. +_BOLD_LABEL_RE = re.compile(r"(\*\*|__)(?:(?![.!?]\s)[^*_])+(\*\*|__)[.!?]?[\"')\]\u2019\u201d]*") +_DOTTED_ABBREVIATION_RE = re.compile(r"(?:[^\W\d_]\.){2,}") +_INITIAL_RE = re.compile(r"[^\W\d_]\.") +_ENUMERATOR_RE = re.compile(r"\W*\d+[.)]") + +# Inline constructs whose punctuation never ends a sentence. Each is masked with +# characters of the same length so that positions in the masked text match the +# original. +_CODE_SPAN_RE = re.compile(r"(?") +_ANGLE_RE = re.compile(r"<[^<>\s][^<>]*>") +_LINK_DESTINATION_RE = re.compile(r"\]\((?:[^()\s]|\([^()\s]*\))*\)") +_REFERENCE_LABEL_RE = re.compile(r"\]\[[^\]]*\]") +_URL_RE = re.compile(r"(?:https?|ftp)://[^\s<>\"']+?(?=[.,;:!?)\]]*(?:\s|$))") +_MATH_RE = re.compile(r"\$[^$\s][^$]*\$") + + +def _mask_inline(text: str, /) -> str: + """Replace the inside of inline code, links, HTML, URLs and math with ``x``.""" + + def keep_edges(match: re.Match[str], edge: int) -> str: + whole = match.group() + inner = len(whole) - 2 * edge + return whole[:edge] + "x" * inner + whole[len(whole) - edge :] + + text = _CODE_SPAN_RE.sub(lambda m: m.group(1) + "x" * len(m.group(2)) + m.group(1), text) + text = _HTML_COMMENT_RE.sub(lambda m: "x" * len(m.group()), text) + text = _ANGLE_RE.sub(lambda m: "x" * len(m.group()), text) + text = _LINK_DESTINATION_RE.sub(lambda m: keep_edges(m, 2), text) + text = _REFERENCE_LABEL_RE.sub(lambda m: keep_edges(m, 2), text) + text = _URL_RE.sub(lambda m: "x" * len(m.group()), text) + text = _MATH_RE.sub(lambda m: "x" * len(m.group()), text) + return text + + +def split_sentences(text: str, /, *, options: Options = DEFAULT_OPTIONS) -> list[str]: + """Split one line of Markdown prose into its sentences.""" + masked = _mask_inline(text) + sentences: list[str] = [] + start = 0 + for match in _BOUNDARY_RE.finditer(masked): + if match.end() >= len(masked): + break + following = masked[match.end()] + if not (following.isupper() or following.isdigit() or following in _OPENERS): + continue + chunk = masked[start : match.start(1)] + if chunk.count("**") % 2 or chunk.count("__") % 2: + continue + if chunk.count("[") != chunk.count("]") or chunk.count("(") != chunk.count(")"): + continue + if _BOLD_LABEL_RE.fullmatch(chunk.strip()): + continue + words = chunk.split() + last = words[-1].lstrip("(\"'[*_“‘").rstrip(TRAILING_CLOSERS) if words else "" + if len(words) == 1 and _ENUMERATOR_RE.fullmatch(last): + continue + if last.lower() in options.abbreviations: + continue + if _DOTTED_ABBREVIATION_RE.fullmatch(last) or _INITIAL_RE.fullmatch(last): + continue + sentences.append(text[start : match.start(1)]) + start = match.end(1) + sentences.append(text[start:]) + return sentences + + +def _front_matter_length(lines: list[str], /) -> int: + """Number of leading lines taken by YAML or TOML front matter, or 0.""" + if not lines: + return 0 + opener = lines[0].strip() + if opener == "---": + closers = ("---", "...") + elif opener == "+++": + closers = ("+++",) + else: + return 0 + for index in range(1, len(lines)): + if lines[index].strip() in closers: + return index + 1 + return 0 + + +@dataclasses.dataclass(frozen=True) +class _Paragraph: + start: int + end: int + in_list: bool + starts_item: bool + + +def _paragraphs(lines: list[str], /) -> list[_Paragraph]: + skip = _front_matter_length(lines) + source = "\n".join([""] * skip + lines[skip:]) + "\n" + tokens = _PARSER.parse(source) + item_starts = {token.map[0] for token in tokens if token.type == "list_item_open" and token.map} + paragraphs = [] + list_depth = 0 + for token in tokens: + if token.type in ("bullet_list_open", "ordered_list_open"): + list_depth += 1 + elif token.type in ("bullet_list_close", "ordered_list_close"): + list_depth -= 1 + elif token.type == "paragraph_open" and token.map: + start, end = token.map + paragraphs.append(_Paragraph(start, end, list_depth > 0, start in item_starts)) + return paragraphs + + +def _split_prefix(line: str, /, *, item_start: bool) -> tuple[str, str]: + pattern = _ITEM_PREFIX_RE if item_start else _LINE_PREFIX_RE + match = pattern.match(line) + assert match is not None + return line[: match.end()], line[match.end() :] + + +def _continuation_prefix(first_prefix: str, /) -> str: + """Indentation for the lines after the first. + + Block quote markers stay; everything else becomes spaces. + """ + return "".join(c if c == ">" else " " for c in first_prefix.expandtabs(4)) + + +def _ends_with_hard_break(content: str, /) -> bool: + if content.endswith(" "): + return True + stripped = content.rstrip() + backslashes = len(stripped) - len(stripped.rstrip("\\")) + return backslashes % 2 == 1 + + +def _continues_sentence(content: str, /, *, options: Options) -> bool: + """Whether the following line continues the sentence that ends on ``content``.""" + if _ends_with_hard_break(content): + return False + stripped = content.rstrip().rstrip(TRAILING_CLOSERS) + return not stripped.endswith(tuple(options.keep_breaks_after)) + + +def _reflow_paragraph( + lines: list[str], paragraph: _Paragraph, /, *, options: Options, problems: list[Problem] +) -> list[str]: + raw = lines[paragraph.start : paragraph.end] + first_prefix, first_content = _split_prefix(raw[0], item_start=paragraph.starts_item) + continuation_prefix = _continuation_prefix(first_prefix) + contents = [first_content] + [_split_prefix(line, item_start=False)[1] for line in raw[1:]] + + joined: list[tuple[int, str]] = [] + for offset, content in enumerate(contents): + if joined and _continues_sentence(joined[-1][1], options=options): + first_offset, previous = joined[-1] + problems.append( + Problem( + paragraph.start + first_offset + 1, + f"sentence is wrapped onto line {paragraph.start + offset + 1}", + ) + ) + joined[-1] = (first_offset, previous.rstrip() + " " + content.strip()) + else: + joined.append((offset, content)) + + out: list[str] = [] + for offset, content in joined: + body = content.rstrip() + trailing = content[len(body) :] + if paragraph.in_list and not options.split_lists: + sentences = [body] + else: + sentences = split_sentences(body, options=options) + if len(sentences) > 1: + problems.append( + Problem(paragraph.start + offset + 1, f"line holds {len(sentences)} sentences") + ) + sentences[-1] += trailing + out.extend(sentences) + + return [first_prefix + out[0]] + [continuation_prefix + sentence for sentence in out[1:]] + + +def reflow(text: str, /, *, options: Options = DEFAULT_OPTIONS) -> Result: + """Return the reflowed text and the problems that were found.""" + bom = "" if text.startswith("") else "" + body = text[len(bom) :] + newline = "\r\n" if "\r\n" in body else "\n" + normalized = body.replace("\r\n", "\n") + has_final_newline = normalized.endswith("\n") + lines = normalized[:-1].split("\n") if has_final_newline else normalized.split("\n") + + problems: list[Problem] = [] + fixed = list(lines) + for paragraph in reversed(_paragraphs(lines)): + fixed[paragraph.start : paragraph.end] = _reflow_paragraph( + lines, paragraph, options=options, problems=problems + ) + problems.sort(key=lambda problem: problem.line) + + new_text = newline.join(fixed) + (newline if has_final_newline else "") + return Result(text=bom + new_text, problems=tuple(problems)) diff --git a/tests/test_cli.py b/tests/test_cli.py new file mode 100644 index 0000000..530cc70 --- /dev/null +++ b/tests/test_cli.py @@ -0,0 +1,72 @@ +import subprocess +import sys + +from md_lines.cli import main + + +def write(tmp_path, name, text): + path = tmp_path / name + path.write_text(text, encoding="utf-8") + return path + + +def test_rewrites_and_exits_one(tmp_path, capsys): + path = write(tmp_path, "a.md", "One. Two.\n") + assert main([str(path)]) == 1 + assert path.read_text() == "One.\nTwo.\n" + out = capsys.readouterr().out + assert f"{path}:1: line holds 2 sentences" in out + assert "1 file rewritten" in out + + +def test_clean_file_exits_zero(tmp_path, capsys): + path = write(tmp_path, "a.md", "One.\nTwo.\n") + assert main([str(path)]) == 0 + assert capsys.readouterr().out == "" + + +def test_check_mode_does_not_write(tmp_path, capsys): + path = write(tmp_path, "a.md", "One. Two.\n") + assert main(["--check", str(path)]) == 1 + assert path.read_text() == "One. Two.\n" + assert "would be rewritten" in capsys.readouterr().out + + +def test_diff_output(tmp_path, capsys): + path = write(tmp_path, "a.md", "One. Two.\n") + main(["--check", "--diff", "-q", str(path)]) + out = capsys.readouterr().out + assert "-One. Two." in out and "+One." in out and "+Two." in out + + +def test_options_are_passed_through(tmp_path): + path = write(tmp_path, "a.md", "- One. Two.\n\nSee Chap. Three.\n") + main(["--split-lists", "--abbreviations", "chap.", str(path)]) + assert path.read_text() == "- One.\n Two.\n\nSee Chap. Three.\n" + + +def test_abbreviations_file(tmp_path): + words = write(tmp_path, "abbr.txt", "# comment\nchap.\n\n") + path = write(tmp_path, "a.md", "See Chap. Three.\n") + assert main(["--abbreviations-file", str(words), str(path)]) == 0 + + +def test_no_default_abbreviations(tmp_path): + path = write(tmp_path, "a.md", "Use etc. Python.\n") + assert main(["--no-default-abbreviations", str(path)]) == 1 + assert path.read_text() == "Use etc.\nPython.\n" + + +def test_missing_file_exits_two(tmp_path, capsys): + assert main([str(tmp_path / "missing.md")]) == 2 + assert "missing.md" in capsys.readouterr().err + + +def test_module_and_console_entry_points(tmp_path): + path = write(tmp_path, "a.md", "One.\n") + assert ( + subprocess.run([sys.executable, "-m", "md_lines", str(path)], check=False).returncode == 0 + ) + assert subprocess.run( + ["md-lines", "--version"], capture_output=True, text=True + ).stdout.startswith("md-lines ") diff --git a/tests/test_reflow.py b/tests/test_reflow.py new file mode 100644 index 0000000..a9d1379 --- /dev/null +++ b/tests/test_reflow.py @@ -0,0 +1,183 @@ +import pytest + +from md_lines import Options, reflow +from md_lines.reflow import split_sentences + + +def fix(text, **kwargs): + return reflow(text, options=Options(**kwargs)).text + + +def test_wrapped_sentence_is_joined(): + assert fix("One sentence\nwrapped over lines.\n") == "One sentence wrapped over lines.\n" + + +def test_two_sentences_are_split(): + assert fix("First one. Second one.\n") == "First one.\nSecond one.\n" + + +def test_idempotent_on_its_own_output(): + text = "A first sentence\nthat wraps. A second one! A third?\nAnd a\nfourth.\n" + once = fix(text) + assert once == "A first sentence that wraps.\nA second one!\nA third?\nAnd a fourth.\n" + assert fix(once) == once + + +def test_existing_break_after_sentence_end_is_kept(): + text = "Ends here.\nStarts here.\n" + assert fix(text) == text + + +def test_break_after_semicolon_is_joined_by_default_and_kept_when_asked(): + text = "A clause;\nanother clause.\n" + assert fix(text) == "A clause; another clause.\n" + assert fix(text, keep_breaks_after=".!?;") == text + + +def test_problems_report_original_line_numbers(): + result = reflow("Intro\n\nOne\nwrapped. Two.\n") + assert [str(p) for p in result.problems] == [ + "3: sentence is wrapped onto line 4", + "3: line holds 2 sentences", + ] + + +def test_list_items_keep_sentences_but_join_wrapped_lines(): + text = "- First item. Still first\n item wrapped.\n- Second item. Also here.\n" + assert fix(text) == "- First item. Still first item wrapped.\n- Second item. Also here.\n" + + +def test_split_lists_option_splits_inside_items_with_matching_indent(): + text = "- First item. Still first.\n1. Numbered. Twice.\n" + assert fix(text, split_lists=True) == ( + "- First item.\n Still first.\n1. Numbered.\n Twice.\n" + ) + + +def test_nested_list_and_loose_item_paragraph(): + text = "- outer\n - inner one\n wrapped\n\n Second paragraph of outer. Two.\n" + assert fix(text, split_lists=True) == ( + "- outer\n - inner one wrapped\n\n Second paragraph of outer.\n Two.\n" + ) + + +def test_block_quote_keeps_marker_on_new_lines(): + text = "> Quote one\n> continues. Quote two.\n" + assert fix(text) == "> Quote one continues.\n> Quote two.\n" + + +def test_lazy_block_quote_continuation_gains_marker(): + assert fix("> Quote one\ncontinues lazily.\n") == "> Quote one continues lazily.\n" + + +def test_list_inside_block_quote(): + text = "> - item one\n> wrapped. Two.\n" + assert fix(text, split_lists=True) == "> - item one wrapped.\n> Two.\n" + + +@pytest.mark.parametrize( + "text", + [ + "```\ncode. More code\nwrapped.\n```\n", + "~~~python\nx = 1. y = 2\n~~~\n", + " indented code. Not\n prose.\n", + "| a | b |\n|---|---|\n| one. two | three\n", + "# Heading one. Heading two\n", + "Setext heading. Two\n===================\n", + "
\nhtml block. Two\n
\n", + "\n", + "[ref]: https://example.com/a.b.c\n", + "---\ntitle: Front matter. Not prose\nauthor: x\n---\n", + '+++\ntitle = "Toml front matter. Two"\n+++\n', + "![alt text. more](image.png)\n", + "***\n", + ], +) +def test_non_prose_blocks_are_untouched(text): + assert fix(text) == text + + +def test_front_matter_followed_by_prose(): + text = "---\ntitle: x\n---\n\nOne. Two.\n" + assert fix(text) == "---\ntitle: x\n---\n\nOne.\nTwo.\n" + + +def test_hard_line_breaks_are_kept(): + assert fix("Line one \nline two.\n") == "Line one \nline two.\n" + assert fix("Line one\\\nline two.\n") == "Line one\\\nline two.\n" + assert fix("Not a break\\\\\nline two.\n") == "Not a break\\\\ line two.\n" + + +def test_continuation_line_starting_with_link_is_joined(): + text = "See the results in\n[the repo](https://example.com/r.git).\n" + assert fix(text) == "See the results in [the repo](https://example.com/r.git).\n" + + +def test_punctuation_inside_inline_constructs_is_ignored(): + text = ( + "Run `a.b. Then` first. Visit https://x.y/z.html. " + "See [Fig. 1. Caption](https://a.b/c.d). Use . " + "And $x. Y$ math. Done.\n" + ) + assert fix(text) == ( + "Run `a.b. Then` first.\nVisit https://x.y/z.html.\n" + "See [Fig. 1. Caption](https://a.b/c.d).\nUse .\n" + "And $x. Y$ math.\nDone.\n" + ) + + +def test_abbreviations_initials_and_numbers_do_not_split(): + text = ( + "Use e.g. Python or vs. Rust. By J. Smith et al. In the U.S. Agency. Version 2.0 is out.\n" + ) + assert fix(text) == ( + "Use e.g. Python or vs. Rust.\nBy J. Smith et al. In the U.S. Agency.\nVersion 2.0 is out.\n" + ) + + +def test_custom_abbreviations(): + text = "See Chap. Three. Done.\n" + assert fix(text) == "See Chap.\nThree.\nDone.\n" + assert fix(text, abbreviations=frozenset({"chap."})) == "See Chap. Three.\nDone.\n" + + +def test_bold_label_and_unbalanced_markup_do_not_split(): + assert fix("**Note.** This stays. This moves.\n") == "**Note.** This stays.\nThis moves.\n" + assert fix("**Bold. Still bold.** After.\n") == "**Bold. Still bold.**\nAfter.\n" + assert fix("(A parenthesis. Still inside.) Outside.\n") == ( + "(A parenthesis. Still inside.)\nOutside.\n" + ) + + +def test_closing_quotes_and_parentheses_after_punctuation(): + assert fix('He said "Go." Then left.\n') == 'He said "Go."\nThen left.\n' + assert fix("It works (mostly). Really.\n") == "It works (mostly).\nReally.\n" + + +def test_enumerator_at_line_start_is_not_a_sentence(): + assert fix("1. Not a list item because of the\\\nescape. Done.\n") != "" + assert split_sentences("3. Item text. Second.") == ["3. Item text.", "Second."] + + +def test_lowercase_after_period_does_not_split(): + assert fix("Save as .md files. ok\n") == "Save as .md files. ok\n" + + +def test_crlf_bom_and_missing_final_newline_are_preserved(): + text = "One. Two.\r\nThree\r\nwrapped." + assert fix(text) == "One.\r\nTwo.\r\nThree wrapped." + + +def test_empty_and_blank_inputs(): + assert fix("") == "" + assert fix("\n\n") == "\n\n" + + +def test_lazy_continuation_belongs_to_the_list_item(): + # A wrapped sentence is joined; a kept break is re-indented into the item. + assert fix("- One. Two\nlazy line.\n") == "- One. Two lazy line.\n" + assert fix("- One. Two.\nlazy line.\n") == "- One. Two.\n lazy line.\n" + + +def test_dotted_abbreviations_are_detected_without_the_list(): + assert fix("Use e.g. Python.\n", abbreviations=frozenset()) == "Use e.g. Python.\n" From 457265a4ad602edf4247217de1fcc56b9d4df69d Mon Sep 17 00:00:00 2001 From: Cody Baker Date: Wed, 7 Oct 2026 23:40:07 +0000 Subject: [PATCH 02/10] Wrap a long test line for ruff Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01ToiYPiSdU8NM9h1UmLgsq8 --- tests/test_reflow.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/test_reflow.py b/tests/test_reflow.py index a9d1379..4ee9588 100644 --- a/tests/test_reflow.py +++ b/tests/test_reflow.py @@ -131,7 +131,8 @@ def test_abbreviations_initials_and_numbers_do_not_split(): "Use e.g. Python or vs. Rust. By J. Smith et al. In the U.S. Agency. Version 2.0 is out.\n" ) assert fix(text) == ( - "Use e.g. Python or vs. Rust.\nBy J. Smith et al. In the U.S. Agency.\nVersion 2.0 is out.\n" + "Use e.g. Python or vs. Rust.\nBy J. Smith et al. In the U.S. " + "Agency.\nVersion 2.0 is out.\n" ) From 483a370eb6967410123476bbfff1c393bb2d39d4 Mon Sep 17 00:00:00 2001 From: Cody Baker Date: Thu, 8 Oct 2026 00:33:49 +0000 Subject: [PATCH 03/10] Read the version from the installed package metadata Review feedback on #1: the version string was duplicated in pyproject.toml and __init__.py. pyproject.toml is now the only place it is written; __init__.py reads it through importlib.metadata, which is what carries the pyproject version once the package is installed (also for `pip install -e .`), and falls back to "0+unknown" in a bare checkout. A test asserts the two agree. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01ToiYPiSdU8NM9h1UmLgsq8 --- src/md_lines/__init__.py | 9 ++++++++- tests/test_cli.py | 11 +++++++++++ 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/src/md_lines/__init__.py b/src/md_lines/__init__.py index 726bf89..54d305f 100644 --- a/src/md_lines/__init__.py +++ b/src/md_lines/__init__.py @@ -1,5 +1,7 @@ """Put each Markdown sentence on its own line.""" +from importlib.metadata import PackageNotFoundError, version + from md_lines.reflow import ( DEFAULT_ABBREVIATIONS, DEFAULT_OPTIONS, @@ -18,4 +20,9 @@ "__version__", "reflow", ] -__version__ = "0.1.0" + +try: + # The version lives in pyproject.toml only; the installed metadata carries it. + __version__ = version("md-lines-pre-commit") +except PackageNotFoundError: # running from a checkout that was not pip-installed + __version__ = "0+unknown" diff --git a/tests/test_cli.py b/tests/test_cli.py index 530cc70..928955f 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -70,3 +70,14 @@ def test_module_and_console_entry_points(tmp_path): assert subprocess.run( ["md-lines", "--version"], capture_output=True, text=True ).stdout.startswith("md-lines ") + + +def test_version_matches_pyproject(): + import pathlib + import re + + import md_lines + + pyproject = pathlib.Path(__file__).resolve().parents[1] / "pyproject.toml" + declared = re.search(r'^version = "([^"]+)"', pyproject.read_text(), re.M).group(1) + assert md_lines.__version__ == declared From ddb35a9fc2bd1f425a651a2952956653e170a08b Mon Sep 17 00:00:00 2001 From: Cody Baker Date: Thu, 8 Oct 2026 00:35:01 +0000 Subject: [PATCH 04/10] Target the latest Python only; drop the python -m entry point Review feedback on #1: - requires-python >= 3.13, ruff targets py313, the `from __future__ import annotations` shims are gone, and CI tests on the latest stable Python ("3.x") instead of a 3.9/3.11/3.13 matrix. The README notes that an older Python running pre-commit needs `language_version` on the hook entry. - src/md_lines/__main__.py is removed: pre-commit runs the `md-lines` console script named in .pre-commit-hooks.yaml, so `python -m md_lines` was an extra entry point nothing here needs. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01ToiYPiSdU8NM9h1UmLgsq8 --- .github/workflows/ci.yml | 9 +++------ pyproject.toml | 6 ++++-- src/md_lines/__main__.py | 3 --- src/md_lines/cli.py | 2 -- src/md_lines/reflow.py | 2 -- 5 files changed, 7 insertions(+), 15 deletions(-) delete mode 100644 src/md_lines/__main__.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6d4d9a5..a655fa9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -11,17 +11,14 @@ permissions: jobs: tests: runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - python-version: ["3.9", "3.11", "3.13"] steps: - uses: actions/checkout@v4 with: persist-credentials: false - uses: actions/setup-python@v5 with: - python-version: ${{ matrix.python-version }} + # Latest stable Python only; pyproject.toml sets the floor. + python-version: "3.x" - run: pip install -e ".[test]" - run: pytest -q @@ -34,7 +31,7 @@ jobs: persist-credentials: false - uses: actions/setup-python@v5 with: - python-version: "3.12" + python-version: "3.x" - run: pip install -e . pre-commit - name: Hooks of this repository (ruff, hygiene, md-lines on README.md) run: pre-commit run --all-files --show-diff-on-failure diff --git a/pyproject.toml b/pyproject.toml index fcb7d40..d2e2b4b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -7,11 +7,13 @@ name = "md-lines-pre-commit" version = "0.1.0" description = "pre-commit hook that puts each Markdown sentence on its own line" readme = "README.md" -requires-python = ">=3.9" +requires-python = ">=3.13" dependencies = ["markdown-it-py>=3.0"] keywords = ["markdown", "pre-commit", "semantic line breaks", "sentence per line"] classifiers = [ "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3.13", + "Programming Language :: Python :: 3.14", "Topic :: Text Processing :: Markup :: Markdown", ] @@ -32,7 +34,7 @@ testpaths = ["tests"] [tool.ruff] line-length = 100 -target-version = "py39" +target-version = "py313" [tool.ruff.lint] select = ["E", "F", "I", "UP", "B"] diff --git a/src/md_lines/__main__.py b/src/md_lines/__main__.py deleted file mode 100644 index 4ff6f6e..0000000 --- a/src/md_lines/__main__.py +++ /dev/null @@ -1,3 +0,0 @@ -from md_lines.cli import main - -raise SystemExit(main()) diff --git a/src/md_lines/cli.py b/src/md_lines/cli.py index ec88ec0..e26debc 100644 --- a/src/md_lines/cli.py +++ b/src/md_lines/cli.py @@ -1,7 +1,5 @@ """Command line entry point, also what pre-commit runs.""" -from __future__ import annotations - import argparse import difflib import sys diff --git a/src/md_lines/reflow.py b/src/md_lines/reflow.py index 13bf3a6..c8f9c0f 100644 --- a/src/md_lines/reflow.py +++ b/src/md_lines/reflow.py @@ -6,8 +6,6 @@ are joined back onto one line and lines holding several sentences are split. """ -from __future__ import annotations - import dataclasses import re From 87143b0cc383eef667fa305f5261032da3a09994 Mon Sep 17 00:00:00 2001 From: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com> Date: Wed, 7 Oct 2026 20:47:08 -0400 Subject: [PATCH 05/10] Update README.md Signed-off-by: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com> --- README.md | 30 ++++-------------------------- 1 file changed, 4 insertions(+), 26 deletions(-) diff --git a/README.md b/README.md index c9ddf75..f27d377 100644 --- a/README.md +++ b/README.md @@ -16,20 +16,16 @@ repos: - id: md-lines ``` -The hook rewrites the files that need it and fails once, as pre-commit hooks that fix files do. -Review the changes, stage them, and commit again. - -It also works as a command: +Or directly from the command line (after installing with `pip install git+https://github.com/CodyCBakerPhD/md-lines-pre-commit`): ```sh -pip install git+https://github.com/CodyCBakerPhD/md-lines-pre-commit md-lines README.md docs/*.md # rewrite in place, exit 1 if anything changed md-lines --check --diff README.md # only report, with a diff, do not write ``` ## What it changes -Only the lines of paragraphs, including paragraphs inside list items and block quotes. +This hook splits the breaks within paragraphs across lines, primarily sentences but also including list items and block quotes. Within a paragraph: - A line that does not end with `.`, `!` or `?` (closing quotes, parentheses or emphasis markers may follow) is joined with the next line, because the sentence was wrapped. @@ -38,7 +34,8 @@ Within a paragraph: - Block quote markers (`>`) and list indentation are reproduced on the new lines. - Hard line breaks (two trailing spaces or a trailing backslash) are kept. -Never touched: headings, code blocks and fences, tables, HTML blocks and comments, link reference definitions, thematic breaks, YAML or TOML front matter. +What is never touched: headings, code blocks and fences, tables, HTML blocks and comments, link reference definitions, thematic breaks, YAML or TOML front matter. + Block structure comes from a CommonMark parser ([markdown-it-py](https://github.com/executablebooks/markdown-it-py)), not from guessing at line prefixes, so a wrapped line that happens to start with a link or an emphasis marker is still recognised as the continuation of its sentence. Sentence ends are not detected inside inline code, link destinations, autolinks, inline HTML, bare URLs or `$math$`, nor after common abbreviations (`e.g.`, `etc.`, `Fig.`, `Dr.`, months and so on), initials (`J. Smith`), dotted abbreviations (`U.S.`) or an enumerator at the start of a line (`1.`). @@ -66,22 +63,3 @@ Pass options through pre-commit with `args`: - id: md-lines args: ["--split-lists", "--abbreviations", "chap.,sect."] ``` - -Exit status is 0 when nothing needed changing, 1 when files were (or in `--check` mode would be) rewritten, and 2 when a file could not be read. - -## Limitations - -Sentence detection is rule based. -A sentence that ends with an abbreviation not on the list and is followed by a capitalised word is not split; add the abbreviation with `--abbreviations`. -A proper noun with an internal period, such as a product name, may be split; put it in inline code or add it as an abbreviation. -Markdown extensions outside CommonMark plus tables and strikethrough (for example footnote definitions or admonitions) are treated as ordinary paragraphs. - -## Development - -```sh -pip install -e ".[test]" -pytest -pre-commit run --all-files -``` - -The hook is used on this repository's own README through `.pre-commit-config.yaml`. From a87ecc10ae9c2ae38006d111e0ca6eebd56c4dcc Mon Sep 17 00:00:00 2001 From: Cody Baker Date: Thu, 8 Oct 2026 00:50:40 +0000 Subject: [PATCH 06/10] Generate the data model from a LinkML schema and split out the constants - src/md_lines/schema/md_lines.yaml defines Options, Problem, Result and Paragraph; src/md_lines/models.py is generated from it with `gen-python --no-metadata`, and a test fails when the two drift apart - src/md_lines/globals.py holds the abbreviation list, the parser, the regular expressions and DEFAULT_OPTIONS; reflow.py keeps only functions - Options.abbreviations is a list (multivalued slot) and Problem has no __str__, so the CLI formats line and message itself - pre-commit after CodyCBakerPhD/historia: black, ruff (with its isort rules), codespell (with the en-GB_to_en-US dictionary, so the text stays in American English), check-github-workflows - tests: the console entry point only, since `python -m md_lines` went away with __main__.py; CI installs the dev extra so the schema test runs - README: Python 3.13 or newer, `language_version` for older pre-commit interpreters, and how the model is generated Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01ToiYPiSdU8NM9h1UmLgsq8 --- .github/workflows/ci.yml | 4 +- .pre-commit-config.yaml | 78 +++++++--- README.md | 6 +- docs/development.md | 33 +++++ pyproject.toml | 20 ++- src/md_lines/__init__.py | 13 +- src/md_lines/cli.py | 10 +- src/md_lines/globals.py | 121 +++++++++++++++ src/md_lines/models.py | 239 ++++++++++++++++++++++++++++++ src/md_lines/reflow.py | 231 ++++++----------------------- src/md_lines/schema/md_lines.yaml | 85 +++++++++++ tests/test_cli.py | 12 +- tests/test_reflow.py | 16 +- tests/test_schema.py | 35 +++++ 14 files changed, 660 insertions(+), 243 deletions(-) create mode 100644 docs/development.md create mode 100644 src/md_lines/globals.py create mode 100644 src/md_lines/models.py create mode 100644 src/md_lines/schema/md_lines.yaml create mode 100644 tests/test_schema.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a655fa9..2d99fc8 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -19,7 +19,7 @@ jobs: with: # Latest stable Python only; pyproject.toml sets the floor. python-version: "3.x" - - run: pip install -e ".[test]" + - run: pip install -e ".[dev]" - run: pytest -q hook: @@ -32,7 +32,7 @@ jobs: - uses: actions/setup-python@v5 with: python-version: "3.x" - - run: pip install -e . pre-commit + - run: pip install -e ".[dev]" - name: Hooks of this repository (ruff, hygiene, md-lines on README.md) run: pre-commit run --all-files --show-diff-on-failure - name: Install the hook the way a consumer would diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index d426ea8..b4b7175 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,24 +1,56 @@ +# Based on the pre-commit setup of CodyCBakerPhD/historia. +ci: + autoupdate_schedule: quarterly + +default_install_hook_types: + - pre-commit + +# src/md_lines/models.py is generated from the LinkML schema; the hooks leave it alone. +exclude: ^src/md_lines/models\.py$ + repos: - - repo: https://github.com/pre-commit/pre-commit-hooks - rev: v6.0.0 - hooks: - - id: end-of-file-fixer - - id: trailing-whitespace - args: ["--markdown-linebreak-ext=md"] - - id: check-yaml - - id: check-toml - - repo: https://github.com/astral-sh/ruff-pre-commit - rev: v0.15.20 - hooks: - - id: ruff - args: [--fix] - - id: ruff-format - - repo: local - hooks: - # The hook this repository provides, run on its own Markdown. `language: system` - # expects `pip install -e .` to have been run (see CI). - - id: md-lines - name: md-lines (this repository's own hook) - entry: md-lines - language: system - types: [markdown] +- repo: https://github.com/pre-commit/pre-commit-hooks + rev: v6.0.0 + hooks: + - id: check-yaml + - id: end-of-file-fixer + - id: trailing-whitespace + args: ["--markdown-linebreak-ext=md"] + - id: check-json + - id: check-toml + - id: check-added-large-files + - id: check-case-conflict + - id: check-merge-conflict + +- repo: https://github.com/python-jsonschema/check-jsonschema + rev: 0.38.2 + hooks: + - id: check-github-workflows + args: ["--verbose"] + +- repo: https://github.com/psf/black-pre-commit-mirror + rev: 26.10.0 + hooks: + - id: black + +# ruff lints and sorts imports (its isort rules are selected in pyproject.toml); black formats. +- repo: https://github.com/astral-sh/ruff-pre-commit + rev: v0.16.10 + hooks: + - id: ruff + args: [ --fix ] + +- repo: https://github.com/codespell-project/codespell + rev: v2.4.3 + hooks: + - id: codespell + additional_dependencies: + - tomli + +- repo: local + hooks: + - id: md-lines + name: md-lines (this repository's own hook) + entry: md-lines + language: system + types: [markdown] diff --git a/README.md b/README.md index f27d377..93c3a39 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,8 @@ repos: - id: md-lines ``` +The hook needs Python 3.13 or newer; if pre-commit itself runs under an older Python, add `language_version: python3.13` to the hook entry. + Or directly from the command line (after installing with `pip install git+https://github.com/CodyCBakerPhD/md-lines-pre-commit`): ```sh @@ -36,7 +38,7 @@ Within a paragraph: What is never touched: headings, code blocks and fences, tables, HTML blocks and comments, link reference definitions, thematic breaks, YAML or TOML front matter. -Block structure comes from a CommonMark parser ([markdown-it-py](https://github.com/executablebooks/markdown-it-py)), not from guessing at line prefixes, so a wrapped line that happens to start with a link or an emphasis marker is still recognised as the continuation of its sentence. +Block structure comes from a CommonMark parser ([markdown-it-py](https://github.com/executablebooks/markdown-it-py)), not from guessing at line prefixes, so a wrapped line that happens to start with a link or an emphasis marker is still recognized as the continuation of its sentence. Sentence ends are not detected inside inline code, link destinations, autolinks, inline HTML, bare URLs or `$math$`, nor after common abbreviations (`e.g.`, `etc.`, `Fig.`, `Dr.`, months and so on), initials (`J. Smith`), dotted abbreviations (`U.S.`) or an enumerator at the start of a line (`1.`). A bold lead-in such as `**Note.**` stays attached to the sentence it introduces, and no split happens inside unbalanced bold markers, brackets or parentheses. @@ -63,3 +65,5 @@ Pass options through pre-commit with `args`: - id: md-lines args: ["--split-lists", "--abbreviations", "chap.,sect."] ``` + +Development notes, including how the data model is generated from its LinkML schema, are in [docs/development.md](docs/development.md). diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..b18bb80 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,33 @@ +# Development + +```sh +pip install -e ".[dev]" +pytest +pre-commit run --all-files +``` + +The `dev` extra brings pytest, pre-commit and [LinkML](https://linkml.io). +The hook is used on this repository's own Markdown through `.pre-commit-config.yaml`. + +## Layout + +- `src/md_lines/schema/md_lines.yaml` defines the data model (`Options`, `Problem`, `Result`, `Paragraph`) as a LinkML schema. +- `src/md_lines/models.py` is generated from the schema and is not edited by hand; the formatters skip it, and a test fails when it and the schema drift apart. +- `src/md_lines/globals.py` holds the module-level constants: the abbreviation list, the parser, the regular expressions and `DEFAULT_OPTIONS`. +- `src/md_lines/reflow.py` holds the reflow functions only, and `src/md_lines/cli.py` the command. +- `__version__` is read from the installed package metadata, so `pyproject.toml` is the only place the version is written. + +After changing the schema, regenerate the model: + +```sh +gen-python --no-metadata src/md_lines/schema/md_lines.yaml > src/md_lines/models.py +``` + +## Repository checks + +The pre-commit configuration follows [historia](https://github.com/CodyCBakerPhD/historia): pre-commit-hooks, check-github-workflows, black, ruff (lint plus isort rules, with `--fix`), codespell with the `en-GB_to_en-US` dictionary so the text stays in American English, and this repository's own `md-lines` hook. +CI runs the tests and the hooks on the newest Python release. + +## Exit status + +`md-lines` exits 0 when nothing needed changing, 1 when files were (or in `--check` mode would be) rewritten, and 2 when a file could not be read. diff --git a/pyproject.toml b/pyproject.toml index d2e2b4b..7c9f935 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -8,7 +8,7 @@ version = "0.1.0" description = "pre-commit hook that puts each Markdown sentence on its own line" readme = "README.md" requires-python = ">=3.13" -dependencies = ["markdown-it-py>=3.0"] +dependencies = ["markdown-it-py>=3.0", "linkml-runtime>=1.9"] keywords = ["markdown", "pre-commit", "semantic line breaks", "sentence per line"] classifiers = [ "Programming Language :: Python :: 3", @@ -19,6 +19,8 @@ classifiers = [ [project.optional-dependencies] test = ["pytest>=7"] +# Regenerating src/md_lines/models.py from the LinkML schema, and the test that checks it. +dev = ["pytest>=7", "linkml>=1.9", "pre-commit"] [project.scripts] md-lines = "md_lines.cli:main" @@ -32,9 +34,25 @@ packages = ["src/md_lines"] [tool.pytest.ini_options] testpaths = ["tests"] +[tool.black] +line-length = 100 +target-version = ["py313"] +# Generated from src/md_lines/schema/md_lines.yaml; regenerate instead of editing. +extend-exclude = "src/md_lines/models\\.py" + [tool.ruff] line-length = 100 target-version = "py313" +# Generated from src/md_lines/schema/md_lines.yaml; regenerate instead of editing. +extend-exclude = ["src/md_lines/models.py"] [tool.ruff.lint] select = ["E", "F", "I", "UP", "B"] + +[tool.ruff.lint.isort] +known-first-party = ["md_lines"] + +[tool.codespell] +skip = ".git,*.lock" +# Also flag British spellings: the project is written in American English. +builtin = "clear,rare,en-GB_to_en-US" diff --git a/src/md_lines/__init__.py b/src/md_lines/__init__.py index 54d305f..d2ac112 100644 --- a/src/md_lines/__init__.py +++ b/src/md_lines/__init__.py @@ -2,23 +2,20 @@ from importlib.metadata import PackageNotFoundError, version -from md_lines.reflow import ( - DEFAULT_ABBREVIATIONS, - DEFAULT_OPTIONS, - Options, - Problem, - Result, - reflow, -) +from md_lines.globals import DEFAULT_ABBREVIATIONS, DEFAULT_OPTIONS +from md_lines.models import Options, Paragraph, Problem, Result +from md_lines.reflow import reflow, split_sentences __all__ = [ "DEFAULT_ABBREVIATIONS", "DEFAULT_OPTIONS", "Options", + "Paragraph", "Problem", "Result", "__version__", "reflow", + "split_sentences", ] try: diff --git a/src/md_lines/cli.py b/src/md_lines/cli.py index e26debc..4e7288c 100644 --- a/src/md_lines/cli.py +++ b/src/md_lines/cli.py @@ -6,7 +6,9 @@ from collections.abc import Sequence from md_lines import __version__ -from md_lines.reflow import DEFAULT_ABBREVIATIONS, Options, reflow +from md_lines.globals import DEFAULT_ABBREVIATIONS +from md_lines.models import Options +from md_lines.reflow import reflow def _build_parser() -> argparse.ArgumentParser: @@ -32,7 +34,7 @@ def _build_parser() -> argparse.ArgumentParser: ) parser.add_argument( "--keep-breaks-after", - default=Options.keep_breaks_after, + default=".!?", metavar="CHARS", help="keep an existing line break after any of these characters (default: %(default)r)", ) @@ -74,7 +76,7 @@ def _options_from(args: argparse.Namespace, /) -> Options: return Options( split_lists=args.split_lists, keep_breaks_after=args.keep_breaks_after, - abbreviations=frozenset(abbreviations), + abbreviations=sorted(abbreviations), ) @@ -107,7 +109,7 @@ def main(argv: Sequence[str] | None = None) -> int: changed += 1 if not args.quiet: for problem in result.problems: - print(f"{path}:{problem}") + print(f"{path}:{problem.line}: {problem.message}") if args.diff: sys.stdout.writelines( difflib.unified_diff( diff --git a/src/md_lines/globals.py b/src/md_lines/globals.py new file mode 100644 index 0000000..8e88bd1 --- /dev/null +++ b/src/md_lines/globals.py @@ -0,0 +1,121 @@ +"""Module-level constants: the abbreviation list, the parser and the regular expressions.""" + +import re + +from markdown_it import MarkdownIt + +from md_lines.models import Options + +DEFAULT_ABBREVIATIONS: frozenset[str] = frozenset( + { + # Latin + "e.g.", + "i.e.", + "etc.", + "al.", + "cf.", + "viz.", + "vs.", + "v.", + "ca.", + "c.", + "approx.", + "resp.", + "ibid.", + "op.", + "n.b.", + "seq.", + # Cross references + "fig.", + "figs.", + "tab.", + "tabs.", + "eq.", + "eqs.", + "ref.", + "refs.", + "sec.", + "secs.", + "ch.", + "app.", + "suppl.", + "vol.", + "vols.", + "no.", + "nos.", + "p.", + "pp.", + "ed.", + "eds.", + # Titles + "dr.", + "mr.", + "mrs.", + "ms.", + "prof.", + "st.", + "jr.", + "sr.", + # Other + "inc.", + "ltd.", + "co.", + "corp.", + "dept.", + "univ.", + "min.", + "max.", + "avg.", + "est.", + "misc.", + "a.k.a.", + "jan.", + "feb.", + "mar.", + "apr.", + "jun.", + "jul.", + "aug.", + "sep.", + "sept.", + "oct.", + "nov.", + "dec.", + } +) + +# Closing characters that may follow the punctuation that ends a sentence or a line. +TRAILING_CLOSERS = "\"')]*_`\u2019\u201d" +# Characters that may begin a new sentence after the punctuation and whitespace. +OPENERS = "\"'([*_`\u201c\u2018" + +#: Options used when none are given: the built-in abbreviation list and the defaults +#: declared in the schema. A bare ``Options()`` has no abbreviations at all. +DEFAULT_OPTIONS = Options(abbreviations=sorted(DEFAULT_ABBREVIATIONS)) + +PARSER = MarkdownIt("commonmark").enable("table").enable("strikethrough") + +# First line of a paragraph that starts a list item: any mix of block quote markers and list +# markers, each followed by whitespace. +ITEM_PREFIX_RE = re.compile(r"^(?:[ \t]*(?:>|(?:[-*+]|\d{1,9}[.)])(?=[ \t])))*[ \t]*") +# Any other paragraph line: indentation and block quote markers only. +LINE_PREFIX_RE = re.compile(r"^[ \t]*(?:>[ \t]*)*") + +# Candidate sentence boundary: terminal punctuation, optional closers, whitespace. +BOUNDARY_RE = re.compile(r"[.!?]+[\"')\]*_`\u2019\u201d]*(\s+)") +# A bold lead-in such as "**Figure 3.**" or "__Note:__". It must not itself contain a sentence +# boundary, otherwise it is a bold sentence and may be split after. +BOLD_LABEL_RE = re.compile(r"(\*\*|__)(?:(?![.!?]\s)[^*_])+(\*\*|__)[.!?]?[\"')\]\u2019\u201d]*") +DOTTED_ABBREVIATION_RE = re.compile(r"(?:[^\W\d_]\.){2,}") +INITIAL_RE = re.compile(r"[^\W\d_]\.") +ENUMERATOR_RE = re.compile(r"\W*\d+[.)]") + +# Inline constructs whose punctuation never ends a sentence. Each is masked with characters +# of the same length so that positions in the masked text match the original. +CODE_SPAN_RE = re.compile(r"(?") +ANGLE_RE = re.compile(r"<[^<>\s][^<>]*>") +LINK_DESTINATION_RE = re.compile(r"\]\((?:[^()\s]|\([^()\s]*\))*\)") +REFERENCE_LABEL_RE = re.compile(r"\]\[[^\]]*\]") +URL_RE = re.compile(r"(?:https?|ftp)://[^\s<>\"']+?(?=[.,;:!?)\]]*(?:\s|$))") +MATH_RE = re.compile(r"\$[^$\s][^$]*\$") diff --git a/src/md_lines/models.py b/src/md_lines/models.py new file mode 100644 index 0000000..6e06ce1 --- /dev/null +++ b/src/md_lines/models.py @@ -0,0 +1,239 @@ +# id: https://github.com/CodyCBakerPhD/md-lines-pre-commit/schema/md_lines +# description: Options for, and results of, reflowing Markdown so that each sentence sits on its own line. The Python dataclasses in md_lines/models.py are generated from this schema with `gen-python --no-metadata`; do not edit them by hand. +# license: https://creativecommons.org/publicdomain/zero/1.0/ + +import dataclasses +import re +from dataclasses import dataclass +from datetime import ( + date, + datetime, + time +) +from typing import ( + Any, + ClassVar, + Dict, + List, + Optional, + Union +) + +from jsonasobj2 import ( + JsonObj, + as_dict +) +from linkml_runtime.linkml_model.meta import ( + EnumDefinition, + PermissibleValue, + PvFormulaOptions +) +from linkml_runtime.utils.curienamespace import CurieNamespace +from linkml_runtime.utils.enumerations import EnumDefinitionImpl +from linkml_runtime.utils.formatutils import ( + camelcase, + sfx, + underscore +) +from linkml_runtime.utils.metamodelcore import ( + bnode, + empty_dict, + empty_list +) +from linkml_runtime.utils.slot import Slot +from linkml_runtime.utils.yamlutils import ( + YAMLRoot, + extended_float, + extended_int, + extended_str +) +from rdflib import ( + Namespace, + URIRef +) + +from linkml_runtime.linkml_model.types import Boolean, Integer, String +from linkml_runtime.utils.metamodelcore import Bool + +metamodel_version = "1.11.0" +version = None + +# Namespaces +LINKML = CurieNamespace('linkml', 'https://w3id.org/linkml/') +MD_LINES = CurieNamespace('md_lines', 'https://github.com/CodyCBakerPhD/md-lines-pre-commit/schema/md_lines/') +DEFAULT_ = MD_LINES + + +# Types + +# Class references + + + +@dataclass(repr=False) +class Options(YAMLRoot): + """ + What counts as a line and a sentence. + """ + _inherited_slots: ClassVar[list[str]] = [] + + class_class_uri: ClassVar[URIRef] = MD_LINES["Options"] + class_class_curie: ClassVar[str] = "md_lines:Options" + class_name: ClassVar[str] = "Options" + class_model_uri: ClassVar[URIRef] = MD_LINES.Options + + split_lists: Optional[Union[bool, Bool]] = False + keep_breaks_after: Optional[str] = ".!?" + abbreviations: Optional[Union[str, list[str]]] = empty_list() + + def __post_init__(self, *_: str, **kwargs: Any): + if self.split_lists is not None and not isinstance(self.split_lists, Bool): + self.split_lists = Bool(self.split_lists) + + if self.keep_breaks_after is not None and not isinstance(self.keep_breaks_after, str): + self.keep_breaks_after = str(self.keep_breaks_after) + + if not isinstance(self.abbreviations, list): + self.abbreviations = [self.abbreviations] if self.abbreviations is not None else [] + self.abbreviations = [v if isinstance(v, str) else str(v) for v in self.abbreviations] + + super().__post_init__(**kwargs) + + +@dataclass(repr=False) +class Problem(YAMLRoot): + """ + One thing that was wrong with a file. + """ + _inherited_slots: ClassVar[list[str]] = [] + + class_class_uri: ClassVar[URIRef] = MD_LINES["Problem"] + class_class_curie: ClassVar[str] = "md_lines:Problem" + class_name: ClassVar[str] = "Problem" + class_model_uri: ClassVar[URIRef] = MD_LINES.Problem + + line: int = None + message: str = None + + def __post_init__(self, *_: str, **kwargs: Any): + if self._is_empty(self.line): + self.MissingRequiredField("line") + if not isinstance(self.line, int): + self.line = int(self.line) + + if self._is_empty(self.message): + self.MissingRequiredField("message") + if not isinstance(self.message, str): + self.message = str(self.message) + + super().__post_init__(**kwargs) + + +@dataclass(repr=False) +class Result(YAMLRoot): + """ + The reflowed text and the problems that were found. + """ + _inherited_slots: ClassVar[list[str]] = [] + + class_class_uri: ClassVar[URIRef] = MD_LINES["Result"] + class_class_curie: ClassVar[str] = "md_lines:Result" + class_name: ClassVar[str] = "Result" + class_model_uri: ClassVar[URIRef] = MD_LINES.Result + + text: str = None + problems: Optional[Union[Union[dict, Problem], list[Union[dict, Problem]]]] = empty_list() + + def __post_init__(self, *_: str, **kwargs: Any): + if self._is_empty(self.text): + self.MissingRequiredField("text") + if not isinstance(self.text, str): + self.text = str(self.text) + + self._normalize_inlined_as_list(slot_name="problems", slot_type=Problem, key_name="line", keyed=False) + + super().__post_init__(**kwargs) + + +@dataclass(repr=False) +class Paragraph(YAMLRoot): + """ + A paragraph's position in the source, as line indices from the parser; the only block type whose lines are + reflowed. + """ + _inherited_slots: ClassVar[list[str]] = [] + + class_class_uri: ClassVar[URIRef] = MD_LINES["Paragraph"] + class_class_curie: ClassVar[str] = "md_lines:Paragraph" + class_name: ClassVar[str] = "Paragraph" + class_model_uri: ClassVar[URIRef] = MD_LINES.Paragraph + + start: int = None + end: int = None + in_list: Union[bool, Bool] = None + starts_item: Union[bool, Bool] = None + + def __post_init__(self, *_: str, **kwargs: Any): + if self._is_empty(self.start): + self.MissingRequiredField("start") + if not isinstance(self.start, int): + self.start = int(self.start) + + if self._is_empty(self.end): + self.MissingRequiredField("end") + if not isinstance(self.end, int): + self.end = int(self.end) + + if self._is_empty(self.in_list): + self.MissingRequiredField("in_list") + if not isinstance(self.in_list, Bool): + self.in_list = Bool(self.in_list) + + if self._is_empty(self.starts_item): + self.MissingRequiredField("starts_item") + if not isinstance(self.starts_item, Bool): + self.starts_item = Bool(self.starts_item) + + super().__post_init__(**kwargs) + + +# Enumerations + + +# Slots +class slots: + pass + +slots.options__split_lists = Slot(uri=MD_LINES.split_lists, name="options__split_lists", curie=MD_LINES.curie('split_lists'), + model_uri=MD_LINES.options__split_lists, domain=None, range=Optional[Union[bool, Bool]]) + +slots.options__keep_breaks_after = Slot(uri=MD_LINES.keep_breaks_after, name="options__keep_breaks_after", curie=MD_LINES.curie('keep_breaks_after'), + model_uri=MD_LINES.options__keep_breaks_after, domain=None, range=Optional[str]) + +slots.options__abbreviations = Slot(uri=MD_LINES.abbreviations, name="options__abbreviations", curie=MD_LINES.curie('abbreviations'), + model_uri=MD_LINES.options__abbreviations, domain=None, range=Optional[Union[str, list[str]]]) + +slots.problem__line = Slot(uri=MD_LINES.line, name="problem__line", curie=MD_LINES.curie('line'), + model_uri=MD_LINES.problem__line, domain=None, range=int) + +slots.problem__message = Slot(uri=MD_LINES.message, name="problem__message", curie=MD_LINES.curie('message'), + model_uri=MD_LINES.problem__message, domain=None, range=str) + +slots.result__text = Slot(uri=MD_LINES.text, name="result__text", curie=MD_LINES.curie('text'), + model_uri=MD_LINES.result__text, domain=None, range=str) + +slots.result__problems = Slot(uri=MD_LINES.problems, name="result__problems", curie=MD_LINES.curie('problems'), + model_uri=MD_LINES.result__problems, domain=None, range=Optional[Union[Union[dict, Problem], list[Union[dict, Problem]]]]) + +slots.paragraph__start = Slot(uri=MD_LINES.start, name="paragraph__start", curie=MD_LINES.curie('start'), + model_uri=MD_LINES.paragraph__start, domain=None, range=int) + +slots.paragraph__end = Slot(uri=MD_LINES.end, name="paragraph__end", curie=MD_LINES.curie('end'), + model_uri=MD_LINES.paragraph__end, domain=None, range=int) + +slots.paragraph__in_list = Slot(uri=MD_LINES.in_list, name="paragraph__in_list", curie=MD_LINES.curie('in_list'), + model_uri=MD_LINES.paragraph__in_list, domain=None, range=Union[bool, Bool]) + +slots.paragraph__starts_item = Slot(uri=MD_LINES.starts_item, name="paragraph__starts_item", curie=MD_LINES.curie('starts_item'), + model_uri=MD_LINES.paragraph__starts_item, domain=None, range=Union[bool, Bool]) + diff --git a/src/md_lines/reflow.py b/src/md_lines/reflow.py index c8f9c0f..777057c 100644 --- a/src/md_lines/reflow.py +++ b/src/md_lines/reflow.py @@ -1,161 +1,18 @@ """Reflow Markdown prose so that each sentence sits on its own line. -Block structure (paragraphs, list items, block quotes, headings, tables, code, -HTML, link reference definitions) comes from a CommonMark parser, so only the -lines of real paragraphs are ever touched. Inside a paragraph, wrapped sentences -are joined back onto one line and lines holding several sentences are split. +Block structure (paragraphs, list items, block quotes, headings, tables, code, HTML, link +reference definitions) comes from a CommonMark parser, so only the lines of real paragraphs +are ever touched. Inside a paragraph, wrapped sentences are joined back onto one line and +lines holding several sentences are split. + +The data model (Options, Problem, Result, Paragraph) is generated from the LinkML schema in +``schema/md_lines.yaml``; the constants live in ``globals.py``. """ -import dataclasses import re -from markdown_it import MarkdownIt - -DEFAULT_ABBREVIATIONS: frozenset[str] = frozenset( - { - # Latin - "e.g.", - "i.e.", - "etc.", - "al.", - "cf.", - "viz.", - "vs.", - "v.", - "ca.", - "c.", - "approx.", - "resp.", - "ibid.", - "op.", - "n.b.", - "seq.", - # Cross references - "fig.", - "figs.", - "tab.", - "tabs.", - "eq.", - "eqs.", - "ref.", - "refs.", - "sec.", - "secs.", - "ch.", - "app.", - "suppl.", - "vol.", - "vols.", - "no.", - "nos.", - "p.", - "pp.", - "ed.", - "eds.", - # Titles - "dr.", - "mr.", - "mrs.", - "ms.", - "prof.", - "st.", - "jr.", - "sr.", - # Other - "inc.", - "ltd.", - "co.", - "corp.", - "dept.", - "univ.", - "min.", - "max.", - "avg.", - "est.", - "misc.", - "a.k.a.", - "jan.", - "feb.", - "mar.", - "apr.", - "jun.", - "jul.", - "aug.", - "sep.", - "sept.", - "oct.", - "nov.", - "dec.", - } -) - -# Closing characters that may follow the punctuation that ends a sentence or a line. -TRAILING_CLOSERS = "\"')]*_`’”" -# Characters that may begin a new sentence after the punctuation and whitespace. -_OPENERS = "\"'([*_`“‘" - - -@dataclasses.dataclass(frozen=True) -class Options: - """What counts as a line and a sentence.""" - - #: Split lines inside list items as well. By default a list item keeps all of - #: its sentences on one line and only wrapped lines are joined. - split_lists: bool = False - #: An existing line break after one of these characters is kept, so the next - #: line is not joined onto it. Sentences are only ever split after ``.``, ``!`` - #: and ``?``; add ``:`` and ``;`` here to also allow line breaks after them. - keep_breaks_after: str = ".!?" - #: Words ending in a period that do not end a sentence, lowercase. - abbreviations: frozenset[str] = DEFAULT_ABBREVIATIONS - - -DEFAULT_OPTIONS = Options() - - -@dataclasses.dataclass(frozen=True) -class Problem: - """One thing that was wrong with a file; ``line`` is 1-based.""" - - line: int - message: str - - def __str__(self) -> str: - return f"{self.line}: {self.message}" - - -@dataclasses.dataclass(frozen=True) -class Result: - text: str - problems: tuple[Problem, ...] - - -_PARSER = MarkdownIt("commonmark").enable("table").enable("strikethrough") - -# First line of a paragraph that starts a list item: any mix of block quote -# markers and list markers, each followed by whitespace. -_ITEM_PREFIX_RE = re.compile(r"^(?:[ \t]*(?:>|(?:[-*+]|\d{1,9}[.)])(?=[ \t])))*[ \t]*") -# Any other paragraph line: indentation and block quote markers only. -_LINE_PREFIX_RE = re.compile(r"^[ \t]*(?:>[ \t]*)*") - -_BOUNDARY_RE = re.compile(r"[.!?]+[\"')\]*_`’”]*(\s+)") -# A bold lead-in such as "**Figure 3.**" or "__Note:__". It must not itself contain a -# sentence boundary, otherwise it is a bold sentence and may be split after. -_BOLD_LABEL_RE = re.compile(r"(\*\*|__)(?:(?![.!?]\s)[^*_])+(\*\*|__)[.!?]?[\"')\]\u2019\u201d]*") -_DOTTED_ABBREVIATION_RE = re.compile(r"(?:[^\W\d_]\.){2,}") -_INITIAL_RE = re.compile(r"[^\W\d_]\.") -_ENUMERATOR_RE = re.compile(r"\W*\d+[.)]") - -# Inline constructs whose punctuation never ends a sentence. Each is masked with -# characters of the same length so that positions in the masked text match the -# original. -_CODE_SPAN_RE = re.compile(r"(?") -_ANGLE_RE = re.compile(r"<[^<>\s][^<>]*>") -_LINK_DESTINATION_RE = re.compile(r"\]\((?:[^()\s]|\([^()\s]*\))*\)") -_REFERENCE_LABEL_RE = re.compile(r"\]\[[^\]]*\]") -_URL_RE = re.compile(r"(?:https?|ftp)://[^\s<>\"']+?(?=[.,;:!?)\]]*(?:\s|$))") -_MATH_RE = re.compile(r"\$[^$\s][^$]*\$") +from md_lines import globals as g +from md_lines.models import Options, Paragraph, Problem, Result def _mask_inline(text: str, /) -> str: @@ -166,41 +23,42 @@ def keep_edges(match: re.Match[str], edge: int) -> str: inner = len(whole) - 2 * edge return whole[:edge] + "x" * inner + whole[len(whole) - edge :] - text = _CODE_SPAN_RE.sub(lambda m: m.group(1) + "x" * len(m.group(2)) + m.group(1), text) - text = _HTML_COMMENT_RE.sub(lambda m: "x" * len(m.group()), text) - text = _ANGLE_RE.sub(lambda m: "x" * len(m.group()), text) - text = _LINK_DESTINATION_RE.sub(lambda m: keep_edges(m, 2), text) - text = _REFERENCE_LABEL_RE.sub(lambda m: keep_edges(m, 2), text) - text = _URL_RE.sub(lambda m: "x" * len(m.group()), text) - text = _MATH_RE.sub(lambda m: "x" * len(m.group()), text) + text = g.CODE_SPAN_RE.sub(lambda m: m.group(1) + "x" * len(m.group(2)) + m.group(1), text) + text = g.HTML_COMMENT_RE.sub(lambda m: "x" * len(m.group()), text) + text = g.ANGLE_RE.sub(lambda m: "x" * len(m.group()), text) + text = g.LINK_DESTINATION_RE.sub(lambda m: keep_edges(m, 2), text) + text = g.REFERENCE_LABEL_RE.sub(lambda m: keep_edges(m, 2), text) + text = g.URL_RE.sub(lambda m: "x" * len(m.group()), text) + text = g.MATH_RE.sub(lambda m: "x" * len(m.group()), text) return text -def split_sentences(text: str, /, *, options: Options = DEFAULT_OPTIONS) -> list[str]: +def split_sentences(text: str, /, *, options: Options = g.DEFAULT_OPTIONS) -> list[str]: """Split one line of Markdown prose into its sentences.""" + abbreviations = set(options.abbreviations or ()) masked = _mask_inline(text) sentences: list[str] = [] start = 0 - for match in _BOUNDARY_RE.finditer(masked): + for match in g.BOUNDARY_RE.finditer(masked): if match.end() >= len(masked): break following = masked[match.end()] - if not (following.isupper() or following.isdigit() or following in _OPENERS): + if not (following.isupper() or following.isdigit() or following in g.OPENERS): continue chunk = masked[start : match.start(1)] if chunk.count("**") % 2 or chunk.count("__") % 2: continue if chunk.count("[") != chunk.count("]") or chunk.count("(") != chunk.count(")"): continue - if _BOLD_LABEL_RE.fullmatch(chunk.strip()): + if g.BOLD_LABEL_RE.fullmatch(chunk.strip()): continue words = chunk.split() - last = words[-1].lstrip("(\"'[*_“‘").rstrip(TRAILING_CLOSERS) if words else "" - if len(words) == 1 and _ENUMERATOR_RE.fullmatch(last): + last = words[-1].lstrip("(\"'[*_“‘").rstrip(g.TRAILING_CLOSERS) if words else "" + if len(words) == 1 and g.ENUMERATOR_RE.fullmatch(last): continue - if last.lower() in options.abbreviations: + if last.lower() in abbreviations: continue - if _DOTTED_ABBREVIATION_RE.fullmatch(last) or _INITIAL_RE.fullmatch(last): + if g.DOTTED_ABBREVIATION_RE.fullmatch(last) or g.INITIAL_RE.fullmatch(last): continue sentences.append(text[start : match.start(1)]) start = match.end(1) @@ -225,18 +83,10 @@ def _front_matter_length(lines: list[str], /) -> int: return 0 -@dataclasses.dataclass(frozen=True) -class _Paragraph: - start: int - end: int - in_list: bool - starts_item: bool - - -def _paragraphs(lines: list[str], /) -> list[_Paragraph]: +def _paragraphs(lines: list[str], /) -> list[Paragraph]: skip = _front_matter_length(lines) source = "\n".join([""] * skip + lines[skip:]) + "\n" - tokens = _PARSER.parse(source) + tokens = g.PARSER.parse(source) item_starts = {token.map[0] for token in tokens if token.type == "list_item_open" and token.map} paragraphs = [] list_depth = 0 @@ -247,12 +97,16 @@ def _paragraphs(lines: list[str], /) -> list[_Paragraph]: list_depth -= 1 elif token.type == "paragraph_open" and token.map: start, end = token.map - paragraphs.append(_Paragraph(start, end, list_depth > 0, start in item_starts)) + paragraphs.append( + Paragraph( + start=start, end=end, in_list=list_depth > 0, starts_item=start in item_starts + ) + ) return paragraphs def _split_prefix(line: str, /, *, item_start: bool) -> tuple[str, str]: - pattern = _ITEM_PREFIX_RE if item_start else _LINE_PREFIX_RE + pattern = g.ITEM_PREFIX_RE if item_start else g.LINE_PREFIX_RE match = pattern.match(line) assert match is not None return line[: match.end()], line[match.end() :] @@ -278,12 +132,12 @@ def _continues_sentence(content: str, /, *, options: Options) -> bool: """Whether the following line continues the sentence that ends on ``content``.""" if _ends_with_hard_break(content): return False - stripped = content.rstrip().rstrip(TRAILING_CLOSERS) - return not stripped.endswith(tuple(options.keep_breaks_after)) + stripped = content.rstrip().rstrip(g.TRAILING_CLOSERS) + return not stripped.endswith(tuple(options.keep_breaks_after or "")) def _reflow_paragraph( - lines: list[str], paragraph: _Paragraph, /, *, options: Options, problems: list[Problem] + lines: list[str], paragraph: Paragraph, /, *, options: Options, problems: list[Problem] ) -> list[str]: raw = lines[paragraph.start : paragraph.end] first_prefix, first_content = _split_prefix(raw[0], item_start=paragraph.starts_item) @@ -296,8 +150,8 @@ def _reflow_paragraph( first_offset, previous = joined[-1] problems.append( Problem( - paragraph.start + first_offset + 1, - f"sentence is wrapped onto line {paragraph.start + offset + 1}", + line=paragraph.start + first_offset + 1, + message=f"sentence is wrapped onto line {paragraph.start + offset + 1}", ) ) joined[-1] = (first_offset, previous.rstrip() + " " + content.strip()) @@ -314,7 +168,10 @@ def _reflow_paragraph( sentences = split_sentences(body, options=options) if len(sentences) > 1: problems.append( - Problem(paragraph.start + offset + 1, f"line holds {len(sentences)} sentences") + Problem( + line=paragraph.start + offset + 1, + message=f"line holds {len(sentences)} sentences", + ) ) sentences[-1] += trailing out.extend(sentences) @@ -322,7 +179,7 @@ def _reflow_paragraph( return [first_prefix + out[0]] + [continuation_prefix + sentence for sentence in out[1:]] -def reflow(text: str, /, *, options: Options = DEFAULT_OPTIONS) -> Result: +def reflow(text: str, /, *, options: Options = g.DEFAULT_OPTIONS) -> Result: """Return the reflowed text and the problems that were found.""" bom = "" if text.startswith("") else "" body = text[len(bom) :] @@ -340,4 +197,4 @@ def reflow(text: str, /, *, options: Options = DEFAULT_OPTIONS) -> Result: problems.sort(key=lambda problem: problem.line) new_text = newline.join(fixed) + (newline if has_final_newline else "") - return Result(text=bom + new_text, problems=tuple(problems)) + return Result(text=bom + new_text, problems=problems) diff --git a/src/md_lines/schema/md_lines.yaml b/src/md_lines/schema/md_lines.yaml new file mode 100644 index 0000000..fb655d9 --- /dev/null +++ b/src/md_lines/schema/md_lines.yaml @@ -0,0 +1,85 @@ +id: https://github.com/CodyCBakerPhD/md-lines-pre-commit/schema/md_lines +name: md_lines +title: md-lines data model +description: >- + Options for, and results of, reflowing Markdown so that each sentence sits on its own line. + The Python dataclasses in md_lines/models.py are generated from this schema with + `gen-python --no-metadata`; do not edit them by hand. +prefixes: + linkml: https://w3id.org/linkml/ + md_lines: https://github.com/CodyCBakerPhD/md-lines-pre-commit/schema/md_lines/ +default_prefix: md_lines +default_range: string +imports: + - linkml:types + +classes: + Options: + description: What counts as a line and a sentence. + attributes: + split_lists: + description: >- + Split lines inside list items as well. By default a list item keeps all of its + sentences on one line and only wrapped lines are joined. + range: boolean + ifabsent: "false" + keep_breaks_after: + description: >- + An existing line break after one of these characters is kept, so the next line is + not joined onto it. Sentences are only ever split after `.`, `!` and `?`; add `:` and + `;` here to also allow line breaks after them. + range: string + ifabsent: string(.!?) + abbreviations: + description: >- + Words ending in a period that do not end a sentence, lowercase. An empty list means + no abbreviations; DEFAULT_OPTIONS carries the built-in list. + range: string + multivalued: true + + Problem: + description: One thing that was wrong with a file. + attributes: + line: + description: 1-based line number in the original file. + range: integer + required: true + message: + description: What was wrong on that line. + range: string + required: true + + Result: + description: The reflowed text and the problems that were found. + attributes: + text: + description: The reflowed file content. + range: string + required: true + problems: + description: The problems found, in line order. + range: Problem + multivalued: true + inlined_as_list: true + + Paragraph: + description: >- + A paragraph's position in the source, as line indices from the parser; the only block + type whose lines are reflowed. + attributes: + start: + description: 0-based index of the first line. + range: integer + required: true + end: + description: 0-based index one past the last line. + range: integer + required: true + in_list: + description: Whether the paragraph sits inside a list item. + range: boolean + required: true + starts_item: + description: Whether the paragraph's first line also starts a list item. + range: boolean + required: true diff --git a/tests/test_cli.py b/tests/test_cli.py index 928955f..8edb455 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -1,5 +1,4 @@ import subprocess -import sys from md_lines.cli import main @@ -62,14 +61,9 @@ def test_missing_file_exits_two(tmp_path, capsys): assert "missing.md" in capsys.readouterr().err -def test_module_and_console_entry_points(tmp_path): - path = write(tmp_path, "a.md", "One.\n") - assert ( - subprocess.run([sys.executable, "-m", "md_lines", str(path)], check=False).returncode == 0 - ) - assert subprocess.run( - ["md-lines", "--version"], capture_output=True, text=True - ).stdout.startswith("md-lines ") +def test_console_entry_point(): + version = subprocess.run(["md-lines", "--version"], capture_output=True, text=True) + assert version.stdout.startswith("md-lines ") def test_version_matches_pyproject(): diff --git a/tests/test_reflow.py b/tests/test_reflow.py index 4ee9588..3859e8a 100644 --- a/tests/test_reflow.py +++ b/tests/test_reflow.py @@ -1,11 +1,11 @@ import pytest -from md_lines import Options, reflow -from md_lines.reflow import split_sentences +from md_lines import DEFAULT_ABBREVIATIONS, Options, reflow, split_sentences def fix(text, **kwargs): - return reflow(text, options=Options(**kwargs)).text + settings = {"abbreviations": sorted(DEFAULT_ABBREVIATIONS), **kwargs} + return reflow(text, options=Options(**settings)).text def test_wrapped_sentence_is_joined(): @@ -36,9 +36,9 @@ def test_break_after_semicolon_is_joined_by_default_and_kept_when_asked(): def test_problems_report_original_line_numbers(): result = reflow("Intro\n\nOne\nwrapped. Two.\n") - assert [str(p) for p in result.problems] == [ - "3: sentence is wrapped onto line 4", - "3: line holds 2 sentences", + assert [(p.line, p.message) for p in result.problems] == [ + (3, "sentence is wrapped onto line 4"), + (3, "line holds 2 sentences"), ] @@ -139,7 +139,7 @@ def test_abbreviations_initials_and_numbers_do_not_split(): def test_custom_abbreviations(): text = "See Chap. Three. Done.\n" assert fix(text) == "See Chap.\nThree.\nDone.\n" - assert fix(text, abbreviations=frozenset({"chap."})) == "See Chap. Three.\nDone.\n" + assert fix(text, abbreviations=["chap."]) == "See Chap. Three.\nDone.\n" def test_bold_label_and_unbalanced_markup_do_not_split(): @@ -181,4 +181,4 @@ def test_lazy_continuation_belongs_to_the_list_item(): def test_dotted_abbreviations_are_detected_without_the_list(): - assert fix("Use e.g. Python.\n", abbreviations=frozenset()) == "Use e.g. Python.\n" + assert fix("Use e.g. Python.\n", abbreviations=[]) == "Use e.g. Python.\n" diff --git a/tests/test_schema.py b/tests/test_schema.py new file mode 100644 index 0000000..b967299 --- /dev/null +++ b/tests/test_schema.py @@ -0,0 +1,35 @@ +"""The LinkML schema is the source of the data model; models.py must match it.""" + +import pathlib + +import pytest + +ROOT = pathlib.Path(__file__).resolve().parents[1] +SCHEMA = ROOT / "src" / "md_lines" / "schema" / "md_lines.yaml" +MODELS = ROOT / "src" / "md_lines" / "models.py" + + +def test_schema_declares_the_model_classes(): + schemaview = pytest.importorskip("linkml_runtime.utils.schemaview") + view = schemaview.SchemaView(str(SCHEMA)) + assert {"Options", "Problem", "Result", "Paragraph"} <= set(view.all_classes()) + assert view.induced_slot("line", "Problem").required + assert view.induced_slot("abbreviations", "Options").multivalued + + +def test_models_are_generated_from_the_schema(): + pythongen = pytest.importorskip("linkml.generators.pythongen") + generated = pythongen.PythonGenerator(str(SCHEMA), metadata=False).serialize() + assert generated.rstrip("\n") == MODELS.read_text().rstrip("\n"), ( + "src/md_lines/models.py is out of date; run " + "`gen-python --no-metadata src/md_lines/schema/md_lines.yaml > src/md_lines/models.py`" + ) + + +def test_generated_defaults_match_the_schema(): + from md_lines import Options + + options = Options() + assert options.split_lists is False + assert options.keep_breaks_after == ".!?" + assert options.abbreviations == [] From b5ef454a2b8d433e86ed3b1a12a15066cbec3ed7 Mon Sep 17 00:00:00 2001 From: Cody Baker Date: Thu, 8 Oct 2026 00:58:44 +0000 Subject: [PATCH 07/10] Adopt the historia conventions: AGENTS.md, relative imports, CI rename Review feedback on #1. - AGENTS.md ported from CodyCBakerPhD/historia, with sections for this repository's data model (LinkML schema, generated models.py, globals) and the American English and sentence-per-line rules; CHANGELOG.md started with an Upcoming entry for this PR - Internal imports are relative; external modules are imported whole (markdown_it, collections.abc, importlib.metadata); multi-input functions take keyword-only arguments, single-input ones positional-only - __init__.py holds imports and __all__ only; the version lookup moves to _version.py - Tests carry the ai_generated marker, registered in pyproject.toml - .github/workflows/ci.yml renamed to continuous_integration.yml with name "Continuous Integration" - pre-commit.ci skips the md-lines hook, which is `language: system` and therefore cannot run there; the GitHub Actions workflow runs it Co-Authored-By: Claude Code 2.1.293 / Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01ToiYPiSdU8NM9h1UmLgsq8 --- .../{ci.yml => continuous_integration.yml} | 2 +- .pre-commit-config.yaml | 2 + AGENTS.md | 49 +++++++++++++++++++ CHANGELOG.md | 7 +++ docs/development.md | 1 + pyproject.toml | 3 ++ src/md_lines/__init__.py | 16 ++---- src/md_lines/_version.py | 11 +++++ src/md_lines/cli.py | 18 +++---- src/md_lines/globals.py | 6 +-- src/md_lines/reflow.py | 30 ++++++------ tests/test_cli.py | 4 ++ tests/test_reflow.py | 8 +-- tests/test_schema.py | 6 ++- 14 files changed, 118 insertions(+), 45 deletions(-) rename .github/workflows/{ci.yml => continuous_integration.yml} (97%) create mode 100644 AGENTS.md create mode 100644 CHANGELOG.md create mode 100644 src/md_lines/_version.py diff --git a/.github/workflows/ci.yml b/.github/workflows/continuous_integration.yml similarity index 97% rename from .github/workflows/ci.yml rename to .github/workflows/continuous_integration.yml index 2d99fc8..d11604e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/continuous_integration.yml @@ -1,4 +1,4 @@ -name: CI +name: Continuous Integration on: pull_request: diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index b4b7175..803d02e 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,6 +1,8 @@ # Based on the pre-commit setup of CodyCBakerPhD/historia. ci: autoupdate_schedule: quarterly + # pre-commit.ci cannot run `language: system` hooks; the GitHub Actions workflow runs this one. + skip: [md-lines] default_install_hook_types: - pre-commit diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..b5e4b2b --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,49 @@ +# Agent instructions + +## Commits and PRs + +- Always run `pre-commit run --all-files` before committing and pushing changes +- Always link PRs to issues when possible +- PR titles should be human-readable and in the past tense. They should NOT use conventional commit style. +- Every commit must include a `Co-Authored-By` trailer identifying your tool name and version and your underlying model and version. Format (replace all `<…>` placeholders with actual values): `Co-Authored-By: / ` + +## Versioning and changelog + +- Bump the version in `pyproject.toml` once per PR when any file under `src/` or `pyproject.toml` itself changes. That is the only place to edit it; `md_lines.__version__` reads it from the installed package metadata. Do NOT bump for changes that are purely CI/workflow, documentation, or configuration (e.g., GitHub Actions workflows, `AGENTS.md`, `README.md` badges). +- The changelog follows the version. If the PR does not bump `pyproject.toml`, it does not need a `CHANGELOG.md` entry either. A documentation-only change needs neither. +- Otherwise, add a short entry to the `## Upcoming` section of `CHANGELOG.md` under the appropriate subsection (`### 🚀 Enhancement`, `### 🐛 Bug Fix`, `### 📝 Documentation`, `### 🔩 Dependency Updates`, `### 🏠 Internal`). Include the PR link at the end of each entry in the format `([#N](https://github.com/CodyCBakerPhD/md-lines-pre-commit/pull/N))`. Create the `## Upcoming` section or subsection if it does not yet exist. +- Releases are git tags of the form `vX.Y.Z`, which is what consumers pin as `rev:` in their pre-commit configuration. + +## Code style + +- Require keyword-only arguments `(*, ...)` for multi-input functions. For any function with exactly one caller-supplied parameter (excluding `self` and `cls`), require positional-only usage with the `/` designator. +- Always add new imports at the top of the file. The only exception is when a local import is needed to avoid a circular dependency. +- For external dependencies, use the full module import style (e.g., `import xyz; xyz.abc`) rather than `from xyz import abc`. +- For internal imports, always use relative style (e.g., `from .foo import bar`). +- Prefer assigning return values to named locals before `return` when this improves readability and debugger breakpoint placement. +- Avoid excessive em-dashes, colons, and semicolons in written text such as documentation. Prefer breaking into separate, shorter sentences instead. +- Write documentation, comments and messages in American English. codespell enforces it. +- In Markdown, put each sentence on its own line. This repository's own hook enforces it. +- Favor defining one-word names for CLI flags, then map those onto longer, more explicit keyword arguments at the API level. +- Keep inline comments sparse. Only explain non-obvious "why", not "what" the code does. Prefer self-documenting code and clear names over narration; do not annotate routine logic. + +## Data model + +- The data model (`Options`, `Problem`, `Result`, `Paragraph`) is defined in the LinkML schema `src/md_lines/schema/md_lines.yaml`. Change the schema, never the generated code. +- `src/md_lines/models.py` is generated from the schema with `gen-python --no-metadata src/md_lines/schema/md_lines.yaml > src/md_lines/models.py`. Regenerate it after every schema change; a test fails when the two drift apart. +- Module-level constants (the abbreviation list, the parser, the regular expressions, `DEFAULT_OPTIONS`) live in `src/md_lines/globals.py`, not next to the models. + +## Tests + +- To the best of your ability, ensure tests are passing before pushing. +- Follow assertion style: actual on left, expected on right. +- Always mark AI-generated tests with the `ai_generated` pytest marker. +- Use `pytest.mark.parametrize` wherever appropriate to reduce duplication in test cases. +- Avoid importing private API (names with a leading underscore) in tests. Always import from what is publicly exposed through `__init__.py` files. +- When monkeypatching internal imports in tests, target the importing module's binding (e.g., `foo.baz`), not the original definition module (e.g., `foo._bar.baz`). + +## Module and API conventions + +- Never expose private names (those with a leading underscore) in any module's `__all__`. +- Never include code other than imports, `__all__`, simple import errors, or magic `__dir__` overrides in any `__init__.py` file. +- Do not add compatibility aliases when renaming functions. Update all call sites to the canonical name instead. diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..5e558b4 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,7 @@ +# Changelog + +## Upcoming + +### 🚀 Enhancement + +- Initial implementation of the `md-lines` hook, which puts each Markdown sentence on its own line. ([#1](https://github.com/CodyCBakerPhD/md-lines-pre-commit/pull/1)) diff --git a/docs/development.md b/docs/development.md index b18bb80..f961f3e 100644 --- a/docs/development.md +++ b/docs/development.md @@ -7,6 +7,7 @@ pre-commit run --all-files ``` The `dev` extra brings pytest, pre-commit and [LinkML](https://linkml.io). +Conventions for contributors and coding agents are in [AGENTS.md](../AGENTS.md). The hook is used on this repository's own Markdown through `.pre-commit-config.yaml`. ## Layout diff --git a/pyproject.toml b/pyproject.toml index 7c9f935..8fb45be 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -33,6 +33,9 @@ packages = ["src/md_lines"] [tool.pytest.ini_options] testpaths = ["tests"] +markers = [ + "ai_generated: mark test as AI-generated", +] [tool.black] line-length = 100 diff --git a/src/md_lines/__init__.py b/src/md_lines/__init__.py index d2ac112..fd2c36e 100644 --- a/src/md_lines/__init__.py +++ b/src/md_lines/__init__.py @@ -1,10 +1,9 @@ """Put each Markdown sentence on its own line.""" -from importlib.metadata import PackageNotFoundError, version - -from md_lines.globals import DEFAULT_ABBREVIATIONS, DEFAULT_OPTIONS -from md_lines.models import Options, Paragraph, Problem, Result -from md_lines.reflow import reflow, split_sentences +from ._version import __version__ as __version__ +from .globals import DEFAULT_ABBREVIATIONS, DEFAULT_OPTIONS +from .models import Options, Paragraph, Problem, Result +from .reflow import reflow, split_sentences __all__ = [ "DEFAULT_ABBREVIATIONS", @@ -13,13 +12,6 @@ "Paragraph", "Problem", "Result", - "__version__", "reflow", "split_sentences", ] - -try: - # The version lives in pyproject.toml only; the installed metadata carries it. - __version__ = version("md-lines-pre-commit") -except PackageNotFoundError: # running from a checkout that was not pip-installed - __version__ = "0+unknown" diff --git a/src/md_lines/_version.py b/src/md_lines/_version.py new file mode 100644 index 0000000..7a71342 --- /dev/null +++ b/src/md_lines/_version.py @@ -0,0 +1,11 @@ +"""The installed version. + +It is read from the package metadata, so pyproject.toml stays the only place the version is written. +""" + +import importlib.metadata + +try: + __version__ = importlib.metadata.version("md-lines-pre-commit") +except importlib.metadata.PackageNotFoundError: # a checkout that was not pip-installed + __version__ = "0+unknown" diff --git a/src/md_lines/cli.py b/src/md_lines/cli.py index 4e7288c..9928c38 100644 --- a/src/md_lines/cli.py +++ b/src/md_lines/cli.py @@ -1,14 +1,14 @@ """Command line entry point, also what pre-commit runs.""" import argparse +import collections.abc import difflib import sys -from collections.abc import Sequence -from md_lines import __version__ -from md_lines.globals import DEFAULT_ABBREVIATIONS -from md_lines.models import Options -from md_lines.reflow import reflow +from ._version import __version__ +from .globals import DEFAULT_ABBREVIATIONS +from .models import Options +from .reflow import reflow def _build_parser() -> argparse.ArgumentParser: @@ -85,12 +85,12 @@ def _read(path: str, /) -> str: return handle.read().decode("utf-8") -def _write(path: str, text: str, /) -> None: +def _write(*, path: str, text: str) -> None: with open(path, "wb") as handle: handle.write(text.encode("utf-8")) -def main(argv: Sequence[str] | None = None) -> int: +def main(argv: collections.abc.Sequence[str] | None = None, /) -> int: args = _build_parser().parse_args(argv) options = _options_from(args) @@ -103,7 +103,7 @@ def main(argv: Sequence[str] | None = None) -> int: print(f"{path}: {error}", file=sys.stderr) errors += 1 continue - result = reflow(original, options=options) + result = reflow(text=original, options=options) if result.text == original: continue changed += 1 @@ -120,7 +120,7 @@ def main(argv: Sequence[str] | None = None) -> int: ) ) if not args.check: - _write(path, result.text) + _write(path=path, text=result.text) if changed and not args.quiet: verb = "would be rewritten" if args.check else "rewritten" diff --git a/src/md_lines/globals.py b/src/md_lines/globals.py index 8e88bd1..1acc5df 100644 --- a/src/md_lines/globals.py +++ b/src/md_lines/globals.py @@ -2,9 +2,9 @@ import re -from markdown_it import MarkdownIt +import markdown_it -from md_lines.models import Options +from .models import Options DEFAULT_ABBREVIATIONS: frozenset[str] = frozenset( { @@ -93,7 +93,7 @@ #: declared in the schema. A bare ``Options()`` has no abbreviations at all. DEFAULT_OPTIONS = Options(abbreviations=sorted(DEFAULT_ABBREVIATIONS)) -PARSER = MarkdownIt("commonmark").enable("table").enable("strikethrough") +PARSER = markdown_it.MarkdownIt("commonmark").enable("table").enable("strikethrough") # First line of a paragraph that starts a list item: any mix of block quote markers and list # markers, each followed by whitespace. diff --git a/src/md_lines/reflow.py b/src/md_lines/reflow.py index 777057c..b7bb4dc 100644 --- a/src/md_lines/reflow.py +++ b/src/md_lines/reflow.py @@ -11,14 +11,14 @@ import re -from md_lines import globals as g -from md_lines.models import Options, Paragraph, Problem, Result +from . import globals as g +from .models import Options, Paragraph, Problem, Result def _mask_inline(text: str, /) -> str: """Replace the inside of inline code, links, HTML, URLs and math with ``x``.""" - def keep_edges(match: re.Match[str], edge: int) -> str: + def keep_edges(*, match: re.Match[str], edge: int) -> str: whole = match.group() inner = len(whole) - 2 * edge return whole[:edge] + "x" * inner + whole[len(whole) - edge :] @@ -26,14 +26,14 @@ def keep_edges(match: re.Match[str], edge: int) -> str: text = g.CODE_SPAN_RE.sub(lambda m: m.group(1) + "x" * len(m.group(2)) + m.group(1), text) text = g.HTML_COMMENT_RE.sub(lambda m: "x" * len(m.group()), text) text = g.ANGLE_RE.sub(lambda m: "x" * len(m.group()), text) - text = g.LINK_DESTINATION_RE.sub(lambda m: keep_edges(m, 2), text) - text = g.REFERENCE_LABEL_RE.sub(lambda m: keep_edges(m, 2), text) + text = g.LINK_DESTINATION_RE.sub(lambda m: keep_edges(match=m, edge=2), text) + text = g.REFERENCE_LABEL_RE.sub(lambda m: keep_edges(match=m, edge=2), text) text = g.URL_RE.sub(lambda m: "x" * len(m.group()), text) text = g.MATH_RE.sub(lambda m: "x" * len(m.group()), text) return text -def split_sentences(text: str, /, *, options: Options = g.DEFAULT_OPTIONS) -> list[str]: +def split_sentences(*, text: str, options: Options = g.DEFAULT_OPTIONS) -> list[str]: """Split one line of Markdown prose into its sentences.""" abbreviations = set(options.abbreviations or ()) masked = _mask_inline(text) @@ -105,7 +105,7 @@ def _paragraphs(lines: list[str], /) -> list[Paragraph]: return paragraphs -def _split_prefix(line: str, /, *, item_start: bool) -> tuple[str, str]: +def _split_prefix(*, line: str, item_start: bool) -> tuple[str, str]: pattern = g.ITEM_PREFIX_RE if item_start else g.LINE_PREFIX_RE match = pattern.match(line) assert match is not None @@ -128,7 +128,7 @@ def _ends_with_hard_break(content: str, /) -> bool: return backslashes % 2 == 1 -def _continues_sentence(content: str, /, *, options: Options) -> bool: +def _continues_sentence(*, content: str, options: Options) -> bool: """Whether the following line continues the sentence that ends on ``content``.""" if _ends_with_hard_break(content): return False @@ -137,16 +137,16 @@ def _continues_sentence(content: str, /, *, options: Options) -> bool: def _reflow_paragraph( - lines: list[str], paragraph: Paragraph, /, *, options: Options, problems: list[Problem] + *, lines: list[str], paragraph: Paragraph, options: Options, problems: list[Problem] ) -> list[str]: raw = lines[paragraph.start : paragraph.end] - first_prefix, first_content = _split_prefix(raw[0], item_start=paragraph.starts_item) + first_prefix, first_content = _split_prefix(line=raw[0], item_start=paragraph.starts_item) continuation_prefix = _continuation_prefix(first_prefix) - contents = [first_content] + [_split_prefix(line, item_start=False)[1] for line in raw[1:]] + contents = [first_content] + [_split_prefix(line=line, item_start=False)[1] for line in raw[1:]] joined: list[tuple[int, str]] = [] for offset, content in enumerate(contents): - if joined and _continues_sentence(joined[-1][1], options=options): + if joined and _continues_sentence(content=joined[-1][1], options=options): first_offset, previous = joined[-1] problems.append( Problem( @@ -165,7 +165,7 @@ def _reflow_paragraph( if paragraph.in_list and not options.split_lists: sentences = [body] else: - sentences = split_sentences(body, options=options) + sentences = split_sentences(text=body, options=options) if len(sentences) > 1: problems.append( Problem( @@ -179,7 +179,7 @@ def _reflow_paragraph( return [first_prefix + out[0]] + [continuation_prefix + sentence for sentence in out[1:]] -def reflow(text: str, /, *, options: Options = g.DEFAULT_OPTIONS) -> Result: +def reflow(*, text: str, options: Options = g.DEFAULT_OPTIONS) -> Result: """Return the reflowed text and the problems that were found.""" bom = "" if text.startswith("") else "" body = text[len(bom) :] @@ -192,7 +192,7 @@ def reflow(text: str, /, *, options: Options = g.DEFAULT_OPTIONS) -> Result: fixed = list(lines) for paragraph in reversed(_paragraphs(lines)): fixed[paragraph.start : paragraph.end] = _reflow_paragraph( - lines, paragraph, options=options, problems=problems + lines=lines, paragraph=paragraph, options=options, problems=problems ) problems.sort(key=lambda problem: problem.line) diff --git a/tests/test_cli.py b/tests/test_cli.py index 8edb455..c682c49 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -1,7 +1,11 @@ import subprocess +import pytest + from md_lines.cli import main +pytestmark = pytest.mark.ai_generated + def write(tmp_path, name, text): path = tmp_path / name diff --git a/tests/test_reflow.py b/tests/test_reflow.py index 3859e8a..c3bf5fb 100644 --- a/tests/test_reflow.py +++ b/tests/test_reflow.py @@ -2,10 +2,12 @@ from md_lines import DEFAULT_ABBREVIATIONS, Options, reflow, split_sentences +pytestmark = pytest.mark.ai_generated + def fix(text, **kwargs): settings = {"abbreviations": sorted(DEFAULT_ABBREVIATIONS), **kwargs} - return reflow(text, options=Options(**settings)).text + return reflow(text=text, options=Options(**settings)).text def test_wrapped_sentence_is_joined(): @@ -35,7 +37,7 @@ def test_break_after_semicolon_is_joined_by_default_and_kept_when_asked(): def test_problems_report_original_line_numbers(): - result = reflow("Intro\n\nOne\nwrapped. Two.\n") + result = reflow(text="Intro\n\nOne\nwrapped. Two.\n") assert [(p.line, p.message) for p in result.problems] == [ (3, "sentence is wrapped onto line 4"), (3, "line holds 2 sentences"), @@ -157,7 +159,7 @@ def test_closing_quotes_and_parentheses_after_punctuation(): def test_enumerator_at_line_start_is_not_a_sentence(): assert fix("1. Not a list item because of the\\\nescape. Done.\n") != "" - assert split_sentences("3. Item text. Second.") == ["3. Item text.", "Second."] + assert split_sentences(text="3. Item text. Second.") == ["3. Item text.", "Second."] def test_lowercase_after_period_does_not_split(): diff --git a/tests/test_schema.py b/tests/test_schema.py index b967299..027d668 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -4,6 +4,10 @@ import pytest +from md_lines import Options + +pytestmark = pytest.mark.ai_generated + ROOT = pathlib.Path(__file__).resolve().parents[1] SCHEMA = ROOT / "src" / "md_lines" / "schema" / "md_lines.yaml" MODELS = ROOT / "src" / "md_lines" / "models.py" @@ -27,8 +31,6 @@ def test_models_are_generated_from_the_schema(): def test_generated_defaults_match_the_schema(): - from md_lines import Options - options = Options() assert options.split_lists is False assert options.keep_breaks_after == ".!?" From 2cc5030887a1e5851e593cbdd32a0595566001db Mon Sep 17 00:00:00 2001 From: Cody Baker Date: Thu, 8 Oct 2026 01:27:33 +0000 Subject: [PATCH 08/10] Upload coverage to Codecov from CI The tests job runs pytest with coverage and uploads coverage.xml with codecov/codecov-action, failing when the upload fails, as in CodyCBakerPhD/historia. pytest-cov joins the dev extra and the generated models.py is omitted from coverage. actions/checkout and actions/setup-python move to v7. docs/development.md names the CODECOV_TOKEN secret the upload needs. Co-Authored-By: Claude Code 2.1.293 / Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01ToiYPiSdU8NM9h1UmLgsq8 --- .github/workflows/continuous_integration.yml | 16 +++++++++++----- docs/development.md | 3 ++- pyproject.toml | 6 +++++- 3 files changed, 18 insertions(+), 7 deletions(-) diff --git a/.github/workflows/continuous_integration.yml b/.github/workflows/continuous_integration.yml index d11604e..78f7a02 100644 --- a/.github/workflows/continuous_integration.yml +++ b/.github/workflows/continuous_integration.yml @@ -12,24 +12,30 @@ jobs: tests: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v7 with: persist-credentials: false - - uses: actions/setup-python@v5 + - uses: actions/setup-python@v7 with: # Latest stable Python only; pyproject.toml sets the floor. python-version: "3.x" - run: pip install -e ".[dev]" - - run: pytest -q + - run: pytest --cov=src/md_lines --cov-report=xml + - name: Upload coverage to Codecov + uses: codecov/codecov-action@v7 + with: + token: ${{ secrets.CODECOV_TOKEN }} + files: ./coverage.xml + fail_ci_if_error: true hook: name: pre-commit hooks and try-repo runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v7 with: persist-credentials: false - - uses: actions/setup-python@v5 + - uses: actions/setup-python@v7 with: python-version: "3.x" - run: pip install -e ".[dev]" diff --git a/docs/development.md b/docs/development.md index f961f3e..9b25ae8 100644 --- a/docs/development.md +++ b/docs/development.md @@ -27,7 +27,8 @@ gen-python --no-metadata src/md_lines/schema/md_lines.yaml > src/md_lines/models ## Repository checks The pre-commit configuration follows [historia](https://github.com/CodyCBakerPhD/historia): pre-commit-hooks, check-github-workflows, black, ruff (lint plus isort rules, with `--fix`), codespell with the `en-GB_to_en-US` dictionary so the text stays in American English, and this repository's own `md-lines` hook. -CI runs the tests and the hooks on the newest Python release. +CI runs the tests and the hooks on the newest Python release, on every pull request and on pushes to `main`. +The tests job uploads coverage to [Codecov](https://codecov.io/gh/CodyCBakerPhD/md-lines-pre-commit) and fails when the upload does, so the repository needs a `CODECOV_TOKEN` secret (Settings, Secrets and variables, Actions). ## Exit status diff --git a/pyproject.toml b/pyproject.toml index 8fb45be..b17b059 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -20,7 +20,7 @@ classifiers = [ [project.optional-dependencies] test = ["pytest>=7"] # Regenerating src/md_lines/models.py from the LinkML schema, and the test that checks it. -dev = ["pytest>=7", "linkml>=1.9", "pre-commit"] +dev = ["pytest>=7", "pytest-cov", "linkml>=1.9", "pre-commit"] [project.scripts] md-lines = "md_lines.cli:main" @@ -31,6 +31,10 @@ Repository = "https://github.com/CodyCBakerPhD/md-lines-pre-commit" [tool.hatch.build.targets.wheel] packages = ["src/md_lines"] +[tool.coverage.run] +# Generated from the LinkML schema; its many unused branches are not ours to cover. +omit = ["src/md_lines/models.py"] + [tool.pytest.ini_options] testpaths = ["tests"] markers = [ From f029bc8f8e15b6063af465b4cc4431c29ef48248 Mon Sep 17 00:00:00 2001 From: Cody Baker Date: Thu, 8 Oct 2026 01:34:24 +0000 Subject: [PATCH 09/10] CI: run only what pre-commit.ci cannot Review feedback on #1. pre-commit.ci runs the hooks on every pull request, so the Actions job no longer repeats them. It keeps the two things pre-commit.ci does not do: this repository's own md-lines hook (language: system, skipped there) and the try-repo installation test. Co-Authored-By: Claude Code 2.1.293 / Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01ToiYPiSdU8NM9h1UmLgsq8 --- .github/workflows/continuous_integration.yml | 9 ++++++--- docs/development.md | 3 ++- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/.github/workflows/continuous_integration.yml b/.github/workflows/continuous_integration.yml index 78f7a02..763767b 100644 --- a/.github/workflows/continuous_integration.yml +++ b/.github/workflows/continuous_integration.yml @@ -28,8 +28,11 @@ jobs: files: ./coverage.xml fail_ci_if_error: true + # pre-commit.ci runs the other hooks on every pull request. It cannot run this repository's + # own md-lines hook (`language: system`, skipped in .pre-commit-config.yaml), and it does not + # test installing the hook the way a consumer does, so this job covers those two things. hook: - name: pre-commit hooks and try-repo + name: md-lines hook and try-repo runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 @@ -39,7 +42,7 @@ jobs: with: python-version: "3.x" - run: pip install -e ".[dev]" - - name: Hooks of this repository (ruff, hygiene, md-lines on README.md) - run: pre-commit run --all-files --show-diff-on-failure + - name: This repository's own md-lines hook on its Markdown + run: pre-commit run md-lines --all-files --show-diff-on-failure - name: Install the hook the way a consumer would run: pre-commit try-repo . md-lines --files README.md --verbose diff --git a/docs/development.md b/docs/development.md index 9b25ae8..bf0d462 100644 --- a/docs/development.md +++ b/docs/development.md @@ -27,7 +27,8 @@ gen-python --no-metadata src/md_lines/schema/md_lines.yaml > src/md_lines/models ## Repository checks The pre-commit configuration follows [historia](https://github.com/CodyCBakerPhD/historia): pre-commit-hooks, check-github-workflows, black, ruff (lint plus isort rules, with `--fix`), codespell with the `en-GB_to_en-US` dictionary so the text stays in American English, and this repository's own `md-lines` hook. -CI runs the tests and the hooks on the newest Python release, on every pull request and on pushes to `main`. +pre-commit.ci runs the hooks on every pull request, except this repository's own `md-lines` hook, which it cannot run. +The Actions workflow runs the tests on the newest Python release, runs that one hook, and installs the hook the way a consumer does with `pre-commit try-repo`. The tests job uploads coverage to [Codecov](https://codecov.io/gh/CodyCBakerPhD/md-lines-pre-commit) and fails when the upload does, so the repository needs a `CODECOV_TOKEN` secret (Settings, Secrets and variables, Actions). ## Exit status From 203bbd9d9eb987503f3a5f63971150b6d578767c Mon Sep 17 00:00:00 2001 From: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com> Date: Wed, 7 Oct 2026 22:47:00 -0400 Subject: [PATCH 10/10] Update README.md Signed-off-by: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com> --- README.md | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 93c3a39..3974c42 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,9 @@ # md-lines-pre-commit A [pre-commit](https://pre-commit.com) hook that puts each sentence of a Markdown file on its own line. + Wrapped sentences are joined back onto one line, and lines that hold several sentences are split, so diffs and reviews work sentence by sentence. + Everything that is not prose stays exactly as it is. ## Usage @@ -16,8 +18,6 @@ repos: - id: md-lines ``` -The hook needs Python 3.13 or newer; if pre-commit itself runs under an older Python, add `language_version: python3.13` to the hook entry. - Or directly from the command line (after installing with `pip install git+https://github.com/CodyCBakerPhD/md-lines-pre-commit`): ```sh @@ -43,9 +43,6 @@ Block structure comes from a CommonMark parser ([markdown-it-py](https://github. Sentence ends are not detected inside inline code, link destinations, autolinks, inline HTML, bare URLs or `$math$`, nor after common abbreviations (`e.g.`, `etc.`, `Fig.`, `Dr.`, months and so on), initials (`J. Smith`), dotted abbreviations (`U.S.`) or an enumerator at the start of a line (`1.`). A bold lead-in such as `**Note.**` stays attached to the sentence it introduces, and no split happens inside unbalanced bold markers, brackets or parentheses. -Line endings (LF or CRLF), a byte order mark and a missing final newline are preserved. -Running the hook on its own output changes nothing. - ## Options | Option | Default | Meaning | @@ -65,5 +62,3 @@ Pass options through pre-commit with `args`: - id: md-lines args: ["--split-lists", "--abbreviations", "chap.,sect."] ``` - -Development notes, including how the data model is generated from its LinkML schema, are in [docs/development.md](docs/development.md).