Skip to content

fix: Windows MinerU parsing — subprocess UTF-8 decoding & render-thread crash - #844

Open
1939566682 wants to merge 1 commit into
HKUDS:mainfrom
1939566682:fix/windows-robustness
Open

fix: Windows MinerU parsing — subprocess UTF-8 decoding & render-thread crash#844
1939566682 wants to merge 1 commit into
HKUDS:mainfrom
1939566682:fix/windows-robustness

Conversation

@1939566682

Copy link
Copy Markdown
## Summary
Two Windows-specific fixes for the local MinerU parsing engine.

- `local.py`: set `encoding="utf-8", errors="replace"` on the MinerU
  `subprocess.Popen` call. Previously `text=True` decoded MinerU's UTF-8
  progress output with the system locale; on Chinese Windows that is GBK,
  causing `UnicodeDecodeError: 'gbk' codec can't decode byte 0x8b` and
  aborting every parse.
- `models.py`: inject `MINERU_PDF_RENDER_THREADS=1` in `model_env_overrides`.
  MinerU 3.x crashes with `ucrtbase.dll 0xc0000409` heap corruption on
  Windows when >1 PDF render thread runs (MinerU issue #5033). Forcing 1
  avoids it with negligible impact on single-doc parsing.

## Test plan
- [ ] Parsed a multi-page Chinese PDF on Windows 11 (zh-CN), Python 3.13,
      MinerU 3.4.4 — no decode error, parse completes.
- [ ] Long-running parse (>30 min) no longer crashes with 0xc0000409.

## Notes
Verified on Windows 11 zh-CN / Python 3.13 / MinerU 3.4.4.

…atomic-write PermissionError

- mineru/local.py: pass encoding="utf-8", errors="replace" to subprocess.Popen so MinerU's combined stdout/stderr is decoded as UTF-8 instead of the platform default (GBK on zh-CN Windows), which previously crashed with a UnicodeDecodeError (0x8b).
- mineru/models.py: set MINERU_PDF_RENDER_THREADS=1 in model_env_overrides to avoid MinerU's multi-threaded PDF rendering heap-corruption crash (0xc0000409) on Windows.
- services/file_io.py: retry atomic replace on transient PermissionError (e.g. target locked by antivirus/indexer) with exponential backoff, instead of failing the whole write on Windows.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant