Not crashing on a metadata provider's HTTP error status like 429

[rokokol]

DOIOrTitleBasedProvider.query is meant to degrade to None when a metadata provider errors, as its comment says: "we're suppressing this error to not fail on 403 or 500 errors from providers". Since #1062 moved the clients from aiohttp to httpx it catches httpx.RequestError, which covers transport failures but not httpx.HTTPStatusError; aiohttp.ClientError, which it replaced, covered both. So any status that is_retryable does not retry escapes the lookup. The one that bites in practice is Semantic Scholar's 429 for anonymous traffic

During indexing that escalates: process_file marks the paper as failed, re-raises because the error is not a ValueError, and the task group ends the whole get_directory_index run. With parsing.use_doc_details on (the default) and no SEMANTIC_SCHOLAR_API_KEY, one 429 on one paper is enough:

Error parsing 2510.09720.pdf, skipping index for this file.
HTTPStatusError: Client error '429 Too Many Requests' for url ...
ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)

Reproduced deterministically on v2026.08.12 with pqa index over one PDF, after patching paperqa.clients.semantic_scholar.SEMANTIC_SCHOLAR_BASE_URL to a local server that answers every request with 429; the same code is on main

This PR catches httpx.HTTPError, the common base of RequestError and HTTPStatusError, which restores what the aiohttp version did. The paper then gets a plain Doc and a warning, the same as when a provider is unreachable

Tests

• test_client_http_status_error, parametrized over CrossrefProvider and SemanticScholarProvider, has the HTTP client answer 429 and expects query to return None. It fails on main with httpx.HTTPStatusError and passes with the change
• tests/test_clients.py passes in full with --record-mode=none (51 tests); mypy and the pinned ruff-check and black are clean on both files

A related behaviour, left alone here: once a file is marked ERROR, SearchIndex.filecheck treats it as indexed, so later builds into the same index skip it without a log line, even when the failure was transient like this one. Happy to open an issue for that if it is worth discussing

[Copilot]

🟢 Approval recommended

The focused change is regression-tested and has no unresolved blocking issues.

Pull request overview

Updates metadata providers to gracefully handle HTTP errors such as 429 responses without aborting indexing.

Changes:

• Catch httpx.HTTPError during provider queries.
• Add regression tests for Crossref and Semantic Scholar 429 responses.

File summaries

File	Summary
tests/test_clients.py	Tests graceful handling of 429 responses.
src/paperqa/clients/client_models.py	Suppresses HTTP transport and status errors during metadata lookup.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.