fix(cache): cache rules never matched data.sec.gov, or a mirror's own host

[joseturegano]

What

Cache rules are keyed by a regex that httpxthrottlecache matches against the request host, before it ever looks at the path (controller.get_rules). _get_cache_rules() built one key, from SEC_BASE_URL:

>>> import re
>>> re.match(r'.*www\.sec\.gov', 'data.sec.gov')
None

/submissions and /api/xbrl/companyfacts are served from SEC_DATA_URL (edgar/urls.py:39 and :45), so no rule under that key has ever applied to a data.sec.gov request, whatever its path. Verified live: a fresh process logs No patterns matched data.sec.gov and goes to the network every time.

Two consequences:

1. The /submissions TTL has never taken effect — including the deliberate 600s → 30s reduction in 655f5d30 for 8-K reports available with a lag #471, made after a user reported same-day 8-Ks taking up to 10 minutes to appear.
2. get_facts() re-downloads companyfacts on every call, in every process.

This started at #490, which replaced the literal r".*\.sec\.gov" — a pattern that did match data.sec.gov — with one derived from the base URL alone. Thanks to @paultiq for pointing there; it also answers the question below about whether excluding companyfacts was ever a decision.

The fix

One key per host edgar.urls actually uses, each rule filed under the host that really serves it, and each key matching that host exactly:

host = httpx.URL(url).host
return f"{re.escape(host)}$"

Exactness is not decoration. An unanchored key leaks across hosts — .*mirror\.example\.org also matches data.mirror.example.org, and get_rules commits to the FIRST key that matches — so a two-host mirror would resolve its data requests to the base rule set, which holds no rule for them. That is this same bug, moved from sec.gov to mirror users, i.e. to exactly the people #490 was written for.

And the host is taken from httpx.URL(...).host, the same parser that produces the value it will be compared against. A hand-rolled https?://([^/]+) keeps what httpx normalises away, so four valid ways to configure a mirror each produced a key no real request could match:

EDGAR_DATA_URL	request.url.host	old key matched
https://DATA.mirror.example.org	data.mirror.example.org	✗
https://data.mirror.example.org:8443	data.mirror.example.org	✗
https://user:pw@data.mirror.example.org	data.mirror.example.org	✗
https://mirr%C3%B6r.example.org	mirr%c3%b6r.example.org	✗

A mirror pointing both env vars at one host still merges into a single key. A URL with no host at all is logged and matches nothing — a misconfigured mirror goes uncached, which is slow, rather than borrowing another host's rules, which would be wrong.

One deliberate narrowing: exact matching no longer covers efts.sec.gov, the only other sec.gov host in the codebase (edgar/search/efts.py). Not a regression — EFTS_BASE_URL is hardcoded rather than configurable, its /LATEST/search-index path matches no rule in either rule set, and main's .*www\.sec\.gov key does not match that host either. Uncached before, uncached after.

The companyfacts TTL, and why not longer

It reuses MAX_SUBMISSIONS_AGE_SECONDS (30s) rather than a new constant — and deliberately not the superseded 600s. companyfacts is invalidated by the same event as /submissions (a filing lands), so a longer budget would reintroduce exactly the staleness #471 removed, for a sibling endpoint.

Honest scope of the benefit. A batch calling get_facts() once per distinct CIK gains ~nothing from a 30s TTL — each CIK is requested once, far outside any 30s window. The real benefit is (a) consistency with the policy already set for /submissions, and (b) no duplicate round-trips for repeated or interactive calls to the same company in a short window, which is what #471 was about. I would rather state that than sell a speedup this does not deliver.

Was excluding companyfacts deliberate?

I looked before proposing:

• PR Implement client cache controller for Edgar #345 (the caching-rules design) and its parent feat: Request caching using Hishel / HTTPX #214 discuss only /submissions, the full-index files and /Archives/edgar/data. XBRL/companyfacts is not mentioned.
• No issue, PR or discussion mentions companyfacts caching or staleness.
• edgar/entity/CLAUDE.md describes a "24 hour TTL for facts" that was never implemented (the use_cache parameter it references does not exist near get_facts()) — a weak signal for caching, never wired up.
• companyfacts responses carry no Cache-Control, ETag or Last-Modified (checked live), so header-based revalidation is not an option either way.

Conclusion: no documented deliberate exclusion — and #490 shows the mechanism by which it was lost.

Upstream dependency

Fixing the host matching reactivates age-based freshness checking for data.sec.gov for the first time, and FileCache.get_if_fresh raised uncaught there when the local clock trails the origin's Date header — taking down the whole get_facts() call. Reproduced live with real skew (~117s), through the real request path.

That was this PR's blocking finding. It is fixed upstream (paultiq/httpxthrottlecache#43, PR #44) and released in 0.6.1, which this PR now pins as the floor. Thank you @paultiq for turning it around quickly.

Tests

tests/issues/regression/test_data_sec_gov_cache_domain.py — 23 cases, offline and deterministic, so they run in the PR gate rather than only in the weekly regression job.

8 of them fail on the previous commit: three cross-host cases (including a lookalike host that inherited the cache-forever /Archives rule) and five parser-mismatch cases. The rest are negative guards — www.sec.gov rules unchanged, an unrelated host still uncached — which pass either way by design and exist to catch a future over-permissive key.

One class is about the tests themselves: they check _get_cache_rules through a local replica of the library's matcher, and a replica can be wrong in the same direction as the code it checks. Mine was, briefly: it fell through to later keys where the library stops at the first, which made every cross-host assertion incapable of failing. It is now pinned against controller.get_rule_for_request itself, over deliberately overlapping keys — the only shape where a fall-through replica and the library disagree.

Checked per CONTRIBUTING: hatch run lint reports 1115 errors with and without this change (the count on main), ruff check and ruff format --check are clean on both changed files, and hatch run check-cassettes passes (214 files, none recorded or touched here). pytest -m fast is 4974 passed / 12 failed locally, and those 12 fail identically with edgar/httpclient.py and pyproject.toml reverted to main — Windows-local, not this change.

[joseturegano]

CI on this PR is red with 2 failures, and since both touch data.sec.gov — the host this PR starts routing through the caching transport — it's fair to suspect the change. I looked before assuming otherwise; here's what I found.

The failures are DNS, not assertions:

FAILED tests/issues/reproductions/data-quality/test_currency_fix.py::test_currency_fix
FAILED tests/issues/reproductions/data-quality/test_currency_usd.py::test_usd_currency
httpx.ConnectError: [Errno -5] No address associated with hostname
2 failed, 4458 passed, 4 skipped, 1 xfailed, 2 rerun in 397.74s

Both retried 3 times and failed all 3 with the same resolver error.

Locally, on this branch, with real network, both pass:

$ EDGAR_IDENTITY="..." python -m pytest \
    tests/issues/reproductions/data-quality/test_currency_fix.py \
    tests/issues/reproductions/data-quality/test_currency_usd.py -q
2 passed, 7 warnings in 6.58s

Why I don't think the change can cause this: [Errno -5] No address associated with hostname is a resolver failure, raised before any HTTP exchange. A cache rule cannot introduce one — it can only avoid network calls, never add them. On a cache hit these requests wouldn't reach the resolver at all.

I can't rerun the job (no admin rights on this repo), so I can't demonstrate flakiness directly. If you'd rather have positive proof than my reasoning, a rerun of the failed jobs would settle it — and if it comes back red the same way, I'll dig further rather than leave it at "flaky".

Worth noting the PR is a draft for a separate reason (the upstream clock-skew crash in the Blocking finding section), so there's no rush on this either way.

[dgunning]

Verified the whole chain independently — the diagnosis and the fix both hold up. One correction to the history in the description, plus two small notes.

Confirmed

Reproduced against 0.6.0, the version pyproject.toml actually requires (worth flagging: system Python here had 0.3.0, so this needs checking inside the project env or you get the wrong answer):

edgartools CACHE_RULES key: ['.*www\\.sec\\.gov']
  www.sec.gov   /submissions/CIK0000320193.json           -> 30
  data.sec.gov  /submissions/CIK0000320193.json           -> None
  data.sec.gov  /api/xbrl/companyfacts/CIK0000320193.json -> None

controller.py:12 matches site_pattern against request_host, and urls.py:39/:45 put both endpoints on SEC_DATA_URL. So no rule under that key has applied to a data.sec.gov request. Your 5 tests pass on the branch, and 55 cache/httpclient tests pass with no regressions.

The blocking finding holds too. filecache/transport.py:116-118 raises unguarded on negative age, and :318 confirms fetched is the server's Date header, not the local clock. Agreed this blocks — trading a caching bug for a hard get_facts() failure on skewed clocks is a bad trade.

Correction: this is a dated regression, not a permanent condition

The /submissions TTL has never taken effect — including the deliberate 600s → 30s reduction in 655f5d30

It did take effect, for about two weeks. Before bd5d2397 the key was a literal:

CACHE_RULES = {
    r".*\.sec\.gov": {            # <- matches data.sec.gov
        "/submissions.*": MAX_SUBMISSIONS_AGE_SECONDS,

Timeline:

• 2025-10-28 655f5d30 cuts the TTL 600 → 30 for #471. Key is still .*\.sec\.gov, so data.sec.gov matches and the 30s does apply.
• 2025-11-10 bd5d2397 ("Configure SEC urls via environment variable") replaces the literal with a domain derived from SEC_BASE_URL, yielding .*www\.sec\.gov. data.sec.gov stops matching that day.

So the #471 fix worked as intended and was then silently undone ~9 months ago by an unrelated mirror-support change. That reframes two things:

1. Your PR restores prior behaviour rather than enabling something new — a stronger and more defensible story for the CHANGELOG entry.
2. It undercuts "reactivates age-based freshness checking for data.sec.gov for the first time" in the blocking section. That path was live before 2025-11-10, so the # pragma: no cover shield is ~9 months old, not permanent. Doesn't change the recommendation — the crash is still real and still gated on the upstream clamp — but the framing should match.

Corroborating from a second direction: upstream still ships httpxthrottlecache.EDGAR_CACHE_RULES keyed .*\.sec\.gov, and it resolves data.sec.gov + /submissions to 600 today. Upstream's own shape never had this bug; edgartools narrowed it locally.

Two smaller notes

www.sec.gov + /submissions goes 30 → None. Right in principle, since submissions is only served from the data host. Worth a deliberate nod for a mirror that splits the two hosts but serves submissions from the base one — the merge branch only covers mirrors where both collapse to one host.

Host patterns are unanchored at the end (pre-existing, not yours):

www.sec.gov.attacker.com  -> matches '.*www\.sec\.gov'
data.sec.gov.evil.net     -> matches '.*data\.sec\.gov'

Low severity — it only selects a cache TTL, and someone would have to point EDGAR_BASE_URL at such a host — but you're rewriting this exact function, so it's the cheap moment to add a $.

Routing

Agreed on landing the upstream clamp first. Please do open that PR against httpxthrottlecache — and once it ships I'd rather take the minimum-version pin here than carry a workaround, given >=0.6.0 is already pinned for the [httpx] extra.

Excellent write-up, incidentally. The "was excluding companyfacts deliberate?" section and the honest scoping of the batch benefit are the two things that made this quick to verify rather than quick to doubt.

🤖 Review assisted by Claude Code

[joseturegano]

Force-pushed a refresh (75b49877 → 5f96c665). What changed, so you don't have to diff it:

• Rebased onto main at 5.48.0 — it was 44 commits behind. No conflicts, and _get_cache_rules() is untouched by anything that landed in between.
• Added the CHANGELOG.md entry I owed you. CONTRIBUTING.md asks for one and the original push didn't have it. Under [Unreleased] / Fixed, 69 words, symptom-first, ending in (GH #989).
• Re-verified the regression rather than inheriting the claim. Reverting edgar/httpclient.py to main and re-running still fails exactly 3 of the 5 cases — submissions, companyfacts, and the same-host mirror. The other 2 are the negative guards and pass either way, as before.

Also checked, since 44 commits is enough for a stale claim to go bad: hatch run lint and ruff format --check report the same 2 findings and the same unformatted status on edgar/httpclient.py with and without my change — the file already carries both on main, so this adds no lint or format debt. I deliberately did not run ruff format on it, since that would reformat your existing code and bury a 60-line fix under it. Say the word if you'd rather have the reformat, and I'll send it as its own commit. check-cassettes is clean (214 files); no cassettes recorded or touched here.

On the two red CI tests from the first run: I said they looked like flaky DNS and I went looking for a way I could be wrong. Two things say I'm not. Neither test_currency_fix nor test_currency_usd uses a cassette, so the failure can't be a VCR miss falling through to the network — which was the one mechanism by which a cache-rule change could plausibly have caused a resolver error. And your 5.48.0 commit records test-fast green at 4,950 passed, with both tests present. This push should produce a fresh run either way.

Still a draft for the same reason as before, and only that reason: the clock-skew crash in the Blocking finding section. I've now opened it upstream with a fix and tests — paultiq/httpxthrottlecache#44, against paultiq/httpxthrottlecache#43, which he'd already accepted. Once it ships I'll pin a minimum version here and take this out of draft.

[paultiq]

paultiq/httpxthrottlecache#44 is merged, and httpxthrottlecache v0.6.1 is released.

FYI: #490 is where data.sec.gov caching was broken. It's not mentioned in this discussion, and should be looked at closely to make sure this PR doesn't introduce a regression when someone uses SEC_DATA_URL.

[joseturegano]

@paultiq Thanks — that pointer found a live hole in this PR, not a hypothetical one. #490 is where it broke, and the mechanism that broke it was still in my fix.

#490 is the commit. It replaced the literal r".*\.sec\.gov" with a key derived from SEC_BASE_URL alone, so .*www\.sec\.gov — and re.match(r'.*www\.sec\.gov', 'data.sec.gov') is None. The old literal did match data.sec.gov. Worth noting for the companyfacts half of this PR: that also means excluding companyfacts was never a decision, it was collateral from a mirror-support change.

What you were right to suspect. My fix added a second key but kept the hand-built .*<domain> shape, and get_rules returns the FIRST key that matches. Reproduced against the real httpxthrottlecache.controller.get_rule_for_request (0.6.0), 5 mirror configurations, asking for /submissions and /api/xbrl/companyfacts on the data host:

                          previous commit          this push
default                   OK                       OK
base=data=mirror.com      OK                       OK
base=mirror.com,          OK                       OK
  data=data.sec.gov
base=mirror.com,          not cached               OK
  data=data.mirror.com
base=sec.mirror.test,     not cached               OK
  data=data.sec.mirror.test

.*mirror\.com matches data.mirror.com, base is inserted first, so the data host resolved to the base rule set — which has no /submissions rule. The bug this PR fixes, reintroduced for exactly the users #490 was written for.

And a second one, which is why the fix is not just an anchor. The key is matched against request.url.host. Building it with https?://([^/]+) keeps what httpx normalises away, so these four perfectly valid ways to configure a mirror each produced a key that no real request could match — the mirror silently loses caching entirely:

EDGAR_DATA_URL                          httpx host              old key matched?
https://DATA.mirror.example.org         data.mirror.example.org   False
https://data.mirror.example.org:8443    data.mirror.example.org   False
https://user:pw@data.mirror.example.org data.mirror.example.org   False
https://mirr%C3%B6r.example.org         mirr%c3%b6r.example.org   False

So _host_key now takes the host from httpx.URL(...).host — the same parser that produces the value it will be compared against — and matches it exactly. The two sides cannot disagree, and a rule written for one host cannot answer for another. A URL with no host is logged and matches nothing: a misconfigured mirror goes uncached rather than borrowing someone else's rules.

One consequence worth stating plainly, since it is a narrowing: exact matching drops efts.sec.gov, the only other sec.gov host in the codebase (edgar/search/efts.py). That is not a regression — EFTS_BASE_URL is hardcoded rather than configurable, its /LATEST/search-index path matches no rule in either rule set, and on main the .*www\.sec\.gov key does not match that host either. It was uncached before this PR and it is uncached after.

Tests. 23 cases, offline and deterministic, and marked fast by your conftest, so they gate the PR rather than only the weekly job. 8 fail on the previous commit — three cross-host and five parser-mismatch. Two of those only started failing after I fixed the test helper itself: it fell through to later keys where the real matcher stops at the first, which made every cross-host assertion incapable of failing. It is now pinned against controller.get_rule_for_request over deliberately overlapping keys — agreement on the live, non-overlapping rules is free and could not fail on its own. The test_lookalike_host_gets_no_rules case is the one I would keep if I could keep only one: under an unanchored key, www.sec.gov.attacker.test inherits every rule including cache-forever on /Archives.

Also in this push: rebased onto main at 40021d1, and pinned httpxthrottlecache>=0.6.1 now that #44 is released — thank you for turning that around so fast. That was the blocking finding, so this is now out of draft.

Per CONTRIBUTING: hatch run lint reports 1115 errors with and without this change, hatch run check-cassettes passes (214 files, none recorded or touched), and ruff check / ruff format --check are clean on both changed files. pytest -m fast is 4974 passed / 12 failed here, and those same 12 fail identically with edgar/httpclient.py and pyproject.toml reverted to main — Windows-local, not this change.

edgar/httpclient.py itself is one of the files ruff format would rewrite on main, and I have again kept my formatter off it: reformatting it would bury a 60-line fix under unrelated churn. Say the word and I will send that as its own commit.