feat(waterdata): replace per-page logger.info with a single progress line

[thodson-usgs]

Summary

Paginated and chunked Water Data queries narrate their progress with per-page logger.info calls — Requesting, Next URL, and Remaining requests this hour. These scroll the console one line per page and are silent unless the caller has configured logging.

This replaces them with a single self-updating status line on stderr, rewritten in place as data arrives:

Progress: chunk 2/5 · 14 pages · 8,421 rows · 4,870 requests remaining, resets in 47m

It surfaces the two kinds of work the caller otherwise can't see — CQL-filter chunks (known up front) and next-link pages (count unknown up front) — plus rows accumulated and the requests remaining for the hour (with a countdown to the limit reset, from x-ratelimit-remaining / x-ratelimit-reset, when the server reports them). No new dependency.

Usage

import dataretrieval.waterdata as wd

# Pages through `next` links; progress shows on a single line.
df, md = wd.get_monitoring_locations(state_code="WI")

On an interactive terminal the one line is rewritten in place as each page arrives — these frames overwrite each other (\r), they don't stack:

Progress: 1 page · 10,000 rows · 4,998 requests remaining, resets in 52m
Progress: 2 pages · 20,000 rows · 4,997 requests remaining, resets in 52m
Progress: 3 pages · 24,213 rows · 4,996 requests remaining, resets in 51m

When a long CQL filter is split into URL-safe chunks, the line gains a chunk i/N segment:

Progress: chunk 3/5 · 11 pages · 38,400 rows · 4,990 requests remaining, resets in 49m

If no API key is configured (no API_USGS_PAT), the line appends a one-time pointer to API-key registration (unauthenticated callers hit much lower limits):

Progress: 3 pages · 24,213 rows
No API key detected — register for higher rate limits at https://api.waterdata.usgs.gov/signup/

The line is shown automatically when stderr is an interactive terminal. A source-documented API_USGS_PROGRESS env var (1/0) can force it on/off; per maintainer preference it's intentionally not advertised in the README — only in the module docstring.

What changed

• New dataretrieval/waterdata/_progress.py — a ProgressReporter (chunks / pages / rows / rate-limit), a progress_context() contextmanager, and a current() accessor. The active reporter lives in a ContextVar so the chunked decorator (chunk counts) and the page-walking loops (page / row / rate-limit counts) both update it without threading a parameter through every signature. On close, if no API_USGS_PAT is set, it prints a one-time API-key-registration pointer.
• utils.py — _walk_pages and get_stats_data report pages + rate-limit; get_ogc_data / get_stats_data wrap their fetch in progress_context(). _next_req_url is now a pure helper. Surviving request-URL logger.info calls drop to logger.debug. The "Geopandas not installed" advisory now logs at WARNING in both the OGC and stats paths (the stats path was previously silent INFO).
• chunking.py — ChunkedCall.resume() reports the total chunk count and per-chunk start (this branch was merged with main's chunking refactor, which moved fan-out out of filters.py into chunking.py; the page/row/rate hooks live in the shared _paginate helper).
• api.py — the two single-request CSV paths (get_samples, get_samples_summary) drop their Request: logger.info to debug.
• README.md — the README's "view request URLs and remaining requests at INFO" note is stale after this change (URLs moved to DEBUG, the hourly count is now in the progress line), so it's replaced with an accurate "Tracking progress" section (progress line + a DEBUG-logging pointer for the request URL; no env-var docs).

Logging vs. warnings

Only logger.info (the per-page narration) is replaced by the progress line. The logger.warning / logger.error failure messages stay as logging — Python's "last resort" handler already prints WARNING+ to stderr with no logging setup, so they're visible by default.

Behavior

• Progress line shown only on an interactive terminal by default, so notebooks, redirected logs, and CI stay clean.
• The API-key pointer is shown at most once per process, and never when API_USGS_PAT is set.
• Failure-path logger.warning / logger.error are unchanged.

Tests

• tests/waterdata_progress_test.py: rendering, no-op-when-disabled, plural/chunk formatting, rate-limit persistence, TTY + env gating, progress_context nesting, the API-key pointer (shown only when unauthenticated, at most once), and a _walk_pages integration check.
• Tests asserting both code paths warn when geopandas is missing.
• Full waterdata suite passes; ruff clean.

Open question for review

• On a failure mid-pagination, a logger.warning routed to stderr can land on the same line as the in-progress status text (cosmetic only). Worth a line-break-before-warning hook, or leave it?

🤖 Generated with Claude Code