[pawank925]

Fixes #165.

Adds a new usaiddec source (alias USAID) for the USAID DEC Archive preserved by AidData, as requested in openzim/zim-requests#1897 (permission granted).

• Discovers evaluation reports from the AidData dataset catalog (ZIP/CSV) and downloads report PDFs for offline reading.

• Supports report selection with --usaid-ids=<Formatted Document IDs> (alphanumeric) and catalog positions via --books.

• Adds --refresh-usaid-catalog (requires --cache-dir) and caches the extracted catalog CSV when --cache-dir is used.

• Registers the source profile (has_popularity=False), adds CLI/offliner definition/README/CHANGELOG updates, and adds a source icon SVG.

• Catalog/metadata are sourced from the AidData 2000–2024 USAID DEC evaluation reports dataset CSV (permanent PDF URLs).

• Formats supported by this source are currently pdf (reports are distributed as PDFs).

Tests

• Added test_usaiddec_catalog.py, test_usaiddec_metadata.py, test_usaiddec_pipeline.py, and updated test_source_registry.py.
• Verified locally: PYTHONPATH=src .venv/bin/pytest tests/test_source_registry.py tests/test_usaiddec_catalog.py tests/test_usaiddec_metadata.py tests/test_usaiddec_pipeline.py -v → 26 passed.

[codecov]

Codecov Report

❌ Patch coverage is 99.36909% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.24%. Comparing base (57f30b8) to head (a3c1892).
⚠️ Report is 12 commits behind head on main.

Files with missing lines	Patch %	Lines
scraper/src/papers2zim/sources/usaiddec/catalog.py	99.35%	0 Missing and 1 partial ⚠️
...craper/src/papers2zim/sources/usaiddec/pipeline.py	98.24%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #168      +/-   ##
==========================================
+ Coverage   77.83%   79.24%   +1.41%     
==========================================
  Files          68       74       +6     
  Lines        4502     4819     +317     
  Branches      725      777      +52     
==========================================
+ Hits         3504     3819     +315     
+ Misses        796      793       -3     
- Partials      202      207       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[pawank925]

@benoit74 ready for the review thanks!

[benoit74]

@pawank925 how many collections do we end-up with, are we in the expected 20-ish range ? We should customize icons obviously for each collection if possible.

[pawank925]

@benoit74 here's the answer on collections.

What we had before: exposing the dataset Class label plus thesaurus descriptors directly gave 1,651 collections for 4,522 reports — mostly near-duplicates (HIV/AIDS vs HIV / AIDS, Health vs Health care, Basic education vs Education (General)), so unusable in the UI.

What we do now: each report is folded into exactly one curated thematic sector using ordered keyword rules (first match wins, so specific sectors like HIV/AIDS are checked before generic Health). Result: 20 curated sectors + Other = 21 collections, right in the expected range.

Reports per collection (all 4,522 reports):

Collection	Reports
Agriculture & Food Security	724
Education	602
Economic Growth & Trade	440
Health	406
Governance & Public Administration	321
Maternal & Child Health	312
Democracy & Civil Society	258
Environment & Climate	231
HIV/AIDS	188
Conflict & Security	186
Water & Sanitation	128
Reproductive Health & Family Planning	128
Humanitarian Assistance & Disaster	127
Poverty & Social Development	117
Capacity Building & Evaluation	94
Gender Equality	76
Energy & Infrastructure	66
Technology & Communications	51
Nutrition	50
Population & Demography	9
Other	8

Icons: every collection has a dedicated MDI icon in ui/src/components/common/SubjectCollectionIcon.vue — sprout (agriculture), school (education), chart-line (economy), stethoscope (health), bank (governance), mother-heart (maternal health), vote (democracy), leaf (environment), ribbon (HIV/AIDS), shield-alert (conflict), water-pump (WASH), human-pregnant (reproductive health), hand-heart (humanitarian), home-heart (poverty), clipboard-check (capacity building), gender-female, lightning-bolt (energy), laptop (technology), food-apple (nutrition), account-multiple (population) — with mdiBookshelf as the fallback.

Taxonomy lives in scraper/src/papers2zim/sources/usaiddec/collections.py and is covered by test_usaiddec_collections.py. Happy to regroup differently (e.g. by region or by a shorter list) if you prefer another target set. Also: all commits are now authored and committed as pawank925, and CI is fully green on fe1a7c9.

[benoit74]

Automated code review (medium effort).