Add USAID DEC Archive source

CHANGELOG.md

@@ -10,6 +10,7 @@ as of 2.0.0.
 
 ### Added
 
+- Add USAID DEC Archive source (`usaiddec`, alias `USAID`) for AidData's archive of USAID evaluation reports, with PDF downloads and catalog selection by Formatted Document ID (#165)
 - Add author biographies and portraits from Wikipedia with `--with-author-details` (#31)
 - Add support for sources without popularity: the source declares it, `config.json` carries the flag, and the UI hides flames and the most popular book and falls back to alphabetical sorting (#14)
 - Wikisource: add the library mission to the about page (#18)

README.md

@@ -1,6 +1,6 @@
 # Papers Offline
 
-This multi-source scraper downloads content from [Project Gutenberg](https://www.gutenberg.org), the [Open Textbook Library](https://open.umn.edu/opentextbooks/) and [Wikisource](https://wikisource.org) and packages it into a [ZIM](https://openzim.org) file, a clean and user-friendly format for storing content for offline usage.
+This multi-source scraper downloads content from [Project Gutenberg](https://www.gutenberg.org), the [Open Textbook Library](https://open.umn.edu/opentextbooks/), [Wikisource](https://wikisource.org), [One More Library](https://onemorelibrary.com), and the [USAID DEC Archive](https://usaid-archive.aiddata.org/) and packages it into a [ZIM](https://openzim.org) file, a clean and user-friendly format for storing content for offline usage.
 
 The ZIM file includes a modern, responsive Vue.js interface with features like:
 - Browse books by title, author, or source-specific categories such as Library of Congress Classification (LCC) shelves or Open Textbook Library subjects
@@ -66,8 +66,8 @@ Note: You'll need to install system dependencies (zim-tools) separately. See [CO
 The scraper supports several sources, select a source with `--source`:
 
 `--source=<source>`    Source slug or short name:
-                     gutenberg (PG), onemorelibrary (OML), opentextbooks (OTL)
-                     or wikisource (WS)
+                     gutenberg (PG), onemorelibrary (OML), opentextbooks (OTL),
+                     usaiddec (USAID) or wikisource (WS)
                      Default: gutenberg
 
 ### Project Gutenberg
@@ -128,6 +128,36 @@ The One More Library-specific options are:
 `--oml-ids` and `--books` cannot be used together. `--refresh-oml-catalog` requires `--cache-dir`.
 Selecting a whole library crawls the entire catalog; use `--cache-dir` to cache it across runs.
 
+### USAID DEC Archive
+
+Select the USAID DEC Archive with `--source=usaiddec` (or `--source=USAID`).
+
+The source packages [USAID's Development Experience Clearinghouse](https://usaid-archive.aiddata.org/)
+evaluation reports, preserved and published by [AidData](https://www.aiddata.org/) with
+permission, from the AidData dataset catalog.
+
+```bash
+docker run -v $(pwd)/output:/output ghcr.io/openzim/papers \
+  papers2zim --source=usaiddec -f pdf
+```
+
+To select specific reports by Formatted Document ID:
+
+```bash
+docker run -v $(pwd)/output:/output ghcr.io/openzim/papers \
+  papers2zim --source=usaiddec --usaid-ids=PA00JMRT,PA00ZH1K
+```
+
+The USAID DEC Archive-specific options are:
+
+```text
+--usaid-ids=<ids>          Exact USAID DEC report IDs (Formatted Document IDs)
+--refresh-usaid-catalog    Refresh the USAID DEC dataset catalog and exit
+```
+
+`--usaid-ids` and `--books` cannot be used together. `--refresh-usaid-catalog`
+requires `--cache-dir`.
+
 ### Open Textbook Library
 
 Select the Open Textbook Library with `--source=opentextbooks` (or `--source=OTL`).
@@ -187,7 +217,8 @@ Spaces and underscores are interchangeable, and percent-encoded names are accept
 ```text
 --source=<source>                  Source slug or short name:
                                    gutenberg (PG), onemorelibrary (OML),
-                                   opentextbooks (OTL) or wikisource (WS)
+                                   opentextbooks (OTL), usaiddec (USAID)
+                                   or wikisource (WS)
                                    Default: gutenberg
 ```
 
@@ -247,6 +278,13 @@ Spaces and underscores are interchangeable, and percent-encoded names are accept
 --wikisource-pages=<pages>          Wikisource page names, as in the page URL
 ```
 
+### USAID DEC Archive Options
+
+```text
+--usaid-ids=<ids>                   Exact USAID DEC report IDs (Formatted Document IDs)
+--refresh-usaid-catalog             Refresh the USAID DEC dataset catalog and exit
+```
+
 ### Caching
 
 Caching is opt-in. Pass `--cache-dir` to persist source metadata and catalog

offliner-definition.json

@@ -40,6 +40,13 @@
           "dependents": [
             "wikisource_pages"
           ]
+        },
+        {
+          "title": "USAID DEC Archive",
+          "value": "usaiddec",
+          "dependents": [
+            "usaid_ids"
+          ]
         }
       ]
     },
@@ -202,6 +209,12 @@
       "title": "Page names (Wikisource only)",
       "description": "Comma-separated Wikisource page names, as in the page URL (e.g. The_Time_Machine). Write a comma inside a name as %2C. Cannot be combined with Books."
     },
+    "usaid_ids": {
+      "type": "string",
+      "required": false,
+      "title": "Report IDs (USAID DEC Archive only)",
+      "description": "Comma-separated exact USAID DEC report IDs (Formatted Document IDs). Cannot be combined with Books."
+    },
     "with_author_bio": {
       "type": "boolean",
       "required": false,

scraper/src/papers2zim/sources/registry.py

@@ -33,6 +33,13 @@
 from papers2zim.sources.opentextbooks.catalog import OpenTextbookLibraryCatalog
 from papers2zim.sources.opentextbooks.metadata import OpenTextbookLibraryMetadata
 from papers2zim.sources.opentextbooks.pipeline import OpenTextbookLibraryPipeline
+from papers2zim.sources.usaiddec import cli as usaiddec_cli
+from papers2zim.sources.usaiddec.catalog import (
+    DEFAULT_CATALOG_URL as USAIDDEC_CATALOG_URL,
+)
+from papers2zim.sources.usaiddec.catalog import UsaidDecCatalog
+from papers2zim.sources.usaiddec.metadata import UsaidDecMetadata
+from papers2zim.sources.usaiddec.pipeline import UsaidDecPipeline
 from papers2zim.sources.wikisource import cli as wikisource_cli
 from papers2zim.sources.wikisource.catalog import WikisourceCatalog
 from papers2zim.sources.wikisource.metadata import WikisourceMetadata
@@ -176,6 +183,35 @@ class SourceProfile:
     pipeline_class=OnemorelibraryPipeline,
 )
 
+USAIDDEC_PROFILE = SourceProfile(
+    slug="usaiddec",
+    aliases=("USAID",),
+    locale_namespace="usaiddec",
+    display_name="USAID DEC Archive",
+    source_creator="aiddata.org",
+    zim_tags="_category:usaiddec;usaiddec",
+    zim_name_prefix="usaiddec",
+    icon_svg="usaiddec.svg",
+    tagline="USAID evaluation reports preserved by AidData",
+    source_description=(
+        "USAID's Development Experience Clearinghouse evaluation reports, "
+        "archived by AidData."
+    ),
+    collection_label="Subjects",
+    collection_icon_style="subject",
+    has_popularity=False,
+    default_mirror_url=USAIDDEC_CATALOG_URL,
+    catalog_feed_path="",
+    catalog=UsaidDecCatalog,
+    cli_options=usaiddec_cli.CLI_OPTIONS,
+    parse_cli_options=usaiddec_cli.parse_options,
+    handle_cli_action=usaiddec_cli.handle_cli_action,
+    pipeline_options=lambda _mirror_url, _cache_dir, _source_options, _languages: {},
+    metadata_options=lambda _mirror_url, _cache_dir, _engine: {},
+    metadata_class=UsaidDecMetadata,
+    pipeline_class=UsaidDecPipeline,
+)
+
 WIKISOURCE_PROFILE = SourceProfile(
     slug="wikisource",
     aliases=("WS",),
@@ -281,3 +317,4 @@ def get_source(slug: str) -> SourceProfile:
 register_source(OPEN_TEXTBOOK_LIBRARY_PROFILE)
 register_source(WIKISOURCE_PROFILE)
 register_source(ONEMORELIBRARY_PROFILE)
+register_source(USAIDDEC_PROFILE)

scraper/src/papers2zim/sources/usaiddec/__init__.py

@@ -0,0 +1,7 @@
+"""USAID DEC Archive source.
+
+Implements the USAID Development Experience Clearinghouse (DEC) archive
+source: a curated collection of USAID-funded evaluation reports made
+available by AidData. Catalog metadata and PDF download links come from
+the AidData dataset CSV.
+"""