Adding transfer package with module docstring

testbed/core/transfer/__init__.py

@@ -0,0 +1,20 @@
+"""
+The destination half of the testbed: everything this server does when it acts as a LOLA *destination*, pulling an account from a source server.
+
+Everything under `testbed/core/` outside this package is the source server. This package is its client.
+
+Module ownership:
+
+    transport.py   HTTP client, scheme policy, retry and backoff, 429 handling
+    discovery.py   RFC8414 metadata, public Actor, authenticated Actor, migration URL resolution
+    auth.py        Authorization URL construction, state, callback parsing, token exchange
+    fetch.py       Collection walking, pagination traversal, raw artifact capture
+    transform.py   ID generation, breadcrumbs, metadata preservation, wrapper activities
+    storage.py     Persisting transformed objects to destination models
+    jobs.py        Job lifecycle, state transitions, incremental advance
+
+`Actor` already carries `ROLE_DESTINATION`, and a destination Actor is created for every user at signup.
+This package is what will eventually give them behaviour.
+"""
+
+__all__ = []

testbed/core/transfer/auth.py

@@ -0,0 +1,17 @@
+"""
+The OAuth authorization process, seen from the destination side.
+
+Owns
+    - Building the authorization URL (`client_id`, `redirect_uri`, `scope`, `state`)
+    - Generating and validating `state`
+    - Parsing the callback's `code`, `state` and `activitypub_actor`
+    - The token exchange
+    - And for Mode B (serve as a Destination Server only), the per-source-server client credentials this
+      server holds as a registered OAuth client on that source server.
+
+Must not
+    - Prefer the Actor ID the user typed over the `activitypub_actor` returned with the
+      authorization code. LOLA §5.3: the destination "MUST use the Actor ID provided with the
+      authorization code rather than the original one the user communicated.".
+    - Accept a callback whose `state` does not match the one issued for that job
+"""

testbed/core/transfer/discovery.py

@@ -0,0 +1,16 @@
+"""
+Turning a source server into a set of URLs, by asking it rather than assuming.
+
+Owns
+    - Fetching `/.well-known/oauth-authorization-server` (RFC 8414)
+    - Reading the authorization and token endpoints plus the advertised scopes
+    - Fetching the public Actor and reading `endpoints.oauthMigrationEndpoint`
+    - Re-fetching the Actor with the token
+    - And resolving the scope-gated `migration` object into the four collection URLs (outbox, content, following, blocked)
+    - Also owns starting from either a base URL or an Actor URL, since a user may supply either
+
+Must not
+    - Read `setttings` for an URL that should have been discovered.
+      In Mode C, this server already knows its own endpoints, so constructing one instead of fetching it
+      silently deletes the discovery half of the protocol.
+"""

testbed/core/transfer/fetch.py

@@ -0,0 +1,22 @@
+"""
+Reading collections off a source server, one page at a time.
+
+Owns
+    - Walking a collection from its `first` page through each `next` link until the links run out
+    - Reading `orderedItems` (and the collection envelope generally) without assuming a page shape
+    - Capturing each raw response as an artifact before anything interprets it
+    - Counting items and pages for progress reporting
+    - Reporting where a walk stopped so it can resume from that exact page
+
+Must not
+    - Import `testbed.core.models`. In loopback, the rows live in the same database as the
+      fetching process, so a direct read is trivially available yet invisible in review and
+      it silently drops the pagination, token, and envelope contracts in one line.
+      Everything this module reads must arrive over HTTP via `transport.py`.
+    - Treat an out-of-range page as an error. Terminates cleanly rather than mistaking the
+      end of a collection for a failure.
+    - Decide what a `429` means. The typed exception comes from `transport.py`; pausing the job and
+      resuming it belongs to `jobs.py`.
+    - Transform or save anything. What comes off the wire is recorded as fetched; rewriting is
+      `transform.py`'s job and persisting is `storage.py`'s.
+"""

testbed/core/transfer/jobs.py

@@ -0,0 +1,18 @@
+"""
+The lifecycle of one transfer: what stage it is in, and what the next bounded step is.
+
+Owns
+    - Job state and its transitions (pending, discovering, authorizing, fetching, paused,
+      dry_run_complete, committing, completed, failed).
+    - `advance()`, which performs exactly one bounded unit of work and returns
+    - resumption state, so a job paused on a `429` restarts at the page it stopped on rather than
+      at the beginning; and the error and progress fields the status page reads.
+
+Must not
+    - Walk a whole collection inside one web request. `Dockerfile` runs `--workers 1 --threads 8`, so a Mode C
+      self-call occupies one thread while waiting on another thread of the same service, and a multi-page fetch
+      inside one request also risks Cloud Run's request timeout. Both failure modes are hangs rather than errors,
+      which is why the unit of work is one collection page and the caller comes back for the next one.
+    - Make HTTP calls, parse payloads, or transform objects. It sequences the modules that do.
+    - Advance past a failure silently. A failed step is a recorded state, visible on the status page.
+"""

testbed/core/transfer/storage.py

@@ -0,0 +1,15 @@
+"""
+Writing transformed objects into this server's own tables.
+
+Owns
+    - Persisting a transformed object against the destination Actor
+    - Recording the source-to-destination ID mapping that the result report is built from
+    - Detecting an object already imported by an earlier job
+    - Reporting a per-item outcome (imported, skipped, duplicate, failed) rather than raising past the first failure
+
+Must not
+    - Read source-actor rows. This module may import `testbed.core.models` to write the destination
+      side; anything belonging to the source must arrive over HTTP, as a stranger's would
+    - Transform. What arrives here is already final
+    - Abort the whole transfer on one bad item. A single object that will not save is an item outcome, not a job failure
+"""

testbed/core/transfer/transform.py

@@ -0,0 +1,22 @@
+"""
+Rewriting a fetched object into the one this server will publish.
+
+Owns
+    - Generating a new object ID for every copied object
+    - Setting the actor ID to the destination Actor for the standard account-migration case
+    - Adding and preserving `previously` breadcrumbs
+    - Preserving metadata that LOLA §7 names (`published`, `to`, `inReplyTo`, likes, replies)
+    - Building the wrapper activity for the destination outbox
+    - Applying the decided rule for attributes it does not recognise
+
+Must not
+    - Reuse a source object's ID. LOLA §7.1.1: the destination "MUST create or choose a new
+      context-appropriate Object ID for each object … since it is making a copy of an object that
+      still exists elsewhere it can't use the exact same object ID.".
+    - Drop breadcrumbs. §7.1.5 is a SHOULD, both for adding one and for preserving those
+      already present, newest first.
+    - Silently keep everything it does not recognise. §7.1.10 leaves this open. The default is to
+      drop unrecognised attributes from the saved object while retaining them in the artifact.
+    - Import `testbed.core.models` or touch the database. It takes fetched shapes and returns
+      transformed ones; persistence is `storage.py`'s.
+"""

testbed/core/transfer/transport.py

@@ -0,0 +1,19 @@
+"""
+The one HTTP client every destination module uses to reach a source server.
+
+Owns
+    - The requests session and its configuration the scheme policy
+    - The `Authorization: Bearer` header
+    - `Accept` negotiation for ActivityPub content types
+    - Timeouts on every call; retry and backoff
+    - Detection of `429` and parsing of `Retry-After` into a typed exception
+    - The descriptive `User-Agent` that lets a source operator see who is fetching
+    - And capture of every exchange (method, URL, status, headers, body) as a replayable artifact
+
+Must not be bypassed. LOLA §6.1 makes two MUST claims:
+    - "The destination MUST fetch data using HTTPS, not HTTP" and "It MUST provide the account
+      migration authorization token in requests even when it believes the requests are for public content",
+      and both are auditable only while there is exactly one exit point.
+    - Relax the scheme policy beyond loopback. HTTPS is required unless the host resolves to
+      loopback and the environment permits it (true in development, test and CI; false in staging and production).
+"""