Adding transfer package with module docstring

[aaronjae22]

Closes #287

Original approach

This PR adds the transfer package and each of the file on it describe the intended purposes. This serves as a roadmap and blueprint for what we're about to implement.

I decided to do it this way so we can actually review or discuss everything ahead of this implementation.

I am planning on creating a data portability loop between source and destination so the Testbed will end up with three working modes:

Mode A: Source server
Mode B: Destination server
Mode C: Loop back between source and destination for data portability (which will work as two independent servers interacting with each other).

The idea with Mode C is to not take any shortcuts and respect the protocol between Source and Destination, both acting as indepent servers. By doing otherwise, I thought that would undermine the value of interaction between them.

---

The new core/transfer/ package is where the destination logic is going to live. Its worth mention that there are a couple of things that both Source and Destination will share but respecting their independence.

Each of the file in the package represents somehow the idea and plan that I want to follow. Probably a few things will change moving forward, a few things could be deleted and / added but this is the start of it. It could be possible that we could even merge a few files together.

• transport is the only place that is in charge on making networks calls
• discovery is responsable for asking the Source where its endpoints are
• auth is in charge on constructing authorization url and oauth flow from our side, and the access token
• fetch will wak collections a page at a time
• transfor rewrites each item so it will belong to us with a note of where it came from
• storage saves the results
• jobs tracks the progress and doing one small step at a time
• policies is kind of a place where we could add constraints, prohibitions and rules about how we should do or approach certains actions. This one could make a bit of a noise right now but it could be useful as I start implementing.
• schemas is a bit like policies for now but it is intended to enforce the shapes the modules hand to each other

There are a few things that came into my mind that probably are not necessary for now but at least having them in the package as a placeholder could be useful so we can state what we're planning to do next.

---

There are a few things worth mention about the approach to implement this:

In the idea of making Mode C a real interaction between the Source and Destination, every request will reach the source via HTTPS, and must not read settings for a URL that should have been discovered.

When an user creates an account in the Testbed we signal to create two Actors for it. The Source and Destion. The source actor is populated but destination is empty.

The destination transfer will use the existing models that we use for the source but a few things should be added or modified.

For example, after a data transfer is made on Mode C this is what we should get:

# After a Mode C transfer from alice_source → alice_dest
Note.objects.filter(actor__username="alice_source")   # 137 — the originals
Note.objects.filter(actor__username="alice_dest")     # 137 — the copies

Same tables and columns. The copy will differs from the original on its actor_id, a new object ID when served and a breadcrumb pointing to its previous home.

Also this is how we would identify the responses. Artifact is the record and the envelope would act as a shape.

┌─ ARTIFACT ─────────────────────────────────────────┐
│ POST/GET  https://source.example/…/content/?page=2  │  ← request line
│ Accept: application/activity+json                   │  ← headers
│ Authorization: Bearer <stripped>                    │
│                                                     │
│ HTTP/1.1 200 OK                                     │  ← status
│ Content-Type: application/activity+json             │  ← response headers
│ Retry-After: —                                      │
│                                                     │
│ ┌─ ENVELOPE ──────────────────────────────────┐    │
│ │ "type": "OrderedCollectionPage",            │    │  ← the wrapper…
│ │ "partOf": "…/content/",                     │    │
│ │ "next": "…/content/?page=3",                │    │
│ │ "totalItems": 137,                          │    │
│ │ "orderedItems": [ ─────────────────────┐    │    │
│ │       {note}, {note}, {note}, …         │ ← …and the items it wraps
│ │   ]  ◄─────────────────────────────────┘    │    │
│ └─────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────┘

There is also a approach to how we are not silently keeping what we do not recognize. LOLA §7.1.10: Unrecognized attributes in objects should probably not be kept.

The default is to drop unrecognised attributes from the saved object but I would like to retaining them in the artifact for progress and reports results and also to alert the user what was missing out of the data transfer. We could iterate over this later on and possible won't be part of the Testbed at first.

This is the order of event:

1. transport.py:  receives the response, stores the artifact — complete, verbatim, immutable
2. fetch.py:      parses items out of it
3. transform.py:  builds a new object from recognised fields only
4. storage.py:    writes that object to a Note row

So given a example of fetched note that has the following structure:

{
  "id": "https://source..../..../notes/42",
  "type": "Note",
  "content": "Testbed",
  "published": "2026-08-27T10:00:00Z",
  "quoteAuthorization": "https://source.example/auth/9",   ← we don't know this
  "x-source": "Source server"                              ← nor this
}

By the time transform runs, the full payload is already preserved. Transfrom will retain attributes in the artifact but simply does not persist them.

Changes made after suggestions

schemas.py and policies.py were deleted from the original transfer package structure.

Schemas docstring pretty much described a running job current state. Everything it named belongs somewhere that already exists. endpoints and the migrations URLs are job state and belong on TransferJob, the per-item result is already TransferredItem and the rest were values handed between fetch, transform, and storage.

Policies is also gone. It describe dry-run and duplicate handling as a module which could be set on a simpler approach (which is described in #290 after changes).

[lisad]

I understand you're thinking about loopback support, but this feels like a really minor point when most of the time the source actor isn't even on this server. I'm sure this will change when the code is filled in.

[lisad]

This doesn't sound like the kind of job a "schema" does. Is this more of an orchestrator? A "Context" object to hold the job's current info as it implies in the "Owns" section?

[aaronjae22]

Yes, docstring was actually describing a complete job running state which is already stores in the database amd we could work from there and not implement some kind of intermediary in between process.