Skip to content

pad-format.md

Latest commit

 

History

History
147 lines (114 loc) · 8.89 KB

pad-format.md

File metadata and controls

147 lines (114 loc) · 8.89 KB
Raw

.pad Format v1

SPDX-License-Identifier: AGPL-3.0-or-later

Overview

File format: etherpad-nextcloud/1

The .pad file consists of:

  1. YAML frontmatter (metadata)
  2. Snapshot body with text and optional HTML

Frontmatter Schema

Required fields:

  • format
  • file_id
  • pad_id
  • access_mode (public|protected)
  • state (active; legacy files may still contain trashed or purged)
  • created_at (ISO8601)
  • updated_at (ISO8601)
  • snapshot_rev (int, -1 before first successful sync)

Additional fields:

  • deleted_at (null or ISO8601)
  • pad_url (optional, absolute http(s) URL)
  • pad_origin (optional, origin of external Etherpad server, e.g. https://pad.example.org)
  • remote_pad_id (optional, actual pad ID on external server)

Example:

---
format: "etherpad-nextcloud/1"
file_id: 994
pad_id: "g.TmDeyA334sIq2LQh$p-4k9x2m7q8r1t5v6n3d0c"
access_mode: "protected"
state: "active"
deleted_at: null
created_at: "2026-03-05T00:40:36+00:00"
updated_at: "2026-03-05T11:10:21+00:00"
snapshot_rev: 42
---

Legacy migration:

  • Old Ownpad format
    • [InternetShortcut]
    • URL=https://.../p/<pad-id>
    • is auto-migrated on first open to etherpad-nextcloud/1. The migration branches on the URL origin (same vs. cross) and the pad-id format; GroupPad IDs (g.<group>$<name>) re-bind as protected, free-form IDs re-bind as public, cross-origin URLs route through the external-pad flow as ext.*.
    • A claim-collision check protects against legacy files being used to claim pads already bound to another user's file — see docs/legacy-ownpad-migration.md for the full state table.

Snapshot Body

Body layout:

[TEXT]
<plain text snapshot>
[HTML-BEGIN]
<html snapshot>
[HTML-END]

Notes:

  • Text is the primary restore snapshot.
  • HTML is an additional structure/format snapshot.
  • External pads (pad_origin + remote_pad_id) are imported and synced as text only for security reasons; their HTML section is written empty rather than omitted.
  • Every stored snapshot writes both sections. (A document that has never synced carries snapshot_rev: -1 and an empty body - no sections at all.) A snapshot with no HTML half - an external pad, whose HTML is deliberately never fetched - gets [HTML-BEGIN] / [HTML-END] with nothing between them. That is what lets the markers say where a half ends: a body always ends in the terminator and always holds an opening marker, so the last opening marker before the terminator is always the structural one, whatever the pad's own text contains.
  • The split relies on the HTML half carrying no [HTML-BEGIN] on a line of its own. Etherpad assembles its export without line separators, so nothing the app fetches produces that shape. This is a property of the export rather than one the format enforces: a hand-written HTML half holding that line takes the split with it, the same way a hand-edited file can break anything else here. Escaping it was tried and dropped - an escape a reader must undo cannot be told apart from content that was already indented, so it would need a format version to be safe, which is the thing this design avoids.
  • The pad's text may contain either marker freely, on as many lines as it likes. It is only ever the last occurrence that is structure.
  • A body written before this - a text-only snapshot with no section at all - is read as all text. That reading stays ambiguous for the one shape it always was: a text ending in [HTML-END]. Nothing the app writes produces that shape any more.
  • Viewer/API responses never expose stored HTML. The read-only viewer is served by LivePadHtmlFetcher, which fetches the pad's current HTML and runs SnapshotHtmlSanitizer over it. That sanitizer allowlists simple formatting tags and drops every attribute except href on <a>, which survives only for http, https and mailto; the browser applies the same allowlist again before the HTML is injected.

Mode Variants

  • Internal + Protected
    • access_mode: protected
    • pad_id: GroupPad (g.<group>$<name>)
    • pad_url: internal Etherpad URL
  • Internal + Public
    • access_mode: public
    • pad_id: public pad ID (for example nc-...)
    • pad_url: internal Etherpad URL
  • External + Public
    • access_mode: public
    • pad_id: external marker (ext.<remote_pad_id>)
    • pad_origin + remote_pad_id are set
    • pad_url: external URL used for viewer open
    • no row in ep_pad_bindings; the .pad frontmatter is the source of truth for the remote target

Protected + external is not supported.

Lifecycle State Semantics

  • active
    • normal editing state
  • trashed / purged
    • legacy parser compatibility only; new writes do not use these states

The DB binding table uses active, and pending_delete for a file deleted for good whose pad has yet to go. A trash keeps the binding row and the pad; a file restored without a row, or whose pad Etherpad lost, gets a new pad from the .pad frontmatter and snapshot. External pads are not managed in the DB binding table, so trash/restore only moves the Nextcloud file and never creates, deletes, or restores anything on the remote Etherpad server.

Parsing/Serializing

Implementation: lib/Service/PadFileService.php

  • parsePadFile(string $content): array{frontmatter, body}
  • serialize(array $frontmatter, string $body): string
  • readPad(string $content): ParsedPadFile parses once and hands back the frontmatter, the body and the fields derived from them
  • withExportSnapshot(ParsedPadFile $pad, PadSnapshot $snapshot) updates export metadata + snapshot body
  • withRestoredSnapshot(ParsedPadFile $pad, ...) writes the document a restore leaves behind: active, undeleted, pointed at the replacement pad, both snapshot halves as given. snapshot_rev is the replacement's own count once the snapshot is in it, -1 when unknown, never the old pad's
  • getSnapshotPartsFromBody(string $body): array{text, html} splits a stored snapshot into its two halves, from a body a caller already has
  • buildInitialDocument(...) takes an optional PadSnapshot for a document that starts out with content; without one the document is unsnapshotted (snapshot_rev: -1) and its body is empty
  • PadSnapshot is text, an HTML half and the revision the snapshot was taken at. Its html is a string, empty where a pad has none: every stored snapshot writes both sections either way. A negative revision is refused — -1 is what an unsnapshotted document uses

Frontmatter values are held to what the format can carry back: a value containing a line terminator (\n, \r) or a NUL byte is refused with PadFileFormatException rather than written. The block is line-based, so a value with a newline in it would parse back as a further key on the next read, and a .pad that says something different after a round trip is not a .pad. The check runs on write, where the whole value is still in hand, and on read against each frontmatter line before anything is matched or trimmed - early enough that a \r a key pattern would swallow as whitespace, or one trim() would drop from an end, is refused rather than silently removed. A CRLF line ending is normalised while the document is split and never reaches it. The read side cannot see a newline inside a value at all, because the block is split into lines first: a hand-edited file that breaks a value across two lines is read as two keys, not as one broken value, and the write-side refusal is what keeps the app from ever producing one.

Snapshot write flow:

  • PadFileService::withExportSnapshot(...) builds the new .pad content after an Etherpad export.
  • PadFileLockRetryService::putContentWithSyncLockRetry(...) writes that content back to the Nextcloud file with bounded lock retry.
  • Stored snapshots are read by RestoreService when restoring a pad, by the forced sync when comparing content, and by an open that may write and the sync to tell whether Etherpad has lost the pad. No viewer path shows them.
  • External public pad create/sync paths both use the validated, host-pinned /export/txt fetch internally (via ExternalPadExportFetcher) and store no HTML snapshot:
    • create uses ExternalPadExportFetcher::normalizeAndFetchExternalPublicPadTextOrEmpty(...), allowing the .pad file to be created with an empty initial snapshot if the export is not available yet.
    • sync uses ExternalPadExportFetcher::normalizeAndFetchExternalPublicPadText(...), keeping later export failures visible.

Sync Semantics

  • Sync writes only when the upstream snapshot actually differs.
  • force=1 requests an immediate upstream re-check, but unchanged snapshots are still not rewritten.
  • A pad Etherpad made anew in place of the file's - without a single revision, with other text than the file saved - is not written, forced or not: the sync answers pad_missing, and the file keeps its content for a new pad.
  • On successful sync:
    • snapshot_rev is updated
    • body is replaced:
      • internal pads: current text + HTML
      • external pads: current text, no HTML