Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs/agent-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,27 @@ are re-linked by peer when their thread shows up under its new id. Watch
in `/api/status` — a burst right after a re-pair is the fix working, not a
fault.

A roster match is not proof of a re-key: two groups with the same members, or
two 1:1 threads with one person, are both live under their own ids. Until
2026-10-08 ingest moved one's id onto the other, so the id and title flipped on
every event and both threads' messages piled into one row. A thread now takes a
new id only when its own id is dead: displaced by a collision, or announced by
a ConversationEvent only in an earlier device ID space. ID spaces are counted
per account (`accounts.remote_idspace_epoch`); a new one starts when an id the
phone announced in the current space arrives naming a different thread. Group
re-keys also need the title to match. A thread whose first messages arrive
with no ConversationEvent is provisional; a second sender makes it a group, so
it is not scattered across its members' 1:1 threads. A reset shows up as the
first collision, and fresh ids that arrive before it (14 threads in the 36
seconds before the first collision on 9/3) first get their own rows. When the
reset is detected, rows minted that way in the previous 15 minutes are merged
into the threads they continue. Google also reuses message ids across a reset; when a
continued thread receives a new message under an old message's id, the old
message keeps its row under `displaced:<id>:<message_id>`. Counters:
`idspace_resets` (expect one per phone swap or restore), `rekeys_declined`
(a thread kept apart from a live same-roster twin; steady trickle),
`rekeys_recovered`, `remote_message_ids_retired`.

**Repairing history after a reset** (daemon must be down — the command takes
the instance lock and refuses while `/api/status` answers; park the watchdog
first):
Expand Down
27 changes: 27 additions & 0 deletions internal/ingest/counters.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,23 @@ type CounterSnapshot struct {
// dropped because identical content already existed under another remote id.
RemoteRebinds uint64 `json:"remote_rebinds"`
ContentDupesSkipped uint64 `json:"content_dupes_skipped"`

// IDSpaceResets counts Google device ID-space resets detected: a wire id
// the phone had announced for one thread in the current ID space arrived
// naming a different one. RekeysDeclined counts frames whose roster matched
// only threads still live under their own ids (two groups with one
// membership, two 1:1 threads with one person), so the frame's id got its
// own thread instead of taking theirs. RekeysRecovered counts threads kept
// apart that way in a reset's opening minutes, before the reset was
// detected, and then merged into the thread they continue.
// RemoteMessageIDsRetired counts older messages whose Google message ID a
// different message reused in the same thread across an ID-space reset;
// each keeps its row under a displaced remote ID instead of being
// overwritten.
IDSpaceResets uint64 `json:"idspace_resets"`
RekeysDeclined uint64 `json:"rekeys_declined"`
RekeysRecovered uint64 `json:"rekeys_recovered"`
RemoteMessageIDsRetired uint64 `json:"remote_message_ids_retired"`
}

type accountCounters struct {
Expand Down Expand Up @@ -65,6 +82,11 @@ type accountCounters struct {
ephemeral atomic.Uint64
remoteRebinds atomic.Uint64
contentDupesSkipped atomic.Uint64

idspaceResets atomic.Uint64
rekeysDeclined atomic.Uint64
rekeysRecovered atomic.Uint64
remoteMessageIDsRetired atomic.Uint64
}

// Counters owns atomic ingest counters partitioned by account. Its zero value
Expand Down Expand Up @@ -149,5 +171,10 @@ func snapshotCounters(c *accountCounters) CounterSnapshot {
Ephemeral: c.ephemeral.Load(),
RemoteRebinds: c.remoteRebinds.Load(),
ContentDupesSkipped: c.contentDupesSkipped.Load(),

IDSpaceResets: c.idspaceResets.Load(),
RekeysDeclined: c.rekeysDeclined.Load(),
RekeysRecovered: c.rekeysRecovered.Load(),
RemoteMessageIDsRetired: c.remoteMessageIDsRetired.Load(),
}
}
Loading
Loading