Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 36 additions & 4 deletions docs/agent-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ fix (this PR) makes v2 reads accept **either** key: unknown ids fall back to a
carries the canonical v2 id. Prefer storing the **v2 id** for anything durable;
the alias exists so old references keep working.

Three more cutover artifacts worth knowing:
Four more cutover artifacts worth knowing:

- **Only the Google decoder emits `ConversationEvent` frames.** Signal and
WhatsApp conversations are minted from message frames, which carry no kind and
Expand Down Expand Up @@ -130,15 +130,47 @@ curl -s http://127.0.0.1:7007/api/status | jq '.freshness'
`UNIQUE constraint failed: devices.account_id`. Each call first rewrote the
account's `bridge_key` (`google_messages` → `google`), and v2 reads derive a
thread's platform from that key, so Google threads would then read as
`google` instead of `sms`. A v2-primary daemon no longer runs the mirror on
`/api/mark-read`. The keys should read `google_messages`, `whatsmeow` and
`signal_cli` (the live store was clean on 2026-10-09):
`google` instead of `sms`. A v2-primary daemon does not run the mirror on
`/api/mark-read`; it writes the cursor natively (next bullet). The keys
should read `google_messages`, `whatsmeow` and `signal_cli` (the live store
was clean on 2026-10-09):

```bash
sqlite3 -readonly "$HOME/Library/Application Support/OpenMessage/v2/store.sqlite3" \
"SELECT account_id, bridge_key, display_name FROM accounts"
```

- **On v2-primary, `/api/mark-read` writes the v2 read cursor natively**
(`v2wire.MarkReadV2`). It resolves `conversation_id` the way v2 reads do:
the v2 id, or a legacy-form id through `remote_conversation_id`
(`v2read.ResolveConversation`). An id that matches nothing writes nothing;
the mirror would have created the thread under its legacy id. The cursor
belongs to the account's local installation device, the one
`GetLocalInstallationDevice` resolves and ingest receipts advance, so on a
migrated store it is the derived-id device. An account with no local device
gets one under the migration's derived id
(`v2keys.LocalInstallationDeviceID`), never `local-primary:<account>`. The
cursor is dated at the request and names no message
(`last_read_message_id` NULL, as the mirror writes it). `read_cursors`
references `messages` with no `ON DELETE` action, so a cursor that named the
newest message would pin it. The outbox deletes a send's echo duplicate on
reconcile, and that delete then fails with `FOREIGN KEY constraint failed`.
Id-space repair likewise leaves cursor-referenced messages in place. Writes
are monotone in read time, so a cursor already dated later than the request
stays (an ingested receipt carries the receipt's own time). The write is best
effort and sends no read receipt to the phone. A failure logs `Failed to write v2 read cursor` with
`conv_id`, and the response stays 200. Nothing reads these cursors yet: v2
reads report `UnreadCount: 0`, and the web UI calls mark-read only when
`UnreadCount > 0`, so on v2-primary only API callers reach it until unread
derivation (S5b/S8) lands. To see the latest cursors:

```bash
sqlite3 -readonly "$HOME/Library/Application Support/OpenMessage/v2/store.sqlite3" \
"SELECT conversation_id, device_id, last_read_message_id,
datetime(last_read_at_ms / 1000, 'unixepoch', 'localtime')
FROM read_cursors ORDER BY last_read_at_ms DESC LIMIT 5"
```

## Google thread ids are device-local: phone swaps re-key everything

Google Messages conversation and message ids are the phone's own row ids, not
Expand Down
7 changes: 2 additions & 5 deletions internal/migration/transform.go
Original file line number Diff line number Diff line change
Expand Up @@ -615,7 +615,7 @@ func writeAccounts(target *sqlite.Store, state *transformState) error {
}); err != nil {
return fmt.Errorf("upsert %s account: %w", platform, err)
}
deviceID := v2keys.DeriveID("device", account.AccountID, account.AccountID+"\x1flocal")
deviceID := v2keys.LocalInstallationDeviceID(account.AccountID)
if err := target.UpsertDevice(sqlite.Device{
DeviceID: deviceID, AccountID: account.AccountID,
Kind: sqlite.DeviceKindLocalInstallation, DisplayName: "OpenMessage",
Expand Down Expand Up @@ -984,10 +984,7 @@ func writeReadCursors(
lastReadID = &value
lastReadAt = positions[index].At
}
deviceID := v2keys.DeriveID(
"device", conversation.Account.AccountID,
conversation.Account.AccountID+"\x1flocal",
)
deviceID := v2keys.LocalInstallationDeviceID(conversation.Account.AccountID)
updatedAt := state.baseTimestampMS
if len(positions) > 0 {
updatedAt = maxInt64(updatedAt, positions[len(positions)-1].At)
Expand Down
9 changes: 9 additions & 0 deletions internal/v2keys/derive.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,15 @@ func DeriveID(entity, accountID, naturalKey string) string {
return hex.EncodeToString(sum[:])[:32]
}

// LocalInstallationDeviceID is the ID the migration mints for an account's
// local installation device. Code that must create that device for an account
// that has none uses it too, so a v2 store holds one ID shape whichever path
// created the device. Code that reads the device resolves it by role
// (sqlite.Store.GetLocalInstallationDevice), never by this ID.
func LocalInstallationDeviceID(accountID string) string {
return DeriveID("device", accountID, accountID+"\x1flocal")
}

var signalACI = regexp.MustCompile(`(?i)^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$`)

// IdentityKey classifies and canonicalizes a platform identity.
Expand Down
17 changes: 17 additions & 0 deletions internal/v2keys/derive_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,23 @@ func TestDeriveIDMigrationFixtureGoldens(t *testing.T) {
}
}

// TestLocalInstallationDeviceIDMatchesMigrationGoldens pins the helper to the
// device IDs the migration minted before the helper existed, so a device the
// native mark-read path creates has the ID a migrated store would hold.
func TestLocalInstallationDeviceIDMatchesMigrationGoldens(t *testing.T) {
t.Parallel()

for accountID, want := range map[string]string{
"google-primary": "9bcc134365b6f21de496ec1693b68421",
"whatsapp-primary": "1aa97228aad2f8a40a7eea579625d579",
"signal-primary": "f5725a0b516450efeab65b0ccdc9d041",
} {
if got := LocalInstallationDeviceID(accountID); got != want {
t.Errorf("LocalInstallationDeviceID(%q) = %q, want migration golden %q", accountID, got, want)
}
}
}

func TestIdentityKey(t *testing.T) {
t.Parallel()

Expand Down
55 changes: 38 additions & 17 deletions internal/v2read/alias.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,44 +2,65 @@ package v2read

import (
"errors"
"fmt"
"strings"

"github.com/maxghenis/openmessage/internal/storage/sqlite"
"github.com/maxghenis/openmessage/internal/v2keys"
)

// resolveConversationID maps a caller-supplied conversation key onto the v2
// primary key. v2 conversation IDs pass through untouched. Any other value is
// tried as a remote conversation ID — the key the legacy store and every
// pre-cutover consumer used ("signal:+15551234567", "signal-group:…", a
// Google Messages thread id, a WhatsApp JID) — across all accounts. Cutover
// re-keys every conversation to a derived hash, so without this fallback each
// stored legacy ID silently reads as an empty conversation the moment a
// restart flips the app to v2-primary (issue #155).
//
// Unknown keys return unchanged so callers keep their existing
// primary key with ResolveConversation. Unknown keys, and keys whose lookup
// fails, return unchanged so callers keep their existing
// empty-result/not-found semantics.
func (s *Source) resolveConversationID(id string) string {
trimmed := strings.TrimSpace(id)
if trimmed == "" {
return id
}
if _, err := s.store.GetConversation(trimmed); err == nil {
return trimmed
} else if !errors.Is(err, sqlite.ErrNotFound) {
conversation, err := ResolveConversation(s.store, trimmed)
if err != nil {
return trimmed
}
accounts, err := s.store.ListAccounts()
return conversation.ConversationID
}

// ResolveConversation returns the v2 conversation a caller-supplied key names.
// A v2 conversation ID resolves to itself. Any other value is tried as a
// remote conversation ID — the key the legacy store and every pre-cutover
// consumer used ("signal:+15551234567", "signal-group:…", a Google Messages
// thread id, a WhatsApp JID) — across all accounts. Cutover re-keys every
// conversation to a derived hash, so without this fallback each stored legacy
// ID silently reads as an empty conversation the moment a restart flips the
// app to v2-primary (issue #155).
//
// It is the one alias rule for v2 reads and for writes that take a
// caller-supplied conversation ID (v2wire.MarkReadV2), so a stored legacy ID
// writes to the same thread it reads. A key that matches nothing returns an
// error wrapping sqlite.ErrNotFound.
func ResolveConversation(store *sqlite.Store, id string) (sqlite.Conversation, error) {
trimmed := strings.TrimSpace(id)
if trimmed == "" {
return sqlite.Conversation{}, fmt.Errorf("resolve conversation: empty id: %w", sqlite.ErrNotFound)
}
conversation, err := store.GetConversation(trimmed)
if err == nil {
return conversation, nil
}
if !errors.Is(err, sqlite.ErrNotFound) {
return sqlite.Conversation{}, fmt.Errorf("resolve conversation %q: %w", trimmed, err)
}
accounts, err := store.ListAccounts()
if err != nil {
return trimmed
return sqlite.Conversation{}, fmt.Errorf("resolve conversation %q: %w", trimmed, err)
}
var best *sqlite.Conversation
for _, account := range accounts {
remoteID := v2keys.NormalizeRemoteConversationID(
platformForBridgeKey(account.BridgeKey),
trimmed,
)
conversation, err := s.store.GetConversationByRemote(account.AccountID, remoteID)
conversation, err := store.GetConversationByRemote(account.AccountID, remoteID)
if err != nil {
continue
}
Expand All @@ -56,7 +77,7 @@ func (s *Source) resolveConversationID(id string) string {
}
}
if best != nil {
return best.ConversationID
return *best, nil
}
return trimmed
return sqlite.Conversation{}, fmt.Errorf("resolve conversation %q: %w", trimmed, sqlite.ErrNotFound)
}
167 changes: 167 additions & 0 deletions internal/v2read/alias_resolve_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
package v2read

import (
"database/sql"
"errors"
"math/rand"
"strings"
"testing"
"testing/quick"

"github.com/maxghenis/openmessage/internal/storage/sqlite"
"github.com/maxghenis/openmessage/internal/v2keys"
)

func TestResolveConversationReturnsCanonicalRowOrNotFound(t *testing.T) {
store, _, _ := openSourceTestStore(t)
seedSourceAccount(t, store, "signal-primary", "signal_cli")
const remoteID = "signal:+15550000002"
conversationID := v2keys.DeriveID("conversation", "signal-primary", remoteID)
seedSourceConversation(t, store, sqlite.Conversation{
ConversationID: conversationID,
AccountID: "signal-primary",
RemoteConversationID: remoteID,
Kind: sqlite.ConversationKindDirect,
NotificationMode: sqlite.NotificationModeAll,
LastMessageAtMS: 400,
})

for _, key := range []string{
conversationID,
" " + conversationID + "\n",
remoteID,
"\t" + remoteID + " ",
// Signal remote IDs normalize the address payload as they did at
// cutover.
"signal: +15550000002",
} {
conversation, err := ResolveConversation(store, key)
if err != nil {
t.Fatalf("ResolveConversation(%q): %v", key, err)
}
if conversation.ConversationID != conversationID || conversation.AccountID != "signal-primary" {
t.Fatalf("ResolveConversation(%q) = %+v, want %q", key, conversation, conversationID)
}
}
for _, key := range []string{"", " ", "signal:+15550009999", v2keys.DeriveID("conversation", "signal-primary", "absent")} {
if _, err := ResolveConversation(store, key); !errors.Is(err, sqlite.ErrNotFound) {
t.Fatalf("ResolveConversation(%q) error = %v, want ErrNotFound", key, err)
}
}
}

// TestResolveConversationMatchesReferenceAndReadPath checks, over random
// stores whose remote IDs collide across accounts and whose recency ties,
// that ResolveConversation agrees with a brute-force reading of the alias
// rule, and that the v2 read path (Source.GetConversation) serves the same
// conversation. The second check is what lets a write keyed by a
// caller-supplied ID land on the thread a read of that ID shows.
func TestResolveConversationMatchesReferenceAndReadPath(t *testing.T) {
accounts := []struct {
accountID string
bridgeKey string
}{
{"google-primary", "google_messages"},
{"signal-primary", "signal_cli"},
{"whatsapp-primary", "whatsmeow"},
}
remotePool := []string{
"thread-1",
"thread-2",
"signal:+15550000001",
"signal-group:QUJD=",
"15550000003@s.whatsapp.net",
}
property := func(seed int64) bool {
random := rand.New(rand.NewSource(seed))
store, _, source := openSourceTestStore(t)
var conversations []sqlite.Conversation
platformOf := map[string]string{}
for _, account := range accounts {
seedSourceAccount(t, store, account.accountID, account.bridgeKey)
platformOf[account.accountID] = platformForBridgeKey(account.bridgeKey)
for _, remoteID := range remotePool {
if random.Intn(2) == 0 {
continue
}
conversation := sqlite.Conversation{
ConversationID: v2keys.DeriveID("conversation", account.accountID, remoteID),
AccountID: account.accountID,
RemoteConversationID: remoteID,
Kind: sqlite.ConversationKindDirect,
NotificationMode: sqlite.NotificationModeAll,
// Few distinct values, so recency ties are common.
LastMessageAtMS: int64(random.Intn(3)) * 100,
}
seedSourceConversation(t, store, conversation)
conversations = append(conversations, conversation)
}
}

// reference is the alias rule written out over the seeded rows.
reference := func(key string) (string, bool) {
trimmed := strings.TrimSpace(key)
if trimmed == "" {
return "", false
}
for _, conversation := range conversations {
if conversation.ConversationID == trimmed {
return trimmed, true
}
}
var best *sqlite.Conversation
for index := range conversations {
conversation := &conversations[index]
if conversation.RemoteConversationID != v2keys.NormalizeRemoteConversationID(platformOf[conversation.AccountID], trimmed) {
continue
}
if best == nil || conversation.LastMessageAtMS > best.LastMessageAtMS ||
(conversation.LastMessageAtMS == best.LastMessageAtMS && conversation.ConversationID < best.ConversationID) {
best = conversation
}
}
if best == nil {
return "", false
}
return best.ConversationID, true
}

keys := []string{"", " ", "unknown-thread", "signal: +15550000001", "signal-group: QUJD="}
for _, remoteID := range remotePool {
keys = append(keys, remoteID, " "+remoteID+"\t")
}
for _, account := range accounts {
for _, remoteID := range remotePool {
keys = append(keys, v2keys.DeriveID("conversation", account.accountID, remoteID))
}
}
for _, key := range keys {
wantID, wantFound := reference(key)
resolved, err := ResolveConversation(store, key)
if wantFound != (err == nil) || (err != nil && !errors.Is(err, sqlite.ErrNotFound)) {
t.Errorf("seed %d: ResolveConversation(%q) error = %v, want found=%v", seed, key, err, wantFound)
return false
}
if wantFound && resolved.ConversationID != wantID {
t.Errorf("seed %d: ResolveConversation(%q) = %q, reference %q", seed, key, resolved.ConversationID, wantID)
return false
}
read, err := source.GetConversation(key)
if !wantFound {
if !errors.Is(err, sql.ErrNoRows) {
t.Errorf("seed %d: GetConversation(%q) = %+v, %v; want sql.ErrNoRows", seed, key, read, err)
return false
}
continue
}
if err != nil || read.ConversationID != wantID {
t.Errorf("seed %d: GetConversation(%q) = %+v, %v; want %q", seed, key, read, err, wantID)
return false
}
}
return true
}
if err := quick.Check(property, &quick.Config{MaxCount: 40, Rand: rand.New(rand.NewSource(20261010))}); err != nil {
t.Fatal(err)
}
}
Loading
Loading