Realtime push (IMAP IDLE → SSE) + delta-sync latency fixes#8
Open
Aravinda-HWK wants to merge 2 commits into
Open
Realtime push (IMAP IDLE → SSE) + delta-sync latency fixes#8Aravinda-HWK wants to merge 2 commits into
Aravinda-HWK wants to merge 2 commits into
Conversation
Implements Phase 3 of the Email 2.0 proposal: refresh now fetches only
what changed instead of re-listing the whole folder.
Because go-imap v1 exposes no CONDSTORE/MODSEQ, the delta is computed from
UIDs + flags rather than a MODSEQ token:
Server
- New GET /api/v1/mailboxes/{mailbox}/changes endpoint returning a
MailboxDelta {uidvalidity, total, resync, added, flags, removed}.
- imap.MailboxChanges: UID SEARCH (since+1):* for new envelopes, a
flags-only UID FETCH over the client's known UIDs to derive flag
changes + removals; UIDVALIDITY mismatch signals a full resync.
- ListMessages now also returns uidvalidity so the client has a baseline.
- Factored shared fetchEnvelopes/fetchFlags helpers.
Client
- Dexie v2 syncMeta table persists uidvalidity per folder.
- mailboxes.changes() endpoint + MailboxDelta/FlagUpdate types.
- DataContext reconciles added/removed/flag changes idempotently (keyed
on UID) and exposes refreshFolder(role); a ThreadList refresh button
wires it into Inbox/Sent/Trash.
Performance
- refreshFolder syncs only the viewed folder, so four folder syncs no
longer serialize on the session's single IMAP connection.
- IMAPFor drops a redundant per-request NOOP probe (one RTT) since every
operation already self-heals via ensureLive.
Tests: parseUIDList unit tests; server build + go test and frontend
typecheck pass.
Phase 4 of the proposal: new mail appears without a manual refresh. Realtime pipeline: - imap.Client.Watch: run IMAP IDLE on a dedicated connection, forwarding mailbox/message/expunge updates via a callback (go-imap handles the capability check, periodic restart, and polling fallback). - realtime.Hub: per-session fan-out to SSE subscribers with ref-counted IDLE watchers (N tabs on one folder share one connection) and capped backoff reconnect. - GET /api/v1/events SSE handler: clears the write deadline so the stream isn't killed by WriteTimeout; 25s heartbeat that also keeps the session warm. - RequireSession also accepts the JWT as an access_token query param, since the browser EventSource API cannot set an Authorization header. - Client opens one EventSource watching the inbox and runs a delta sync on each push; a green "Live" indicator shows the stream is connected. Latency / correctness fixes surfaced during end-to-end testing on Gmail: - selectMailbox now tracks read-only vs read-write mode and re-SELECTs when a write follows a read-only SELECT, fixing a "STORE on READ-ONLY folder" 502 storm on mark-as-read. - Keep the session's IMAP connection warm (keepalive NOOP on each sweep) and skip the per-request liveness NOOP within a freshness window, moving the ~1.5s reconnect cost off the user-facing sync path. - Delta sync fetches the newest messages by sequence number in one round trip instead of SEARCH-then-FETCH, removing a round trip. Measured warm-path delta sync on Gmail dropped from ~3.5s to ~1.5s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements Phase 4 (Realtime) of the proposal: new mail appears in the client within ~1–2s, no reload.
How it works
Gateway holds an IMAP IDLE connection per watched mailbox → on activity it pushes a tiny
changedevent over SSE → the client runs a delta sync. The push carries only the mailbox name; the delta is pulled over the existing REST API, keeping the event lightweight.Server
imap.Client.Watch— runs IMAP IDLE on a dedicated connection (never the shared request connection), forwarding mailbox/message/expunge updates via a callback. go-imap handles the IDLE capability check, periodic restart, and polling fallback.realtime.Hub— per-session fan-out to SSE subscribers with ref-counted watchers (N tabs on one folder share one IMAP connection; torn down at zero subscribers) and capped-backoff reconnect. Unit-tested with the race detector.GET /api/v1/events— SSE stream. Clears the per-connection write deadline so it isn't killed by the serverWriteTimeout; 25s heartbeat that also keeps the session warm.RequireSessionalso accepts the JWT as anaccess_tokenquery param, because the browserEventSourceAPI can't set anAuthorizationheader. (Access logs record only the path, not the query.)Client
EventSourcewatching the inbox; on each push it runs a debounced delta sync for the affected folder.Latency & correctness fixes (found during end-to-end testing on Gmail)
selectMailboxreused a cached read-onlySELECTfor a write, causing aSTORE on READ-ONLY folder502 retry-storm on mark-as-read. Now tracks access mode and re-SELECTs when needed.Measured: warm-path delta sync on Gmail dropped from ~3.5s → ~1.5s.
Testing
go build,go vet,go test ./...(incl. race-testedrealtimehub), and frontendtsc --noEmitall pass.Notes
mailboxquery param is CSV-capable for future extension.net/http+ browserEventSource; IDLE uses the existingemersion/go-imap.