feat: add ML-based antispam classifier (v0.2) - #388
Open
rafpigna wants to merge 19 commits into
Open
Conversation
Renumbered migrations to avoid collisions with upstream: - 0022 -> 0048 (upstream took 0022 for password_reset_tokens) - 0035 -> 0049 (upstream took 0035 for oidc_rp_initiated_logout) - 0036 -> 0050 (upstream took 0036 for unified_inbox_accounts)
Core text pipeline for the v0.2 ML classifier (V2-3): - stopWords.js: stop-word sets for the 7 UI locales (en, it, de, es, fr, ru, zhCN), >=50 words per language - spamParser.js: Authentication-Results parsing per RFC 7601/8601 (not RFC 8054, which is NNTP), multi-header and folded-header safe - spamTokenizer.js: bag-of-words tokenization (subject weighted x2, HTML stripping via htmlparser2, Unicode scripts incl. CJK and Cyrillic, URL-host tokens), flag feature extraction matching the spam_training_log.flag_features schema (migration 0048), and a stable token fingerprint for classify-dedupe Pure functions only; no I/O. Design: spam-classifier-v0.2.md §6.1.
Always-on Layer 1 of the hybrid classifier, per the maintainer-reviewed rules catalog (spam-rules-detailed.md): - 14 rules: 9 SpamAssassin-style + 4 best-practice + 1 MailFlow-specific (FROM_IN_USER_CONTACTS with Gmail local-part normalization) - scoreRules() clamps the weighted sum to [0,1]; attachment rules (EXECUTABLE + DOUBLE_EXT) are deduplicated in the score but both stay visible in the fired list for explainability - explainRules() drives the future "Why?" modal - AUTH_*_FAIL rules stay neutral when no Authentication-Results header exists at all (a client-side mailbox would otherwise score +1.2 on every message and clamp to 1.0); individual methods absent from a present header still fire - Registrable-domain comparison for FROM_REPLYTO_MISMATCH uses a pragmatic multi-label suffix list, no new dependency (same approach as senderFavicon.js)
Layer 2 of the hybrid classifier — pure math, no I/O (V2-3): - updateIncremental: per-token count increments + derived priors from the training set (total_spam/total_ham proxy per design §6.2), never hardcoded 0.5 except cold start - classifyMessage: log-probability-space scoring with Laplace smoothing (alpha=1.0); fixed asymmetric auth weights from ADR-001 v2 (dkim/spf fail +0.7, dmarc fail +0.5, pass -0.2/-0.1/-0.1) applied as token-free log-odds offsets, absent header = neutral - heuristic flags learned as special __tokens__ (attachment, mismatch, all-caps); auth flags NOT stored in the vocabulary (would double-count the fixed weights) - pruneVocabulary: chi-square feature selection, cap 10k, drop count<3 and tf>0.05 (design §14) - blendScores: rules-only <50, 60/40 50-500, 20/80 >500 (design §6.3) - extractTopTokens: top discriminative tokens for the explain modal Implements the SpamClassifier interface contract (score/train/prune) so v0.3 can swap the algorithm behind the same boundary.
V2-4 DB integration layer: - spamModelStore.js: spam_models DB access behind a 5-min in-memory cache (same pattern as categorizer.js socialDomainCache). Exposes getModelForUser (null on cold start), saveModel (UPSERT + cache invalidate), updateIncrementalForUser (feature extraction at mark time + spamModel.updateIncremental), getAllUsersWithTraining(Log) and retrainUser (full rebuild from spam_training_log, NO JOIN, with exponential decay 2^-age/days) - spamScheduler.js: hourly job that retrains only the current staggered bucket (offset = hash(user_id) % 24, design §7.2 fix v4) to avoid a 00:00 UTC thundering herd; runFullRetrain() for the admin endpoint; stable per-user offset with uniform distribution Decay is downweighting, never deletion (ADR v2). Retrain reads pre-tokenized token_counts and falls back to raw text for v0.1 rows.
- spamModel.js: retrainFromRecords(records, decayThresholdDays) rebuilds a model from spam_training_log rows with exponential time decay (2^-age_days/threshold), supporting pre-tokenized token_counts and raw-text fallback for v0.1-era rows; priors derive from the weighted totals; pure function with injectable clock for tests - index.js: startSpamScheduler() at boot next to the CardDAV scheduler
V2-5: wire the /spam and /ham endpoints to the v0.2 ML pipeline: - extractSpamFeatures: builds token_counts, flag_features, sender_domain and attachment_types at mark time (Solution C, migration 0048) from the messages row, so the retrain path never joins back to messages - logSpamTraining: single shared INSERT that now also persists the new feature columns, then feeds the per-user Naive Bayes model via spamModelStore.updateIncrementalForUser (non-blocking on failure so user feedback never breaks the HTTP path) - both spam and ham flows (no-op already-in-folder and IMAP move path) now train the model in <1s - mail.spam.test.js: integration tests for both routes asserting the feature columns are stored and updateIncrementalForUser is called with the right label
V2-6: wire the classification pipeline into message ingest:
- spamPipeline.classifyAndTagMessage(messageId): hybrid 3-layer scoring
for newly-inserted messages (imapManager processMsg is_new branch)
- user_override always wins (skip)
- antispam_enabled gate (default OFF — opt-in per account)
- rules engine always on; MNB model joins when training_records >= 50;
blend 60/40 (50-500) / 20/80 (>500) per design §6.3
- verdict spam >= 0.85, auto-move to folder_mappings.spam >= 0.95
and only with an ML-backed score (rules-only never auto-moves)
- persists spam_verdict/spam_score_ml/spam_analyzed_at/spam_details
(JSON: method, blended+rules scores, fired rules, top tokens)
- imapManager hook is fire-and-forget: the sync loop is never blocked,
failures only log; the IMAP move is delegated through an injected
facade (no import cycle with routes)
- auto-verdicts never write spam_training_log (only explicit /spam /ham
trains the model — avoids poisoning)
10 unit tests covering gates, fresh-install no-move, active-user
auto-move, unconfigured spam folder, borderline ham, blend shift and
move-failure resilience.
The greenmail E2E script (e2e_antispam.py) is a local dev tool like seed_inbox.py; keeping it out of the PR per the v0.2 file convention documented in .hermes/v0.2-notes/reports.md.
V2-7: /api/spam/* namespace with per-user threshold control, GDPR erasure and explainability: - GET /status: model version, training records, last trained, decay days, maturity band and per-user/account enable state - GET/PATCH /thresholds: read/update per-user spam thresholds stored in users.preferences.spam_thresholds (PATCH admin-only, range spamThreshold 0.5-0.99, autoMoveThreshold 0.7-0.99) - GET/PATCH /decay-threshold: user-configurable decay window 7-365 days (upserted on spam_models, invalidates the model cache) - POST /retrain-now (admin): full staggered retrain - POST /enable: per-user master switch (users.preferences.spamEnabled) - POST /reset-training-all + GET /deletions: GDPR right-to-erasure with explicit confirm, audit trail row, counts; per-account reset on the /api/accounts/:id/spam/reset-training namespace (design §16) - GET /explain: rules fired + top ML tokens for the Why? modal Migration 0052: spam_training_deletions audit table (Option B).
- index.js: mount /api/spam router and the per-account GDPR reset router on /api/accounts/:id/spam/reset-training - accounts.js: allow antispam_enabled in the account PUT allowlist (per-account toggle for the settings page) - spamPipeline: skip classification when the new per-user master switch users.preferences.spamEnabled is off (JOIN users in the ingest query); test for the new skip reason
The account edit form needs the per-account antispam toggle to load its current state. GET /api/accounts now projects antispam_enabled alongside categorization_enabled.
V2-8 frontend: - SpamSettings.jsx: per-user antispam tab in AdminPanel (model status, thresholds sliders, decay window, master switch, retrain-now, GDPR reset-all with confirm, deletion audit trail) - SpamExplainModal.jsx: "Why was this marked as spam?" modal showing fired rules + top ML tokens (opened from the badge) - SpamBadge.jsx: inline spam verdict badge (score %) next to the subject in MessageRow, ThreadRow and MessagePane; click opens the explain modal - AdminPanel: new Antispam tab (TAB_GROUPS + TABS + content), per-account antispam_enabled toggle in the account edit form - Modal + badge are wired in MessageList (regular + thread rows) and MessagePane i18n (all 7 locales): top-level spam.* block + admin.tabs.antispam + admin.accounts.antispamSection/Enabled/EnabledDesc; i18n.test.js updated with the dynamic keys and intentional same-value groups (Antispam, Spam, Verdict, Mature, Fresco loanwords).
Top-level spam.* block (settings, explain modal, badge) plus admin.tabs.antispam and the three admin.accounts.antispam* keys. i18n.test.js: register the template-literal keys (maturity, deleteScope, verdict, method) in DYNAMIC_KEYS and the intentional shared loanword values in SAME_VALUE_ALLOWED (Antispam, Spam, Verdict, Mature, Fresco).
V2-9 coverage: spamModelStore was at 67% lines; retrainUser (the scheduled/admin full rebuild) had no direct tests. Added: - no_training_data fast path (no upsert attempted) - full rebuild from training rows: vocabulary aggregation, decay default 90, special-token flags, upsert payload shape - decay_threshold_days carried from an existing model row Store coverage now 100% statements/lines. Uses a partial module mock so the existing query-assertion tests still observe the real upsert.
The full-retrain test seeded training rows with created_at = test-run now; retrainFromRecords applies 2^-age/90d which, a few ms after seed, yields 0.9999999999 instead of exactly 1.0. Assertions now use toBeCloseTo(…, 5) for counts — the test is time-independent and no longer flaky in the full suite run.
Fix four issues surfaced by the v0.2 UI walkthrough: - Badge: the message list and pane never exposed spam_verdict/spam_user_override/spam_score_ml in their SQL projections (listMessages threaded + flat, GET /messages/:id, resolve-message COLS), so SpamBadge had no data to render. Add the three columns to all of them. - Feedback: SpamSettings showed saves via an inline box that was nearly invisible (var(--accent-dim)). Switch to the app-wide toast system (addNotification) so Save / Retrain now / GDPR reset and errors surface a self-dismissing popup, matching the manual spam/ham classification UX. - Contrast: the status box labels used var(--text-tertiary) on --bg-hover; use --text-secondary with weight 600. - Account toggle staleness: antispam_enabled was missing from SAFE_FIELDS in safeAccount, so PUT /accounts/:id returned a payload without it and the store never refreshed the toggle until a full reload. Add it to SAFE_FIELDS.
Polish from the second UI walkthrough: - Badge: replace the ambiguous two-star glyph with a clear text label (SPAM INDEX / INDICE SPAM / … in all 7 locales) plus the score percentage. This applies both in the message pane and the list. - List visibility: the subject row was not a flex container, so the badge sat inside an overflow:hidden + ellipsis box and was clipped out of view on long subjects. Make the row flex with the subject truncating (flex:1 min-width:0) and the badge shrink-proof. - Explain modal: section titles and the Verdict / Method / Confidence labels used var(--text-tertiary) on the dark card, which was nearly invisible. Bump them to var(--text-secondary), same as SpamSettings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements the v0.2 ML classifier for the antispam filter, following the plan discussed in #156 (v0.1 manual feedback → v0.2 Bayesian ML → v0.3 SpamAssassin). It builds on the
spam_training_loginfrastructure shipped in v0.1 (PR #181): when a user marks a message as spam/ham, the message features are now captured at mark time (Solution C — no JOIN on the messages table) and a per-user Multinomial Naive Bayes model is trained and used to auto-classify incoming messages into the Junk folder.The classifier is deliberately a hybrid 3-layer system, designed so that v0.3 (SpamAssassin) can slot into the same interface later:
Because this is a self-hosted product with the user's own mail data, the design puts privacy and auditability first (see the threat model in Testing).
Changes
Backend
0052_spam_training_features— captures subject/body/token counts/flags/sender/attachments at mark time (Solution C).0053_spam_models— per-user model state: JSONB vocabulary, weighted totals (REAL, since the decay retrain produces fractional counts), priors, decay window, model version.0054_email_accounts_antispam_enabled— per-account toggle (default OFF).0055_spam_training_deletions— GDPR audit trail for training-data resets.src/services/spam*.js):spamTokenizer— bag-of-words (subject ×2, HTML strip, unicode CJK/Cyrillic, URL-host tokens, stop-words in 7 languages, heuristic flag features, message fingerprint).spamParser—Authentication-Resultsparsing per RFC 7601/8601.spamRules— the 14 heuristic rules approved in the design doc (attachment dedup uses max, not sum).spamModel— MNB scoring in log-space: priors derived from the training set, fixed asymmetric auth weights (dkim/spf fail +0.7, dmarc fail +0.5, pass −0.2/−0.1/−0.1), chi-square feature selection (cap 10k vocab), confidence blending.spamModelStore— DB I/O with a 5-minute in-process cache;spamScheduler— hourly retrain with per-user staggered bucket (hash(user_id) % 24) and configurable exponential decay (default 90 days, range 7–365).spamPipeline— classifies new messages on ingest (hook inimapManager), auto-moves to Junk at ≥ 0.95 only when the ML is active, persists verdict/spam_details. Auto-verdicts are never written back to the training log (prevents self-poisoning; only manual feedback trains).src/routes/spam.js): status, per-user thresholds (GET/PATCH, admin), decay threshold (7–365), admin retrain-now, master enable, GDPR reset (per-account + global, with explicit confirm), deletions audit, and aWhy?explain endpoint (fired rules + top ML tokens). Two-level enable: per-user master switch + per-account toggle; the pipeline does aJOIN usersand skips when the master is off.Frontend
spam 98%badge on message rows/panes (shown only for auto verdicts, so a manual override never looks like a model decision).antispam_enabledtoggle in the account form.Testing
spamModel97%, tokenizer 96%, rules 96%, parser 98%, store 100%, pipeline 100%, scheduler 89%, routes 85%.spam_training_logrows, messages), then ran my migration set on top —0052–0055applied cleanly, existing data intact (features stay NULL for legacy rows), and the backend boots with the spam routes live andantispam_enableddefaulting to false./api/spam/*: none of the endpoints are public — 5 layers: (1) global CSRF guard on/api(X-Requested-With, 403 without it); (2)requireAuthon every spam router; (3)requireAdminon threshold PATCH + retrain-now; (4) ownership check on explain and per-account reset; (5) explicitconfirm:trueon GDPR resets. Residual, intentional: a non-admin may read their own status and delete their own training data (a GDPR right). Future hardening: per-user rate-limiting on resets/retrain-now; IP address is already audited in0055.Design decisions: antispam toggle semantics (edge cases)
This section pins down what enabling/disabling the antispam feature does, so the
behavior in these edge cases is explicit and reviewed rather than discovered.
Two-level enablement. There are two independent switches, both gating
automatic classification: a per-user master switch
(
users.preferences.spamEnabled, default on) and a per-account toggle(
email_accounts.antispam_enabled, default off / opt-in). The pipeline returnsskippedbefore scoring if either is off.A toggle controls the whole automatic pipeline, not just auto-move. Turning
antispam off disables automatic scoring, auto-move to the spam folder, and the
auto-spam badge on new messages. It does not disable manual spam/ham marks —
those still train the model on any account.
The badge is historical, not a live toggle mirror.
SpamBadgerenders fromthe persisted
messages.spam_verdictwritten at analysis time. So:spam_user_override; badge hidden for that message.I deliberately chose the historical behavior: it does not "re-legitimize" months
of auto-filtered mail because a toggle was briefly flipped, does not erase
history on disable, and does not retroactively "pop" badges on re-enable
(
spam_verdictis a property of the message, likecategory). If you'd preferthe badge to mirror the live toggle state instead, that's a small isolated
frontend change — happy to switch on request.
Note: screenshots and user documentation
The UI walkthrough screenshots are shared as a public zip rather than embedded
inline, to keep the diff focused on code:
📦 antispam-v0.2-screenshots.zip
They cover the Antispam tab, thresholds/decay, the per-account toggle, the
auto-spam badge (list + pane), the "Why?" explain modal, and the GDPR reset
confirm dialog.
On documentation: I noticed this repository currently has no
docs/directoryor end-user/help documentation — the only markdown beyond the code is
README.md,CONTRIBUTING.md,ROADMAP.mdandCLA.md, even though theofficial site does ship user docs at https://mailflow.sh/docs. Rather than
introducing the repository's first end-user docs as part of this (already large)
feature PR, I kept the antispam "user guide" out of scope. I'd welcome a quick
steer on whether you'd like the antispam UI documented in the docs site and/or
as a new e.g.
docs/antispam.mdhere — happy to prepare that in a smallseparate PR if you confirm the preferred location/format.
Contributor License Agreement
By submitting this pull request I confirm that:
On AI assistance
AI tools and LLMs were used to speed up development, scaffolding, code generation,
bug fixing and analysis. All key decisions about architecture, design and project
strategy, as well as the real-world E2E tests on actual usage scenarios, were made
and supervised by a human.
— r.