SentTriage — Behavioral Email Classifier

6 buckets. 8 signals. Zero guesswork.
An LLM-powered email triage system that routes inbox messages based on behavioral metadata, not just content keywords.

---

What Is SentTriage?

Most email filters ask: what is this email about?
SentTriage asks: what does this email require from me?

Instead of reading every word, it reads 8 behavioral signals — reply history, thread direction, email age, attachment presence, sender tone, urgency type, and more — and classifies each email into one of 6 action buckets.

The result: a fully automated triage layer that works on top of any inbox.

---

The 6 Action Buckets

Bucket	Icon	Suggested Action
VIP_ENGAGED	⭐	Keep in inbox — known contact
GHOSTED_THREAD	💬	Follow up — you're leaving someone waiting
VITAL_LOGISTICS	📄	Keep — time-sensitive or official document
EXPIRED_LOGISTICS	⏰	Archive — the window has passed
DECOY_MARKETING	🎭	Archive + save unsubscribe link
ZOMBIE_NOISE	🧟	Delete instantly

---

The 8 Behavioral Signals

Each email is represented as a structured metadata record — no raw body, just signals:

Subject: Re: Contract draft
From: legal@partnerco.com
Days old: 2
Reply history: Have replied before
Thread initiated: By me
Has attachment: Yes
Preview: Attached is the revised contract with your redlines...

Signal	What it tells us
Reply history	Is this a known contact?
Thread initiated	Did the user start this? (key for GHOSTED detection)
Email age	Is time-sensitive content still valid?
Has attachment	Official document indicator
Sender tone	Personal vs. automated vs. bulk
Prior stop requests	Unsubscribe ignored = ZOMBIE
Content type	OTP vs. logistics vs. marketing
Urgency framing	"ends tonight" vs. company newsletter

---

Screenshots

Single Email — Live Classifier

Paste any email metadata, click Classify, get an instant bucket + action recommendation.

Eval & Coverage — Version History

Side-by-side accuracy across all prompt versions with delta indicators.

Eval & Coverage — Per-Bucket Accuracy

Per-bucket accuracy bars and failure browser for the selected version.

LangSmith — Experiment Dashboard

Each prompt version is a separate experiment — sort by exact_match score to compare.

LangSmith — Per-Email Trace

Every classify_email() call is @traceable — full input/output visibility per email.

To add screenshots: Run streamlit run ui/app.py, navigate each tab, and save screenshots to docs/screenshots/ using the filenames above.

---

Architecture

┌─ Input ──────────────────┐
│  Structured Email         │
│  Metadata (8 signals)     │
└────────────┬─────────────┘
             │
             ▼
┌─ Streamlit UI ───────────┐      ┌─ Nebius AI ──────────────┐
│  ui/app.py               │─────▶│  Llama-3.3-70B-Instruct  │
│  ┌ Single Email tab      │      │  OpenAI-compatible API   │
│  ├ About Buckets tab     │◀─────│  (api.studio.nebius.com) │
│  └ Eval & Coverage tab   │      └─────────────────────────-┘
└────────────┬─────────────┘               ▲
             │                             │
             ▼                    ┌────────┴──────────┐
┌─ LangSmith ──────────────┐     │  classifier_v*.txt │
│  Dataset: SentTriage-100  │     │  (6 versions)      │
│  Experiments per version  │     └────────────────────┘
│  @traceable per-email run │
│  Exact match + LLM judge  │
└────────────┬─────────────┘
             │
             ▼
     Observe → Evaluate → Iterate
     (Proximal Prompt Optimization — Phase 2)

---

Component Map

File	Role
src/classifier.py	Core classify_email() + run_batch(). Sends structured email metadata to the LLM, parses the 6-bucket response. Exposes NEBIUS_BASE_URL, NEBIUS_MODEL, VALID_LABELS.
src/judge.py	LLM-as-judge (eval layer 2). Scores each prediction 0/1/2 on a rubric, returns reasoning. Used by both langsmith_eval.py and the standalone judge runner.
src/langsmith_eval.py	Full LangSmith integration: dataset upload, exact_match evaluator, build_llm_judge(), @traceable target function, run_evaluation(). ProximalPromptOptimizer (PPO) commented out — Phase 2.
ui/app.py	Streamlit app. Three tabs: (1) Single Email classifier with sample emails; (2) Bucket descriptions; (3) Eval & Coverage — version history table, per-bucket accuracy, failure browser.
score_v2.py	CLI scorer. python score_v2.py v4 runs the full 100-email dataset against prompts/classifier_v4.txt, saves data/eval_results_v4.csv, appends to data/eval_summary.csv.
prompts/classifier_v*.txt	Prompt versions. Each is a standalone system prompt. Best: v4 (86%).
prompts/judge_v1.txt	LLM judge rubric — rates predictions 0 (wrong), 1 (borderline), 2 (correct).
data/dataset.csv	100 labeled emails. Columns: id, query, intent.
data/eval_results_v*.csv	Per-email results: id, query, ground_truth, predicted, match.
data/eval_summary.csv	Accuracy history across all versions — read by the Streamlit Coverage tab.

---

Prompt Version History

Version	Accuracy	Key Change	Notable Failure Pattern
v1	84%	Baseline — 8-signal behavioral prompt	GHOSTED/VIP confusion; OTP→VITAL; ZOMBIE→DECOY
v2	85%	GHOSTED priority rule (Re: + sender waiting)	Overcorrected: all Re: threads → GHOSTED
v3	80%	DECOY/ZOMBIE emotional framing rule	Swung ZOMBIE re-engagement → DECOY
v4	86% ← best	B2B cold sales = DECOY; OTP always EXPIRED; Thread by sender → never GHOSTED	Still DECOY/ZOMBIE on brand commercial
v5	76%	"Prior relationship" as DECOY signal	SaaS lifecycle emails → DECOY (overfit)

Best prompt: prompts/classifier_v4.txt at 86%.

---

Eval Methodology

SentTriage follows the Observe → Evaluate → Iterate loop from:

• The Iteration Machine — Nubank / Aman Gupta (June 2026)
  Building a fast, reliable prompt-iteration flywheel using LangSmith. "Drives 60%+ of our prompt edits."

• The Eval Proving Ground — LangChain (June 2026)
  Why eval ladders (cheap evals first) beat running LLM judges on every example.

The Eval Ladder

Layer 1 — Exact Match (free, deterministic, always runs)
         exact_match = 1 if predicted == ground_truth else 0

Layer 2 — LLM-as-Judge (optional, ~100 extra LLM calls per experiment)
         Judge reads the email + predicted label
         Returns: 0 (wrong), 1 (borderline), 2 (correct) + reasoning text
         Use with: python src/langsmith_eval.py --eval v4
         Skip with: python src/langsmith_eval.py --eval v4 --no-judge

Proximal Prompt Optimization — Phase 2

The PPO class is implemented but commented out in src/langsmith_eval.py:

1. Score current prompt on eval dataset → baseline accuracy
2. Fetch recent failure traces from LangSmith
3. Ask LLM to generate N candidate rewrites targeting those failures
4. Score each candidate against the full dataset
5. Accept best candidate that improves on baseline
6. Epsilon clip: if any candidate drops >5% below baseline, revert and stop (prevents catastrophic forgetting)

To activate: uncomment class ProximalPromptOptimizer and run_ppo() in langsmith_eval.py.

---

Setup

Prerequisites

• Python 3.11+
• uv package manager
• Nebius AI API key — free tier available
• LangSmith API key — free tier available

Install

git clone <repo-url>
cd SentTriage
uv venv
uv pip install -r requirements.txt

Configure

Create .env in the project root (never commit this file — it's in .gitignore):

NEBIUS_API_KEY=your_nebius_key_here
NEBIUS_BASE_URL=https://api.studio.nebius.com/v1/
NEBIUS_CHAT_MODEL=meta-llama/Llama-3.3-70B-Instruct
LANGSMITH_API_KEY=your_langsmith_key_here
LANGSMITH_PROJECT=SentTriage
LANGCHAIN_TRACING_V2=true

Run

# Launch the web UI
streamlit run ui/app.py

# Score a prompt version locally (saves data/eval_results_v4.csv)
python score_v2.py v4

# Upload dataset to LangSmith + run experiment with LLM judge
python src/langsmith_eval.py --upload --eval v4

# Fast eval — exact match only, no extra API calls
python src/langsmith_eval.py --eval v4 --no-judge

---

Project Structure

SentTriage/
├── src/
│   ├── classifier.py        # Core LLM classifier (classify_email, run_batch)
│   ├── judge.py             # LLM-as-judge evaluator
│   └── langsmith_eval.py    # LangSmith integration + PPO stub (Phase 2)
├── ui/
│   └── app.py               # Streamlit web app (3 tabs)
├── prompts/
│   ├── classifier_v1.txt    # Baseline prompt — 84%
│   ├── classifier_v2.txt    # GHOSTED priority rules — 85%
│   ├── classifier_v3.txt    # DECOY/ZOMBIE attempt — 80%
│   ├── classifier_v4.txt    # Best prompt — 86% ← use this
│   ├── classifier_v5.txt    # Prior-relationship attempt — 76%
│   └── judge_v1.txt         # LLM judge rubric
├── data/
│   ├── dataset.csv          # 100 labeled emails
│   ├── eval_results_v*.csv  # Per-version per-email results
│   └── eval_summary.csv     # Accuracy history (drives version history UI)
├── docs/
│   ├── architecture.svg     # System architecture diagram
│   ├── components.svg       # File-level component diagram
│   └── screenshots/         # UI and LangSmith screenshots (add manually)
├── score_v2.py              # Multi-version local scorer CLI
├── requirements.txt         # openai, langsmith, streamlit, python-dotenv
├── .env                     # Secrets — NOT committed
└── .gitignore

---

References

Resource	Purpose
Nebius AI Studio	OpenAI-compatible endpoint (Llama 3.3-70B, Qwen3, Gemma 3)
LangSmith	Dataset management, tracing, evaluation experiments
The Iteration Machine — Nubank	Observe→Evaluate→Iterate loop + PPO methodology
The Eval Proving Ground — LangChain	Eval ladders, LLM-as-judge calibration, exact-match-first strategy
SentTriage.txt (local)	Full project specification