Raqib (رقيب, "the watchful observer") continuously monitors for lookalike / typosquat domains impersonating banks, passively captures evidence, scores each suspect with AI (and a transparent heuristic), and gives a human analyst a triage console to confirm threats and generate an incident report for takedown.
It implements the kind of continuous digital-impersonation monitoring that the Central Bank of the UAE (CBUAE) expects licensed banks to run: detecting fake banking sites, lookalike domains, and credential-harvesting pages, then preparing them for reporting and takedown.
- Passive, defensive only. For each suspect domain Raqib performs at most a
single, rate-limited
GETplus one headless render to capture evidence. It never submits forms, enters credentials, brute-forces, actively scans or exploits, and never repeatedly hammers a host. Detection and reporting only. - Authorized use only. The operator configures a watchlist of brands they are authorized to protect — their own bank, official domains, and logos. Do not use Raqib to monitor brands you are not authorized to protect.
- Never auto-contacts authorities. The "report" feature generates an incident report for a human to review and file. Nothing is ever sent automatically to aeCERT, CBUAE, a registrar, or anyone else.
- Secrets live in
.env(git-ignored). Keys are never committed.
- Generates lookalike variants of each protected domain (dnstwist) and watches the Certificate Transparency stream for newly-issued certs that fuzzy-match a protected brand.
- For each live suspect, passively collects evidence: DNS/MX, registration age (RDAP→WHOIS), TLS issuer + cert age, a headless render (screenshot, login / password / OTP field detection, redirect chain, final URL, HTTP status), and a perceptual-hash logo similarity score.
- Scores each suspect with a transparent heuristic and an Anthropic model (degrading gracefully to heuristic-only when no API key is set).
- Streams detections live to a dark "threat-console" dashboard where an analyst confirms threats, marks false positives, and generates a takedown/incident report.
dnstwist (scheduled sweep) ─┐
├─► candidate ─► ingest/dedupe ─► Detection
CertStream CT (optional) ───┘ │
▼
Enrichment pipeline (passive, rate-limited, one render per check)
DNS/MX ─► RDAP/WHOIS age ─► SSL age ─► Playwright render ─► logo phash
│
▼
Evidence ─► Heuristic score
│ │
▼ ▼
Anthropic AI assessment
│
SQLite / Postgres ◄──────────────┤
▼
REST /api/* + WS /ws/detections ─► React console
Backend: Python 3.11+, FastAPI, SQLAlchemy 2.x, APScheduler, SQLite (single
DATABASE_URL, Postgres drop-in). Frontend: React + TypeScript + Vite, live
over WebSocket with REST-polling fallback.
cp .env.example .env # then edit .env (see env vars below)
docker compose up --build- Dashboard: http://localhost:5173
- API + docs: http://localhost:8000/api/health, http://localhost:8000/docs
The backend image is built on the official Playwright Python image, so Chromium and its system libraries are installed for you.
Backend
cd backend
python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txt
python -m playwright install chromium # one-time browser download
cp ../.env.example ../.env # edit as needed
uvicorn app.main:app --reload --port 8000Frontend
cd frontend
npm install
npm run dev # http://localhost:5173 (proxies /api and /ws to :8000)Tests
cd backend
pytest # domain-diff, heuristic scorer, AI parsing (mocked), enrichment (mocked)| Variable | Default | Purpose |
|---|---|---|
ANTHROPIC_API_KEY |
(empty) | Enables AI scoring. Optional — without it Raqib is heuristic-only. |
ANTHROPIC_MODEL |
claude-sonnet-5 |
Model used for triage assessments. |
DATABASE_URL |
sqlite:///./data/raqib.db |
SQLite by default; set a postgresql+psycopg://… URL for Postgres. |
CT_ENABLED |
false |
Turn on the Certificate Transparency source (resilient, optional). |
CT_WEBSOCKET_URL |
wss://certstream.calidog.io/ |
CertStream-compatible firehose. |
SCAN_INTERVAL_MINUTES |
60 |
How often the dnstwist sweep runs. |
REQUEST_RATE_LIMIT |
2.0 |
Minimum seconds between outbound hits to a single host. |
SCREENSHOT_DIR |
./data/screenshots |
Where page screenshots are written. |
WATCHLIST_PATH |
./seed/watchlist.json |
The operator's authorized-brand watchlist. |
FRONTEND_ORIGIN |
http://localhost:5173 |
Allowed CORS origin. |
Edit seed/watchlist.json (loaded on startup), then restart or POST /api/scan:
{
"brands": [
{
"name": "My Bank",
"official_domain": "mybank.ae",
"aliases": ["mybank.com", "mybank-online.ae"],
"logo_path": "./seed/logos/mybank.png"
}
]
}official_domainandaliasesare the legitimate domains — they are excluded from detection so you never flag yourself.logo_pathis your reference logo; Raqib perceptual-hashes rendered pages against it to score visual impersonation. Drop the image inseed/logos/.
You can also add a brand at runtime via POST /api/brands.
Raqib uses two independent, pluggable detection sources behind a common
interface (app/sources/base.py). Either can run without the other.
dnstwist (always-on backbone). On the SCAN_INTERVAL_MINUTES schedule (and
on demand via Run sweep), Raqib generates typo/permutation variants of each
protected domain and keeps only those that actually resolve — i.e. a real,
registered lookalike exists. This is the reliable core: Raqib is fully
functional on dnstwist alone.
Certificate Transparency (near-real-time, optional). When CT_ENABLED=true,
Raqib subscribes to a CertStream-compatible WebSocket firehose and flags
newly-issued certificates whose hostnames fuzzy-match a protected brand — often
catching a phishing site the moment its TLS cert is issued, before it's widely
live. The public firehose is notoriously flaky, so this source is resilient
(auto-reconnect with exponential backoff) and entirely optional. If it drops or
is disabled, the dnstwist scheduler keeps everything working.
- Discover. dnstwist permutations + CT fuzzy matches produce candidate lookalike domains for each brand.
- Ingest. A new candidate that resolves (or has a fresh cert) becomes a
Detection(deduped per brand + domain). - Enrich (passive, rate-limited, one render).
DNS/MX → registration age (RDAP, WHOIS fallback) → SSL issuer + cert age →
Playwright render (screenshot, login-form /
input[type=password]/ OTP heuristics, redirect chain, final URL, HTTP status) → logo perceptual-hash compare. Persisted asEvidence. - Score. A transparent heuristic (young domain, fresh cert, password+OTP
harvesting, high logo match, offshore hosting, MX present…) produces a 0-100
score with the exact signals that fired. Then the Anthropic model returns a
structured JSON assessment (
score,verdict,confidence,reasons,recommendedAction). Parsing is defensive; any failure falls back to the heuristic and the UI flags AI as unavailable. - Push. New/updated detections stream to connected dashboards over WebSocket (REST polling fallback).
- Triage. The analyst confirms a threat (TP) or marks a false positive (FP); labels and actions persist. Report to authorities generates a copyable/downloadable incident report (brand, suspect domain, technique, hosting, SSL, harvesting indicators, analyst label, AI reasoning, and suggested recipients — aeCERT / CBUAE / bank SOC / registrar abuse). It is a draft for a human to file — Raqib sends nothing.
| Method | Path | Purpose |
|---|---|---|
| GET | /api/detections |
List (filter by status/brand_id/view, paginate). |
| GET | /api/detections/{id} |
Detail with evidence, latest AI assessment, char-diff. |
| POST | /api/detections/{id}/analyze |
Re-run AI (?reenrich=true re-captures evidence). |
| POST | /api/detections/{id}/label |
Analyst label tp / fp. |
| POST | /api/detections/{id}/action |
reported / dismissed. |
| GET | /api/detections/{id}/report |
Generated incident-report text. |
| GET | /api/export/labeled |
All labeled cases + evidence as a JSON training set. |
| GET/POST | /api/brands |
List / add protected brands. |
| POST | /api/scan |
Trigger an immediate dnstwist sweep. |
| GET | /api/stats |
open, confirmed, reported, labeled, false-positive rate. |
| WS | /ws/detections |
Live detection push. |
The model is prompted as a UAE bank brand-protection triage analyst operating under CBUAE digital-impersonation requirements and must return only minified JSON:
{"score":0-100,"verdict":"malicious|suspicious|benign","confidence":"low|medium|high","reasons":["evidence-based reason under 18 words"],"recommendedAction":"report|monitor|dismiss"}2–4 reasons, each citing a specific supplied evidence field. Responses are parsed defensively (code fences stripped, score clamped, enums validated); on any error Raqib falls back to the heuristic score.
Raqib is a defensive brand-protection tool. Operate it only against brands you are authorized to protect, keep all activity passive, and have a human review every incident report before it is filed.