Skip to content
SmoothiecoolPublic

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

13 Commits

Folders and files

Repository files navigation

Raqib — Bank Digital-Impersonation Monitor

Raqib (رقيب, "the watchful observer") continuously monitors for lookalike / typosquat domains impersonating banks, passively captures evidence, scores each suspect with AI (and a transparent heuristic), and gives a human analyst a triage console to confirm threats and generate an incident report for takedown.

It implements the kind of continuous digital-impersonation monitoring that the Central Bank of the UAE (CBUAE) expects licensed banks to run: detecting fake banking sites, lookalike domains, and credential-harvesting pages, then preparing them for reporting and takedown.


⚖️ Scope & guardrails (hard rules)

  • Passive, defensive only. For each suspect domain Raqib performs at most a single, rate-limited GET plus one headless render to capture evidence. It never submits forms, enters credentials, brute-forces, actively scans or exploits, and never repeatedly hammers a host. Detection and reporting only.
  • Authorized use only. The operator configures a watchlist of brands they are authorized to protect — their own bank, official domains, and logos. Do not use Raqib to monitor brands you are not authorized to protect.
  • Never auto-contacts authorities. The "report" feature generates an incident report for a human to review and file. Nothing is ever sent automatically to aeCERT, CBUAE, a registrar, or anyone else.
  • Secrets live in .env (git-ignored). Keys are never committed.

What it does

  1. Generates lookalike variants of each protected domain (dnstwist) and watches the Certificate Transparency stream for newly-issued certs that fuzzy-match a protected brand.
  2. For each live suspect, passively collects evidence: DNS/MX, registration age (RDAP→WHOIS), TLS issuer + cert age, a headless render (screenshot, login / password / OTP field detection, redirect chain, final URL, HTTP status), and a perceptual-hash logo similarity score.
  3. Scores each suspect with a transparent heuristic and an Anthropic model (degrading gracefully to heuristic-only when no API key is set).
  4. Streams detections live to a dark "threat-console" dashboard where an analyst confirms threats, marks false positives, and generates a takedown/incident report.

Architecture (text diagram)

   dnstwist (scheduled sweep) ─┐
                               ├─► candidate ─► ingest/dedupe ─► Detection
   CertStream CT (optional) ───┘                                    │
                                                                    ▼
   Enrichment pipeline (passive, rate-limited, one render per check)
     DNS/MX ─► RDAP/WHOIS age ─► SSL age ─► Playwright render ─► logo phash
                                                                    │
                                                                    ▼
                                          Evidence  ─►  Heuristic score
                                                          │        │
                                                          ▼        ▼
                                              Anthropic AI assessment
                                                          │
                        SQLite / Postgres  ◄──────────────┤
                                                          ▼
                          REST /api/*  +  WS /ws/detections  ─► React console

Backend: Python 3.11+, FastAPI, SQLAlchemy 2.x, APScheduler, SQLite (single DATABASE_URL, Postgres drop-in). Frontend: React + TypeScript + Vite, live over WebSocket with REST-polling fallback.


Setup

Option A — Docker (one command)

cp .env.example .env          # then edit .env (see env vars below)
docker compose up --build

The backend image is built on the official Playwright Python image, so Chromium and its system libraries are installed for you.

Option B — Local dev

Backend

cd backend
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt
python -m playwright install chromium                # one-time browser download
cp ../.env.example ../.env                           # edit as needed
uvicorn app.main:app --reload --port 8000

Frontend

cd frontend
npm install
npm run dev        # http://localhost:5173 (proxies /api and /ws to :8000)

Tests

cd backend
pytest             # domain-diff, heuristic scorer, AI parsing (mocked), enrichment (mocked)

Environment variables (.env)

Variable Default Purpose
ANTHROPIC_API_KEY (empty) Enables AI scoring. Optional — without it Raqib is heuristic-only.
ANTHROPIC_MODEL claude-sonnet-5 Model used for triage assessments.
DATABASE_URL sqlite:///./data/raqib.db SQLite by default; set a postgresql+psycopg://… URL for Postgres.
CT_ENABLED false Turn on the Certificate Transparency source (resilient, optional).
CT_WEBSOCKET_URL wss://certstream.calidog.io/ CertStream-compatible firehose.
SCAN_INTERVAL_MINUTES 60 How often the dnstwist sweep runs.
REQUEST_RATE_LIMIT 2.0 Minimum seconds between outbound hits to a single host.
SCREENSHOT_DIR ./data/screenshots Where page screenshots are written.
WATCHLIST_PATH ./seed/watchlist.json The operator's authorized-brand watchlist.
FRONTEND_ORIGIN http://localhost:5173 Allowed CORS origin.

How to add a brand

Edit seed/watchlist.json (loaded on startup), then restart or POST /api/scan:

{
  "brands": [
    {
      "name": "My Bank",
      "official_domain": "mybank.ae",
      "aliases": ["mybank.com", "mybank-online.ae"],
      "logo_path": "./seed/logos/mybank.png"
    }
  ]
}
  • official_domain and aliases are the legitimate domains — they are excluded from detection so you never flag yourself.
  • logo_path is your reference logo; Raqib perceptual-hashes rendered pages against it to score visual impersonation. Drop the image in seed/logos/.

You can also add a brand at runtime via POST /api/brands.


CT vs dnstwist sources

Raqib uses two independent, pluggable detection sources behind a common interface (app/sources/base.py). Either can run without the other.

dnstwist (always-on backbone). On the SCAN_INTERVAL_MINUTES schedule (and on demand via Run sweep), Raqib generates typo/permutation variants of each protected domain and keeps only those that actually resolve — i.e. a real, registered lookalike exists. This is the reliable core: Raqib is fully functional on dnstwist alone.

Certificate Transparency (near-real-time, optional). When CT_ENABLED=true, Raqib subscribes to a CertStream-compatible WebSocket firehose and flags newly-issued certificates whose hostnames fuzzy-match a protected brand — often catching a phishing site the moment its TLS cert is issued, before it's widely live. The public firehose is notoriously flaky, so this source is resilient (auto-reconnect with exponential backoff) and entirely optional. If it drops or is disabled, the dnstwist scheduler keeps everything working.


How detection works (end to end)

  1. Discover. dnstwist permutations + CT fuzzy matches produce candidate lookalike domains for each brand.
  2. Ingest. A new candidate that resolves (or has a fresh cert) becomes a Detection (deduped per brand + domain).
  3. Enrich (passive, rate-limited, one render). DNS/MX → registration age (RDAP, WHOIS fallback) → SSL issuer + cert age → Playwright render (screenshot, login-form / input[type=password] / OTP heuristics, redirect chain, final URL, HTTP status) → logo perceptual-hash compare. Persisted as Evidence.
  4. Score. A transparent heuristic (young domain, fresh cert, password+OTP harvesting, high logo match, offshore hosting, MX present…) produces a 0-100 score with the exact signals that fired. Then the Anthropic model returns a structured JSON assessment (score, verdict, confidence, reasons, recommendedAction). Parsing is defensive; any failure falls back to the heuristic and the UI flags AI as unavailable.
  5. Push. New/updated detections stream to connected dashboards over WebSocket (REST polling fallback).
  6. Triage. The analyst confirms a threat (TP) or marks a false positive (FP); labels and actions persist. Report to authorities generates a copyable/downloadable incident report (brand, suspect domain, technique, hosting, SSL, harvesting indicators, analyst label, AI reasoning, and suggested recipients — aeCERT / CBUAE / bank SOC / registrar abuse). It is a draft for a human to file — Raqib sends nothing.

API reference

Method Path Purpose
GET /api/detections List (filter by status/brand_id/view, paginate).
GET /api/detections/{id} Detail with evidence, latest AI assessment, char-diff.
POST /api/detections/{id}/analyze Re-run AI (?reenrich=true re-captures evidence).
POST /api/detections/{id}/label Analyst label tp / fp.
POST /api/detections/{id}/action reported / dismissed.
GET /api/detections/{id}/report Generated incident-report text.
GET /api/export/labeled All labeled cases + evidence as a JSON training set.
GET/POST /api/brands List / add protected brands.
POST /api/scan Trigger an immediate dnstwist sweep.
GET /api/stats open, confirmed, reported, labeled, false-positive rate.
WS /ws/detections Live detection push.

AI assessment contract

The model is prompted as a UAE bank brand-protection triage analyst operating under CBUAE digital-impersonation requirements and must return only minified JSON:

{"score":0-100,"verdict":"malicious|suspicious|benign","confidence":"low|medium|high","reasons":["evidence-based reason under 18 words"],"recommendedAction":"report|monitor|dismiss"}

2–4 reasons, each citing a specific supplied evidence field. Responses are parsed defensively (code fences stripped, score clamped, enums validated); on any error Raqib falls back to the heuristic score.


License / responsible use

Raqib is a defensive brand-protection tool. Operate it only against brands you are authorized to protect, keep all activity passive, and have a human review every incident report before it is filed.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages