Skip to content

Detect lookalike domains under wildcard public suffixes - #83

Open
rosa wants to merge 7 commits into
mainfrom
detect-lookalikes-under-wildcard-suffixes
Open

rosa wants to merge 7 commits into
mainfrom
detect-lookalikes-under-wildcard-suffixes

Conversation

@rosa

@rosa rosa commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

The public suffix list has wildcard entries such as *.mm, *.kh, *.nom.br and *.sch.uk: every name directly under the base is itself a public suffix. Lookalike labels in those positions went undetected:

  • раураӏ.mm, раураӏ.kh, раураӏ.nom.br, раураӏ.sch.uk failed PublicSuffix.valid?, fell through to a PublicSuffix::Domain with no second- or third-level label, and no rule ran on them at all.
  • One level down (login.раураӏ.mm), the wildcard label was folded into the TLD, so it was never checked. A Cyrillic wildcard label also made Cyrillic lookalikes next to it acceptable (ѕсоре.почта.kh), because the TLD then "contained" Cyrillic.
  • The same zero-label fallthrough hid names matching a private entry (раураӏ.compute.amazonaws.com), because valid? considered private entries and parse didn't.

What changes

Detector (Detector::Idn)

  • Splits the domain at the part of the suffix the registry spells out (mm for *.mm; kawasaki.jp for !city.kawasaki.jp), and checks the wildcard label like any other label. Private entries are ignored throughout, so names under them are ordinary names.
  • Checks each label on its own, as Chrome does, instead of the second-level label plus the rest of the chain joined into one string. Joined, a Cyrillic label next to an ASCII one (пример.compute.amazonaws.com) tripped the mixed-scripts rule; per label it doesn't.
  • Pairs each checked label with the label as written by position, rather than searching the domain for it. That finds every casing of a repeated label (РАУРАӀ.раураӏ.mm) without matching inside longer labels, and stays linear on many repeated labels.
  • The Myanmar allow-list now matches the .mm registry suffix directly (/mm/ instead of /[a-z]+\.mm/), since the wildcard label is no longer part of the TLD.

Sanitizer (Sanitizer::Base)

  • Encodes the longest detected label first, so encoding a short detected label inside a longer detected one (а inside а‮) can't leave the longer one unencoded.
  • Inserts the encoded label through a block, so a backslash sequence in a label isn't read as a back-reference (twenty labels containing \` grew a 131-byte domain to 4 MB).

Behaviour changes worth knowing

  • New detections: lookalike labels directly under or below a wildcard base, and under private wildcard entries. Thai and other script lookalikes directly under wildcard bases (ทนบพรห.mm) are flagged too, as they would be under any non-Thai TLD.
  • Fewer detections: scripts mixed only across labels on a TLD that allows that script (раураӏ.www.ac.ru, ทนบพรห.www.shop.th). Each label alone was already safe there.
  • Leading-dot names under a wildcard base (.раураӏ.mm) are now invalid_domain, as they are elsewhere.
  • Unchanged: when a detected label also appears inside a longer safe label, it is still encoded there too (а.академия.com). Main already does this, and it never leaves a lookalike unencoded.

Overlap with #76 and #77

#76 also moves every rule to one label at a time, and #77 (stacked on it) scopes sanitizer replacements to the address part each detection came from. This PR's per-label split and sanitizer changes overlap with both and will conflict. If #76 lands first, this PR shrinks to the wildcard and private-entry split plus the sanitizer ordering and literal-replacement fixes, rebased onto it. If this lands first, #76 rebases onto it.

Verification

  • bin/test: 58 runs, 398 assertions, 0 failures, with new tests for the four wildcard bases, exception entries, private entries, per-label checks and the sanitizer cases above.
  • Ran the detector over every public suffix list entry with eleven sample labels (107k names), and over the same entries with ten sample subdomain chains, comparing results against main. On the single-label set, the only new results are detections under wildcard bases and private entries. Nothing that was detected stopped being detected. The chain set shows the cross-label losses described above. These are synthetic names, not real traffic.
  • Adversarial review by Codex over seven rounds, until it reported no new findings. Its findings, all addressed here: casing recovery for repeated labels, the joined-chain false positive, a quadratic slowdown on repeated labels, two attempts at label-boundary matching in the sanitizer that were replaced by the longest-first order, and the backslash replacement blow-up. It also noted that binary-encoded (non-UTF-8) strings raise; that already happens on main for any TLD and is out of scope here.

🤖 Generated with Claude Code

rosa and others added 7 commits October 9, 2026 22:55
Under a wildcard entry of the public suffix list (*.mm, *.kh, *.nom.br,
*.sch.uk), every name directly below the base is itself a public suffix.
Such a name failed PublicSuffix.valid?, fell through to a Domain with no
second- or third-level label, and no rule ran on it, so раураӏ.mm and
friends went undetected. Below one, the wildcard label was folded into the
TLD, so it was never checked either, and a Cyrillic wildcard label made
Cyrillic look-alikes next to it acceptable.

Split the domain at the part of the suffix the registry spells out, check
the wildcard label like any other, and keep PublicSuffix's own sld/trd
split for the rest. Ignore private entries consistently, so names matching
one (раураӏ.compute.amazonaws.com) are checked too. The Myanmar allow-list
now matches the .mm registry suffix directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…abel

Joining the subdomain chain into one string made the mixed-scripts rule
flag a Cyrillic label next to an ASCII one (пример.compute.amazonaws.com),
which Chrome allows: it checks scripts per label. Names matching a private
wildcard entry newly reached that, so split every label out.

The sanitizer substitutes the original-cased label, but only the first
casing was recovered, so РАУРАӀ.раураӏ.mm left раураӏ unencoded. Return a
detection for every distinct casing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Searching the domain for every casing of a detected label matched it
inside longer safe labels too (раураӏ.РАУРАӀмузей.mm), and rebuilding the
casing map for every rule made repeated labels quadratic. Pair each
checked label with the label as written by position instead, and check a
label repeated verbatim once.

Take the labels by stripping the registry suffix off the name rather than
through PublicSuffix's line-anchored match, which skipped anything before
a newline.

The sanitizer replaced every occurrence of a detected label, including
inside longer labels (а inside академия). Replace whole occurrences only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A display name in another encoding (UTF-16) is reported as
invalid_unicode, and interpolating it into the UTF-8 pattern raised.
Substitute it wherever it appears, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A label followed by punctuation or a format character inside a longer
label (ד in ד׳שלום, а in а‮) still counted as a whole occurrence.
Require a separator, the edge of the field, a dot, the @, or the quotes
and brackets around an address, on both sides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No list of separators identifies a whole label in every field the
sanitizer handles: escaped display names and malformed labels with
spaces or punctuation still left a detected label partly encoded. Go back
to substituting every occurrence, as before, and do the longest label
first, so encoding a short detected label inside a longer one can't leave
the longer one unencoded now that each label is checked on its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gsub reads backslash sequences in a replacement string, so a label with
\` inserted the text before the match; with every label substituted on
its own, twenty such labels grew a 131-byte domain to 4 MB. Substitute
through a block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 9, 2026 21:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approved

The implementation is internally consistent and includes focused regression coverage for the identified edge cases.

0 open findings

What changed in this PR

Detects spoofing across individual labels, including wildcard and private public suffix cases, while hardening sanitization.

Changes:

  • Splits domains by registry-controlled suffix and checks each label independently.
  • Corrects Myanmar suffix handling.
  • Safely sanitizes nested labels, backslashes, casing, and non-UTF-8 input.

[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

File Description
lib/​homographic_spoofing/​detector/​idn.rb Implements per-label detection and registry suffix splitting.
lib/​homographic_spoofing/​detector/​rule/​idn/​script_confusable.rb Updates Myanmar suffix matching.
lib/​homographic_spoofing/​sanitizer/​base.rb Safely orders and replaces detected labels.
test/​detector/​idn_test.rb Covers wildcard, private suffix, exception, and per-label behavior.
test/​sanitizer/​idn_test.rb Covers wildcard and replacement edge cases.
test/​sanitizer/​quoted_string_test.rb Covers non-UTF-8 sanitization.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants