Show and ideally warn when "strange" unicode chars are used like VS Code does.

[Frulfump]

I'd like it to show and ideally warn when "strange" unicode chars are used like VS Code does.

Originally posted by Frulfump in #188 (comment)

[qaijuang]

By "strange" here, i assume you mean characters that are not C0/C1 controls, like?

• RLO bidi controls
• zero-width space and word joiner
• embedded BOM
• non-ASCII whitespace
• Unicode line and paragraph separators

[Frulfump]

Yes, and maybe also U+007F DELETE if that makes sense and any others that follow the similar themes as the ones you mentioned or anything else that also might make sense. Maybe malformed Unicode. (And we probably shouldn't blindly follow VS Code and instead look at https://www.unicode.org/reports/tr55/ (explained a bit more further down) and do our own thing it was just something that was easy to reference initially)

So essentially characters whose textual representation is difficult or impossible to infer correctly from their normal visual representation.

I tried looking at https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-5/#G7730 "Unknown and Missing Characters" as well for guidance.
Especially https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-5/#G7742 "Interpretable but Unrenderable Characters"

https://www.unicode.org/reports/tr55/ "Unicode Source Code Handling" is also most likely interesting to check out

Summary

Because Unicode contains such a large number of characters and incorporates the varied writing systems of the world, usability and security issues can arise from improper handling of Unicode program text. This document provides guidance for programming language designers and programming environment developers, and specifies mechanisms to alleviate those issues.

https://www.unicode.org/reports/tr55/#Show-Hidden-Options

highlight text not in Normalization Form C

https://github.blog/changelog/2021-10-31-warning-about-bidirectional-unicode-text/ Talks a bit about the bidirectional category

I think maybe a separate issue for "Confusables" also makes sense i.e
Latin: a U+0061
Cyrillic: а U+0430
https://www.unicode.org/reports/tr55/#Confusable-Display
https://www.unicode.org/reports/tr55/#Confusable-Detection
What do you think about that?

[qaijuang]

Quite straightforward!

Let's hear from Leonard Hecker (@lhecker)

[hexbinoct]

I had a look through the code before proposing anything, and the first thing worth reporting is
that a good part of this already ships.

What edit does today

Framebuffer::replace_text runs every string it draws through sanitize_control_chars
(crates/stdext/src/unicode/sanitize.rs), which replaces C0 controls with U+2400..U+241F,
U+007F with U+2421, C1 controls with U+2426, and invalid UTF-8 with U+FFFD.
Framebuffer::highlight_sanitized then paints a yellow background over each replacement. It is
also careful to tell an authored U+FFFD apart from a malformed sequence, so a genuine
replacement character in the file is left alone.

So two items from this thread are already done: Frulfump's U+007F, and malformed Unicode.

I probed the rest of the list against the current build. Width is what the layout engine
measures, "today" is what the sanitizer currently does:

character	category	width	today
U+0001, U+0085	C0 / C1 control	1	replaced, yellow
U+007F	DELETE	1	replaced, yellow
invalid UTF-8	malformed	n/a	replaced, yellow
U+00A0, U+1680, U+2000..U+200A, U+202F, U+205F	non-ASCII whitespace	1	untouched
U+2028, U+2029	line / paragraph separator	1	untouched
U+3000	ideographic space	2	untouched
U+061C, U+200E, U+200F, U+202A..U+202E, U+2066..U+2069	bidi controls	0	untouched
U+200B, U+200C, U+200D, U+2060	zero width space, joiners, word joiner	0	untouched
U+FEFF	embedded BOM	0	untouched
U+E0000 block	tag characters	0	untouched

The constraint that decides the rest

There are two layers. TextBuffer::render measures the original bytes and derives columns,
cursor and selection from them. Framebuffer::replace_text substitutes afterwards. A
substitution is therefore only safe when it has the same width as the character it replaces. The
existing code is written to that rule and says so out loud: C1 controls become U+2426 rather than
the U+FFFD that Unicode recommends, with the comment "Unicode says to use U+FFFD, but that one is
ambiguous width."

Measured widths split the request into two groups.

Group A is already width 1. Non-ASCII whitespace and the line and paragraph separators go
through the existing path unchanged: a table of code points, a stand-in glyph each, and the
yellow highlight comes for free. No layout risk, and it is roughly the size of the C1 case that
is already there. U+3000 is the exception at width 2, so it either needs a two cell stand-in or
it sits out of a first pass.

Group B is width 0 and cannot be substituted at all. crates/unicode-gen/src/main.rs forces
every gc=Cf character to zero width and rewrites its cluster break to Extend, with a comment
explaining that terminals have no zero width columns. Putting a one cell picture where one of
these sits would make the framebuffer line wider than the layout the buffer already computed, and
the cursor, the selection rectangle and the syntax highlight spans would all drift by a column.

Two shapes for Group B, and the tradeoff

Because those characters are Extend, they are already attached to a neighbouring grapheme
cluster. So the option with no layout consequences is: do not substitute anything, just paint that
cluster's cell. highlight_sanitized already works on rectangles, so the mechanism is in place.
The cost is that the cell cannot say which invisible it was, only that one is there, and a BOM at
offset 0 has no preceding cluster so it needs its own answer.

The other shape is to give these characters width 1 while the option is on, which is the only way
to show a distinct picture for each. The table has room for it: bits 13..15 of the u16 STAGE3
values are unused (the largest value in the tree is 0x1DCE), and the generator already reads the
UCD XML, which carries the properties this would key off. The catch is that it changes measured
widths, and AMBIGUOUS_WIDTH in the same module is documented as set once per process and never
changed afterwards. So this variant would have to be a startup setting rather than a live toggle,
unless that assumption is revisited. Worth knowing before anyone commits to a shape.

I lean towards the first for a first pass and the second only if you want the per character
detail, your call.

Normalization Form C

UTS #55 §4.2.3 lists three independent options: whitespace, default ignorable code points, and
text not in Normalization Form C. The first two are the two groups above. NFC is a different kind
of check: it needs a normalization implementation, which edit does not carry today, and it flags
runs of ordinary visible text rather than individual characters, so it wants a different
affordance. I would leave it out of this issue.

Confusables

Agreed with Frulfump that these belong in their own issue. They need the confusables data, and
the judgement is per identifier and per script mixture rather than per character, so almost none
of the machinery above applies.

On the affordance, since edit has no hover

I do not think it needs one. VS Code uses a hover because its squiggle carries no information by
itself. Here the character is replaced by a legible stand-in and the cell turns yellow, which is
already the whole message, and it has been shipping for control characters without a hover. So my
suggestion is to keep exactly that and add nothing new.

For the toggle: settings.json currently has one key in it (files.associations), so I would not
open a settings surface for this. A View menu checkbox next to Word Wrap, via the existing
menubar_menu_checkbox, matches what is already there.

Leonard Hecker (@lhecker) if this is roughly the direction you want, I can send Group A on its own, since it is
the existing mechanism plus a table, and hold Group B until I know your preference. If you would
rather this stayed smaller, or larger, let me know.