Diff breaks unicode characters for emojis

[orromis]

I'm working message log in our app. We want to show diff of changes any user made on text posts. Those posts may include emoji characters but diff_match_patch replaces those characters with � character (but only if they changed in the text).

The behaviour can be reproduced here:
https://neil.fraser.name/software/diff_match_patch/demos/diff.html

Paste 😉and 😀into textareas and compute the diff.

Why is this happening?

[mcataford]

Looks like the diffing doesn't consider whether a character is non-ASCII and breaks unicode emojis in pieces if they are different but occupy the same space. That would result in some unknown character � after all is said and done. Assuming it's conversion related.

Looking into it out of curiosity.

[josephrocca]

Looks like @yetanotherape has solved it for their PHP fork:

• Remove UCS-2LE encoding/decoding? yetanotherape/diff-match-patch#9
• yetanotherape/diff-match-patch@3e7b024

The two current attempts at solving it in this repo have both had complications:

• Use UTF-32 Aware String in JavaScript #13
• JS: Handle surrogate pairs correctly #69

[ndvbd]

Any updates here?

[josephrocca]

@ndvbd I ended up "solving" it by just escaping all the special unicode stuff with text = encodeURI(text) before saving the text (and using decodeURI(...) to undo it, of course). Bit of a hack but it works for my use case.

[dmsnell]

Check out #80, as referenced above @ndvbd's comment. It should handle all the surrogate pairs properly.