Skip to content

[C++] UnbufferedCharStream: don't treat U+FFFF as end of input - #4997

Open
Bhav-ya-H wants to merge 1 commit into
antlr:devfrom
Bhav-ya-H:cpp-unbuffered-charstream-ffff
Open

Bhav-ya-H wants to merge 1 commit into
antlr:devfrom
Bhav-ya-H:cpp-unbuffered-charstream-ffff

Conversation

@Bhav-ya-H

Copy link
Copy Markdown

UnbufferedCharStream::fill() and getText() still detect end of input by comparing the last buffered char against 0xFFFF, while #3420 moved LA() to std::char_traits<wchar_t>::eof(). On Linux and macOS wchar_t is 32 bits and wistream::get() reports end of input as WEOF (0xFFFFFFFF), so it is a literal U+FFFF in the input that makes fill() stop buffering. Lexer::nextToken() always holds a marker, so right after that character LA(1) returns EOF, the lexer sets hitEOF, and the rest of the input is dropped without a syntax error: a parser reading through UnbufferedCharStream accepts just the prefix, while the same text through ANTLRInputStream is lexed in full. With the in-tree XPathLexer (where � is a valid NameStartChar), a�b/c comes out as RULE_REF EOF through the unbuffered stream and RULE_REF ROOT RULE_REF EOF through ANTLRInputStream. All three checks now use one constant derived from std::char_traits<wchar_t>::eof(), so Windows (16-bit wchar_t, where WEOF is 0xFFFF) behaves as before and the -Wsign-compare warning from #4517 goes away. The new UnbufferedCharStreamTest fails on dev and passes with the fix, and the runtime builds warning-free in Debug (regular and unity) with Apple clang; I could not run gcc, MSVC or the Java-driven runtime-testsuite locally, though its C++ harness does not use this stream.

This fix was found and prepared with AI assistance and reviewed by me before submitting.

fill() and getText() still compared the last buffered char against
0xFFFF, while LA() uses std::char_traits<wchar_t>::eof(). With a 32-bit
wchar_t, WEOF is 0xFFFFFFFF, so a literal U+FFFF in the input stopped
buffering and, while a marker was held, everything after it was
reported as EOF. Use one constant derived from WEOF for all three
checks.

Signed-off-by: Bhavya H <bhavya@labs.digiscrypt.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant