Skip to content

fix sliced chunks and dangling state in C++ ParseTreePatternMatcher - #4993

Open
abdul7056 wants to merge 1 commit into
antlr:devfrom
abdul7056:cpp-pattern-matcher-lifetimes
Open

abdul7056 wants to merge 1 commit into
antlr:devfrom
abdul7056:cpp-pattern-matcher-lifetimes

Conversation

@abdul7056

Copy link
Copy Markdown

The C++ port of the tree pattern matcher cannot be used: every call into it is undefined behaviour.

  • split() keeps TextChunk/TagChunk objects in a std::vector, so they are sliced to the base class and tokenize() reads the text of a chunk that is not there (SEGV under ASan for any pattern with a tag)
  • compile() parses the pattern with a ParserInterpreter and CommonTokenStream that are locals, so the returned ParseTreePattern refers to a freed tree and freed tokens; text-chunk tokens likewise point at a stack ANTLRInputStream, and match(tree, string, ruleIndex) returns a ParseTreeMatch holding a reference to a local pattern
  • Lexer::reset() dereferences the null input that tokenize() leaves behind, so a pattern with two text chunks crashes before any of the above is reached (the Java runtime guards this)

The pattern now owns the parser that created its tree, and through it the tokens; chunks are held through unique_ptr; token text is copied while the temporary stream is alive. split() therefore returns std::vector<std::unique_ptr>, which no working caller can depend on today since the old signature could only return sliced objects.

New gtest compiles and matches patterns against an interpreted grammar: it crashes on dev and passes under ASan with this change. The C++ runtime testsuite (362 tests) passes.

ParseTreePatternMatcher::split stored TextChunk and TagChunk objects by value
in a std::vector<Chunk>, slicing them to the base class, so tokenize() could
never recognise a tag and read a TextChunk's text out of bounds from the
sliced object. compile() then built the pattern tree with a ParserInterpreter
and CommonTokenStream that were locals, so the returned ParseTreePattern
pointed at a freed tree and freed tokens, the text-chunk tokens pointed at a
stack ANTLRInputStream, match(tree, string, ruleIndex) returned a
ParseTreeMatch referencing a local pattern, and Lexer::reset() dereferenced
the null input left behind by tokenize(), which crashed on the second text
chunk of a pattern.

Keep chunks in unique_ptrs, copy token text while the temporary stream is
alive, let compile() parse with a ParserInterpreter that owns its token
source and stream and hand it to the ParseTreePattern so the tree and tokens
live as long as the pattern, store the pattern by value in ParseTreeMatch,
and guard the null input in Lexer::reset() as the Java runtime does.

Signed-off-by: obsidian_nomad <abdul@sproutxp.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant