[SPARK-59628][SQL] Reject ASOF JOIN without MATCH_CONDITION instead of a silent plain join - #59001
LukaZdravic wants to merge 2 commits into
Conversation
…f a silent plain join ASOF JOIN requires MATCH_CONDITION. Without it, and with no alias on the left table, the parser read ASOF as that table's alias and built a plain INNER JOIN, so the query ran and returned wrong results with no error. Move ASOF into strictNonReserved so it can no longer be a table alias. The query now fails at parse time. ASOF stays usable as a column and table name.
With spark.sql.ansi.enforceReservedKeywords=true, the parser still read ASOF as the alias of the left table, because ASOF was in ansiNonReserved. So a missing MATCH_CONDITION still ran as a plain join, and MATCH_CONDITION (flag) parsed as a join with no condition. Remove ASOF from ansiNonReserved, so it is reserved in that mode. Run the parser test in both keyword modes and pin the plans. Use a USING golden case, which ran as a plain join before this change. Co-authored-by: Isaac <no-reply@databricks.com>
| SELECT * FROM trades t ASOF JOIN quotes q ON t.symbol = q.symbol; | ||
|
|
||
| -- FVT-ASOF-1-020: missing MATCH_CONDITION with no left alias is rejected, not a plain join | ||
| SELECT * FROM trades ASOF JOIN quotes USING (symbol); |
There was a problem hiding this comment.
- what about
AS ASOF? - what about
AS ASOFwithMATCH_CONDITION?
There was a problem hiding this comment.
When ANSI is off, AS gets treated as an alias to the table because it's a non-reserved word, and that means
FROM trades AS ASOF JOIN quotes USING (symbol) this fails because it's an asof join
FROM trades AS ASOF JOIN quotes MATCH_CONDITION (trades.trade_time >= quotes.quote_time) USING(symbol) this fails because it expects the AS to be an alias, so no trades.trade_time allowed
When ANSI is on, AS is a reserved word so every attempt at writing AS ASOF is failing.
I should add goldens so we solidify this behaviour for someone else who may come in contact with this, what do you think?
| |ASC|non-reserved|non-reserved|non-reserved| | ||
| |ASENSITIVE|non-reserved|non-reserved|non-reserved| | ||
| |ASOF|non-reserved|non-reserved|non-reserved| | ||
| |ASOF|reserved|strict-non-reserved|non-reserved| |
There was a problem hiding this comment.
ANSI mode should probably match SQL-2016, like in ANTI case
There was a problem hiding this comment.
ASOF cannot be non-reserved, because it can then be read as a table alias, same as the AS we talked about above. However ANTI that you mention has the same problem SELECT * FROM t ANTI JOIN u USING (k) this return the inner join rows not the anti join rows. Definitely worth a follow up, what do you think? Also probably an investigation as to what else can be written normally and have different behaviours
What changes were proposed in this pull request?
Stop the parser from reading
asofas a table alias:ASOFbecomes strict-non-reserved.spark.sql.ansi.enforceReservedKeywords=true):ASOFbecomes reserved.Why are the changes needed?
The parser read
asofas the alias of the left table, so these queries became plain joins with no error:Now the first fails with
PARSE_SYNTAX_ERROR. The second fails withASOF_JOIN_MATCH_CONDITION_INVALID_OPERATOR.Does this PR introduce any user-facing change?
Yes.
asofcan no longer be an unquoted table alias, for exampleFROM t asof. That worked in Spark 4.2.0. In ANSI keyword mode,asofis reserved. Use`asof`instead.How was this patch tested?
New tests in
PlanParserSuite(both keyword modes) andjoin-asof-grammar.sql. Regenerated the golden files.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code