[VL] Translate Java \uXXXX Unicode escapes to RE2 syntax to fix CJK regex fallback by KushShriv · Pull Request #13128 · apache/gluten

KushShriv · 2026-09-25T04:51:43Z

Problem

Gluten's Velox backend uses RE2 for regex execution. RE2 does not understand Java's \uXXXX
Unicode escape syntax, so patterns such as [\u4e00-\u9fa5] (CJK Unified Ideograph range) fail
to compile in RE2 and cause the entire operator to fall back to the Spark JVM engine. This was a
confirmed regression tracked in FallbackSuite.

Solution

Two-phase defence-in-depth translation of \uXXXX → \x{XXXX}:

Phase 1 — Scala (primary). Before a regex pattern literal is written into the Substrait
protobuf, ExpressionConverter now calls a new withRe2PatternTranslation() helper. This ensures
both the C++ validator and Velox's execution engine receive the already-translated string. The
helper is wired into all six regex expression types: RegExpReplace, RLike, RegExpExtract,
RegExpExtractAll, RegExpInStr, StringSplit. A new VeloxRegexUtils object (in
gluten-substrait) provides translateJavaPatternToRe2() and translateUnicodeEscapes().
Patterns with no RE2 equivalent (lookaheads, lookbehinds, backreferences) are detected and still
throw GlutenNotSupportException, preserving the existing graceful fallback for those cases.

Phase 2 — C++ (safeguard). validatePattern() in Common.cc now calls
translateJavaUnicodeToRe2() before the RE2::RE2() compile attempt. This is a last line of
defence for any future regex expression type added to kRegexFunctions whose Scala transformer is
not yet hooked into the Phase 1 helper. The function is idempotent — patterns already in RE2 syntax
pass through unchanged.

Changes

File	Change
`gluten-substrait/.../VeloxRegexUtils.scala`	New utility: `translateJavaPatternToRe2`, `containsUnsupportedConstruct`, `translateUnicodeEscapes`
`gluten-substrait/.../ExpressionConverter.scala`	New `withRe2PatternTranslation` helper; explicit cases for `RLike`, `RegExpExtract`, `RegExpExtractAll`; wired into all 6 regex expressions
`cpp/velox/utils/Common.h`	Declaration of `translateJavaUnicodeToRe2`
`cpp/velox/utils/Common.cc`	Implementation of `translateJavaUnicodeToRe2`; updated `validatePattern`
`cpp/velox/tests/CommonTest.cc`	New C++ unit tests for `translateJavaUnicodeToRe2` and `validatePattern`
`cpp/velox/tests/CMakeLists.txt`	Register `common_test` target
`backends-velox/.../FallbackSuite.scala`	Invert the `\u4e00` test — asserts native execution now, not fallback
`backends-velox/.../VeloxStringFunctionsSuite.scala`	Two new regression tests: Unicode escape runs natively; lookbehind still falls back

Testing

FallbackSuite — existing test updated; \uXXXX patterns in join post-filters now run natively
VeloxStringFunctionsSuite — new test confirms [\u0041-\u005A]+ uses ProjectExecTransformer; new test confirms (?<=x)y still falls back gracefully
CommonTest.cc — 13 unit tests covering translation correctness and validatePattern end-to-end

Notes

No new configuration keys introduced
No behaviour change for patterns that were already RE2-compatible
No behaviour change for unsupported constructs (lookaheads, lookbehinds, backreferences) — those still fall back via GlutenNotSupportException
Character class union/intersection ([a[b]], [a&&[b]]) unaffected — ensureRegexIsCompatible operates on the translated string, which has identical bracket depth to the original

Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-sonnet-4-5

…lidatePattern Adds translateJavaUnicodeToRe2() to cpp/velox/utils/Common.cc (.h) which converts Java \uXXXX Unicode escapes (exactly 4 hex digits) to RE2's \x{XXXX} syntax. The function is idempotent: patterns already in RE2 form pass through unchanged. validatePattern() now calls translateJavaUnicodeToRe2() before the RE2::RE2() compile attempt, so patterns like [\u4e00-\u9fa5] no longer cause a spurious compile failure in the native validator. Also adds CommonTest.cc (registered in CMakeLists.txt) with unit tests covering translateJavaUnicodeToRe2 and the updated validatePattern behaviour. Generated-by: Claude claude-sonnet-4-5

…pressionConverter Adds VeloxRegexUtils (gluten-substrait) with translateJavaPatternToRe2(), which converts Java \uXXXX Unicode escapes to RE2's \x{XXXX} form before the pattern is written into the Substrait protobuf literal. ExpressionConverter gains a private withRe2PatternTranslation() helper that is wired into the pattern argument of every regex expression: RegExpReplace, RLike, RegExpExtract, RegExpExtractAll, RegExpInStr, StringSplit. Previously, patterns such as [\u4e00-\u9fa5] caused a RE2 compile failure in the native validator and fell back to the JVM engine. With this change they are translated to [\x{4e00}-\x{9fa5}] before serialisation, so both the validator and the Velox execution engine see RE2-compatible syntax. Patterns that contain constructs with no RE2 equivalent (lookaheads, lookbehinds, backreferences) are detected by containsUnsupportedConstruct() and trigger GlutenNotSupportException, preserving the existing graceful fallback behaviour for those cases. Generated-by: Claude claude-sonnet-4-5

… patterns FallbackSuite: the test that asserted '\u4e00' patterns cause a fallback is inverted. The pattern now runs natively after the translation fix, so the test is renamed and asserts no 'rlike due to Pattern' fallback reason is present. VeloxStringFunctionsSuite: adds two regression tests: - 'rlike with Java Unicode escape runs natively' — asserts ProjectExecTransformer (native Velox path) is used for a \uXXXX pattern. - 'rlike with lookbehind falls back to JVM' — asserts (?<=x)y still falls back gracefully, confirming unsupported-construct detection is intact. Generated-by: Claude claude-sonnet-4-5

github-actions · 2026-09-25T04:52:14Z