Skip to content

Avoid capturing default config in numeric parser UDF#110

Open
PhotonData-oss wants to merge 1 commit into
AbsaOSS:masterfrom
PhotonData-oss:fix-date-udf-default-config-serialization
Open

Avoid capturing default config in numeric parser UDF#110
PhotonData-oss wants to merge 1 commit into
AbsaOSS:masterfrom
PhotonData-oss:fix-date-udf-default-config-serialization

Conversation

@PhotonData-oss

Copy link
Copy Markdown
Contributor

Summary

  • copy the configured error-code values into a serializable BasicErrorCodesConfig before building the numeric parser UDF
  • pass the narrower ErrorCodesConfig through UDFResult.fromTry, while preserving the existing StandardizationConfig overload
  • add a serialization regression covering DefaultStandardizationConfig

Verification

  • git diff --check

Scala/Spark tests were not run locally because this environment does not have Java, sbt, or Maven installed (java, sbt, and mvn are not available on PATH).

Fixes #57

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standardization can fail with Task not serializable when casting to date is requested

2 participants