Repository navigation
Conversation
A declared vector_length on an Array field with vector_index=False is now enforced on both the DataFrame write path and the Arrow materialization path, instead of being stored and ignored. Fields that declare no length are unchanged, and the vector-index field keeps its existing behaviour and error messages. Refs feast-dev#7185 Signed-off-by: Nick Quinn <nicholas_quinn@apple.com>
| for field in _get_feature_view_fixed_length_fields(feature_view): | ||
| _validate_list_field_length(table, feature_view, field, "List") |
There was a problem hiding this comment.
🟡 All-null lists block valid writes
When a fixed-length non-vector column contains only None, _validate_list_field_length rejects its inferred Arrow null type. The ingestion preparation creates that type from the DataFrame. Valid writes with other populated features fail.
Learn more
The newly checked non-vector fields can contain null values without violating their length constraint. The ingestion preparation converts a DataFrame to Arrow without supplying a schema, so an object column containing only None becomes Arrow null, not a list type. The Arrow length helper rejects that type instead of treating it as having no lengths to compare. Other populated features prevent the empty-feature guard from skipping the write.
Example: Define embedding as Array(Float32) with vector_length=4. Write two rows with embedding=[None, None] and label=['a', 'b']. DataFrame validation passes, but Arrow validation raises has non-list Arrow type null before either label is stored.
Recommended fix: Accept Arrow null-typed columns as having no lengths to validate, while retaining rejection of non-null scalar columns. Add an ingestion regression test with an all-null fixed-length list and a populated second feature.
Was this helpful? React with 👍 or 👎 to provide feedback.
| for list_field in _get_feature_view_fixed_length_fields(feature_view): | ||
| self._validate_list_field_length(feature_view, df, list_field, "List") |
There was a problem hiding this comment.
🟡 Output constraints reject valid transformation inputs
For same-named ODFV inputs and outputs, _validate_list_field_length rejects raw lists whose transformation produces the declared length. The online-write preparation validates before transforming, so valid online writes fail.
Learn more
An on-demand feature view's schema describes its output, not its raw request data. The online-write preparation calls this validator before applying the transformation. The Python transformation path permits an output to replace a same-named input. Extending validation to non-vector lists therefore rejects valid raw inputs when the transformation changes their length.
Example: A non-singleton Python ODFV accepts embedding lists of length eight and returns their first four elements under the same name. Its output field is Array(Float32) with vector_length=4 and no vector index. Writing an eight-element input now raises before the UDF runs, although its four-element output satisfies the constraint.
Recommended fix: Validate ODFV output constraints after transformation, not against raw inputs. Preserve validation when transform_on_write=False, and add a write-path test where a Python transformation changes a same-named list's length.
Was this helpful? React with 👍 or 👎 to provide feedback.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7186 +/- ##
=======================================
Coverage 53.63% 53.64%
=======================================
Files 462 462
Lines 58537 58546 +9
Branches 8632 8634 +2
=======================================
+ Hits 31397 31406 +9
Misses 25005 25005
Partials 2135 2135
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
A declared vector_length on an Array field with vector_index=False is now enforced on both the DataFrame write path and the Arrow materialization path, instead of being stored and ignored. Fields that declare no length are unchanged, and the vector-index field keeps its existing behaviour and error messages.
Refs #7185
What this PR does / why we need it:
Which issue(s) this PR fixes:
Checks
git commit -s)Testing Strategy
Misc