Skip to content

feat: Enforce vector_length on non-vector list fields - #7186

Open
nquinn408 wants to merge 1 commit into
feast-dev:masterfrom
nquinn408:fixed-length-list-features
Open

nquinn408 wants to merge 1 commit into
feast-dev:masterfrom
nquinn408:fixed-length-list-features

Conversation

@nquinn408

@nquinn408 nquinn408 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

A declared vector_length on an Array field with vector_index=False is now enforced on both the DataFrame write path and the Arrow materialization path, instead of being stored and ignored. Fields that declare no length are unchanged, and the vector-index field keeps its existing behaviour and error messages.

Refs #7185

What this PR does / why we need it:

Which issue(s) this PR fixes:

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests
  • Integration tests
  • Manual tests
  • Testing is not required for this change

Misc


Devin Review

A declared vector_length on an Array field with vector_index=False is now enforced on both the DataFrame write path and the Arrow materialization path, instead of being stored and ignored. Fields that declare no length are unchanged, and the vector-index field keeps its existing behaviour and error messages.

Refs feast-dev#7185

Signed-off-by: Nick Quinn <nicholas_quinn@apple.com>
@nquinn408 nquinn408 self-assigned this Nov 19, 2026
@nquinn408
nquinn408 requested a review from a team as a code owner November 19, 2026 15:39

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Devin Review

Comment thread sdk/python/feast/utils.py
Comment on lines +2115 to +2116
for field in _get_feature_view_fixed_length_fields(feature_view):
_validate_list_field_length(table, feature_view, field, "List")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 All-null lists block valid writes

When a fixed-length non-vector column contains only None, _validate_list_field_length rejects its inferred Arrow null type. The ingestion preparation creates that type from the DataFrame. Valid writes with other populated features fail.

Learn more

The newly checked non-vector fields can contain null values without violating their length constraint. The ingestion preparation converts a DataFrame to Arrow without supplying a schema, so an object column containing only None becomes Arrow null, not a list type. The Arrow length helper rejects that type instead of treating it as having no lengths to compare. Other populated features prevent the empty-feature guard from skipping the write.

Example: Define embedding as Array(Float32) with vector_length=4. Write two rows with embedding=[None, None] and label=['a', 'b']. DataFrame validation passes, but Arrow validation raises has non-list Arrow type null before either label is stored.

Recommended fix: Accept Arrow null-typed columns as having no lengths to validate, while retaining rejection of non-null scalar columns. Add an ingestion regression test with an all-null fixed-length list and a populated second feature.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +3693 to +3694
for list_field in _get_feature_view_fixed_length_fields(feature_view):
self._validate_list_field_length(feature_view, df, list_field, "List")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Output constraints reject valid transformation inputs

For same-named ODFV inputs and outputs, _validate_list_field_length rejects raw lists whose transformation produces the declared length. The online-write preparation validates before transforming, so valid online writes fail.

Learn more

An on-demand feature view's schema describes its output, not its raw request data. The online-write preparation calls this validator before applying the transformation. The Python transformation path permits an output to replace a same-named input. Extending validation to non-vector lists therefore rejects valid raw inputs when the transformation changes their length.

Example: A non-singleton Python ODFV accepts embedding lists of length eight and returns their first four elements under the same name. Its output field is Array(Float32) with vector_length=4 and no vector index. Writing an eight-element input now raises before the UDF runs, although its four-element output satisfies the constraint.

Recommended fix: Validate ODFV output constraints after transformation, not against raw inputs. Preserve validation when transform_on_write=False, and add a write-path test where a Python transformation changes a same-named list's length.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

@codecov-commenter

codecov-commenter commented Nov 19, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 53.64%. Comparing base (1d6cfb4) to head (c1d05b2).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #7186   +/-   ##
=======================================
  Coverage   53.63%   53.64%           
=======================================
  Files         462      462           
  Lines       58537    58546    +9     
  Branches     8632     8634    +2     
=======================================
+ Hits        31397    31406    +9     
  Misses      25005    25005           
  Partials     2135     2135           
Flag Coverage Δ
go-feature-server 30.96% <ø> (ø)
python-unit 55.22% <100.00%> (+<0.01%) ⬆️
Files with missing lines Coverage Δ
sdk/python/feast/feature_store.py 50.47% <100.00%> (+0.11%) ⬆️
sdk/python/feast/field.py 81.51% <ø> (ø)
sdk/python/feast/utils.py 83.15% <100.00%> (+0.09%) ⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1d6cfb4...c1d05b2. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants