Skip to content

fix: Read DuckDB file sources without a file_format as Parquet - #7172

Merged
ntkathole merged 1 commit into
feast-dev:masterfrom
ShivamSharma43:fix/duckdb-file-source-without-format
Nov 18, 2026
Merged

ntkathole merged 1 commit into
feast-dev:masterfrom
ShivamSharma43:fix/duckdb-file-source-without-format

Conversation

@ShivamSharma43

@ShivamSharma43 ShivamSharma43 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

FileSource(path=...) leaves file_format as None. Elsewhere in Feast such a source is read as Parquet: the DuckDB write path (offline_write_batch), FileSource.get_table_column_names_and_types and the Dask offline store all do. The DuckDB offline store's readers don't:

  1. _read_data_source reads Parquet only when file_format is ParquetFormat, or when it is None and the path ends in .parquet. Any other path, for example a directory of Parquet files (which is what the DuckDB write path writes when the path doesn't end in .parquet), matches no branch, so the function returns None. pull_latest_from_table_or_query, pull_all_from_table_or_query and get_historical_features then fail with AttributeError: 'NoneType' object has no attribute 'select' (or 'mutate').
  2. _duckdb_sql_from_expression, which compute_monitoring_metrics and get_monitoring_max_timestamp use, accepts only an explicit ParquetFormat or DeltaFormat. Monitoring therefore raises NotImplementedError: DuckDB monitoring compute supports Parquet and Delta file sources only. for every FileSource without a file_format, including the usual FileSource(path="data/driver_stats.parquet", ...).

Changes:

  • Both functions treat file_format is None as Parquet, the same check the write path already uses.
  • _read_data_source raises NotImplementedError naming the source for any other file format, instead of returning None.

Which issue(s) this PR fixes:

No issue filed; found while reading the DuckDB offline store.

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests
  • Integration tests
  • Manual tests
  • Testing is not required for this change

Added sdk/python/tests/unit/infra/offline_stores/test_duckdb_file_format.py. It reads a FileSource without a file_format from a directory of Parquet files and from a .pq file, calls pull_latest_from_table_or_query on the directory source, runs get_monitoring_max_timestamp and compute_monitoring_metrics on a .parquet source, and checks that an unsupported format raises. All five tests fail on master and pass with this change.

I also ran tests/unit/infra/offline_stores, tests/unit/monitoring and tests/unit/infra/data_sources/test_lance_source.py. The only failures and collection errors are BigQuery, Redshift, Athena, Snowflake, Spark and Trino tests that need extras I don't have installed (boto3, google-api-core, pyspark, ...); the same tests fail the same way on master. ruff check, ruff format --check and mypy on the changed files pass.

Misc

This change was written with help from an AI assistant (Claude Code). I reviewed it and ran the tests above.

🤖 Generated with Claude Code


Devin Review

@ShivamSharma43
ShivamSharma43 requested a review from a team as a code owner November 17, 2026 08:50
Copilot AI balanced review requested due to automatic review settings November 17, 2026 08:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Devin Review: 2 flags

Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 50.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 53.49%. Comparing base (2a312cc) to head (d39563b).
⚠️ Report is 4 commits behind head on master.

Files with missing lines Patch % Lines
sdk/python/feast/infra/offline_stores/duckdb.py 50.00% 0 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7172      +/-   ##
==========================================
+ Coverage   53.35%   53.49%   +0.14%     
==========================================
  Files         462      462              
  Lines       58534    58539       +5     
  Branches     8632     8633       +1     
==========================================
+ Hits        31230    31315      +85     
+ Misses      25180    25089      -91     
- Partials     2124     2135      +11     
Flag Coverage Δ
go-feature-server 30.96% <ø> (ø)
python-unit 55.06% <50.00%> (+0.15%) ⬆️
Files with missing lines Coverage Δ
sdk/python/feast/infra/offline_stores/duckdb.py 53.20% <50.00%> (+9.09%) ⬆️

... and 3 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 67cbcd5...d39563b. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

FileSource(path=...) leaves file_format as None. The DuckDB write path,
FileSource schema inference and the Dask store read such a source as
Parquet, but the DuckDB offline store did not:

- _read_data_source used Parquet only when the path ended in ".parquet".
  For any other path, such as a directory of Parquet files (which the
  DuckDB write path itself creates), it fell through and returned None,
  and retrieval failed later with "'NoneType' object has no attribute
  'select'" (or 'mutate').
- _duckdb_sql_from_expression, used by compute_monitoring_metrics and
  get_monitoring_max_timestamp, accepted only an explicit ParquetFormat
  or DeltaFormat, so monitoring raised NotImplementedError for every
  FileSource without a file_format, including "data.parquet".

Treat a missing file_format as Parquet in both places, and raise
NotImplementedError naming the source for any other format instead of
returning None.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Shivam Sharma <192242476+ShivamSharma43@users.noreply.github.com>
@ntkathole
ntkathole force-pushed the fix/duckdb-file-source-without-format branch from 047d8ef to d39563b Compare November 18, 2026 11:58
@ntkathole
ntkathole merged commit a037863 into feast-dev:master Nov 18, 2026
18 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants