Repository navigation
Conversation
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
There was a problem hiding this comment.
🔍 Devin Review: 2 flags
Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #7172 +/- ##
==========================================
+ Coverage 53.35% 53.49% +0.14%
==========================================
Files 462 462
Lines 58534 58539 +5
Branches 8632 8633 +1
==========================================
+ Hits 31230 31315 +85
+ Misses 25180 25089 -91
- Partials 2124 2135 +11
... and 3 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
32b81e9 to
047d8ef
Compare
FileSource(path=...) leaves file_format as None. The DuckDB write path, FileSource schema inference and the Dask store read such a source as Parquet, but the DuckDB offline store did not: - _read_data_source used Parquet only when the path ended in ".parquet". For any other path, such as a directory of Parquet files (which the DuckDB write path itself creates), it fell through and returned None, and retrieval failed later with "'NoneType' object has no attribute 'select'" (or 'mutate'). - _duckdb_sql_from_expression, used by compute_monitoring_metrics and get_monitoring_max_timestamp, accepted only an explicit ParquetFormat or DeltaFormat, so monitoring raised NotImplementedError for every FileSource without a file_format, including "data.parquet". Treat a missing file_format as Parquet in both places, and raise NotImplementedError naming the source for any other format instead of returning None. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Shivam Sharma <192242476+ShivamSharma43@users.noreply.github.com>
047d8ef to
d39563b
Compare
What this PR does / why we need it:
FileSource(path=...)leavesfile_formatasNone. Elsewhere in Feast such a source is read as Parquet: the DuckDB write path (offline_write_batch),FileSource.get_table_column_names_and_typesand the Dask offline store all do. The DuckDB offline store's readers don't:_read_data_sourcereads Parquet only whenfile_formatisParquetFormat, or when it isNoneand the path ends in.parquet. Any other path, for example a directory of Parquet files (which is what the DuckDB write path writes when the path doesn't end in.parquet), matches no branch, so the function returnsNone.pull_latest_from_table_or_query,pull_all_from_table_or_queryandget_historical_featuresthen fail withAttributeError: 'NoneType' object has no attribute 'select'(or'mutate')._duckdb_sql_from_expression, whichcompute_monitoring_metricsandget_monitoring_max_timestampuse, accepts only an explicitParquetFormatorDeltaFormat. Monitoring therefore raisesNotImplementedError: DuckDB monitoring compute supports Parquet and Delta file sources only.for everyFileSourcewithout afile_format, including the usualFileSource(path="data/driver_stats.parquet", ...).Changes:
file_format is Noneas Parquet, the same check the write path already uses._read_data_sourceraisesNotImplementedErrornaming the source for any other file format, instead of returningNone.Which issue(s) this PR fixes:
No issue filed; found while reading the DuckDB offline store.
Checks
git commit -s)Testing Strategy
Added
sdk/python/tests/unit/infra/offline_stores/test_duckdb_file_format.py. It reads aFileSourcewithout afile_formatfrom a directory of Parquet files and from a.pqfile, callspull_latest_from_table_or_queryon the directory source, runsget_monitoring_max_timestampandcompute_monitoring_metricson a.parquetsource, and checks that an unsupported format raises. All five tests fail on master and pass with this change.I also ran
tests/unit/infra/offline_stores,tests/unit/monitoringandtests/unit/infra/data_sources/test_lance_source.py. The only failures and collection errors are BigQuery, Redshift, Athena, Snowflake, Spark and Trino tests that need extras I don't have installed (boto3,google-api-core,pyspark, ...); the same tests fail the same way on master.ruff check,ruff format --checkandmypyon the changed files pass.Misc
This change was written with help from an AI assistant (Claude Code). I reviewed it and ran the tests above.
🤖 Generated with Claude Code