Skip to content

Make index-entry-to-record conversion an explicit contract - #4547

Merged
hatyo merged 1 commit into
mainfrom
apple/hatyo/index-entry-conversion
Sep 3, 2026
Merged

Make index-entry-to-record conversion an explicit contract#4547
hatyo merged 1 commit into
mainfrom
apple/hatyo/index-entry-conversion

Conversation

@hatyo

@hatyo hatyo commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Three physical plans read an index directly and have to hand back a record rather than the raw index entry: the covering index plan, the aggregate index plan, and the Lucene spell-check plan. They shared the mechanism for doing so — IndexKeyValueToPartialRecord, reached through a static helper in QueryPlanUtils — but not the thing they actually differ on, which is the shape an entry is decoded into. The covering plan decodes into a partial copy of the base record, resolving the descriptor from the record type. The aggregate plan decodes into the result of its select-having, whose aggregate column exists in no stored record, so it resolves a descriptor from the TypeRepository and looks its record type up differently as well. Because of that it could not use the shared helper and carried an inlined copy of it instead. The difference between the two was visible only as a difference between two blocks of code that otherwise look alike, which is an easy thing to get wrong while every individual step reads correctly.

IndexEntryToRecordConversion states that contract. An implementor converts a single entry through indexEntryToQueriedRecord, and does so via one of two named static helpers: intoStoredRecordShape for a partial copy of the record type, or intoShape when the target is a shape no stored record has. The choice is now said out loud rather than inferred from which lookup a plan happens to use. The pieces are passed as arguments rather than bundled into an object, since conversion runs once per index entry read. Making the Lucene spell-check plan an implementor too leaves QueryPlanUtils holding nothing, so it is removed, and RecordQueryCoveringIndexPlan's function-returning overload goes with it — its one caller in ComposedBitmapIndexQueryPlan now converts per entry, still with an empty evaluation context and a comment explaining why that suffices.

No behaviour change is intended, so the existing suites are the safety net: FDBCoveringIndexQueryTest, BitmapValueIndexTest, GroupByTest, FDBPermutedMinMaxQueryTest and the LuceneIndexTest spell-check cases all pass (116 tests), along with checkstyle and PMD on both touched modules. There is no new unit test — the remaining seam needs a record store, so it is covered by those FDB-backed tests rather than in isolation. Two things worth a reviewer's attention: QueryPlanUtils was public with no @API annotation, so if the team treats that as API surface this wants the breaking change label rather than enhancement; and conversion now resolves its pieces per entry rather than once per cursor, which is a repeated meta-data lookup — and, for the Lucene plan, a rebuilt converter. Nothing is allocated to wrap them, and memoizing is left to implementors with a note on the interface that anything cached must be scoped to one execution, since a plan may run against more than one store or context.

@hatyo hatyo added the enhancement New feature or request label Sep 2, 2026
@hatyo
hatyo marked this pull request as ready for review September 2, 2026 15:05
@hatyo hatyo added the Run mixed-mode Label to add to Pull Requests to have it run mixed mode tests label Sep 2, 2026
@hatyo
hatyo force-pushed the apple/hatyo/index-entry-conversion branch 2 times, most recently from 2afe76e to 459503c Compare September 2, 2026 16:22

@normen662 normen662 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice little improvement! One comment re: naming and/or trait conventions.

* </p>
*/
@API(API.Status.INTERNAL)
public interface IndexEntryToRecordConversion {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you name this something that makes it explicit it's a plan trait? So something ...Plan.... Also if I look at the existing class IndexKeyValueToPartialRecord, from the name I cannot discern which one does what.

Also, for plans I think we always make the trait extend RecordQueryPlan.

@hatyo hatyo Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, there are some exceptions in existing mixin / traits such that they extend nothing, but we could fix that later.

Three plans read an index directly and have to hand back a record rather than the raw
entry: the covering, aggregate and Lucene spell-check plans. They shared the mechanism --
IndexKeyValueToPartialRecord -- through a static helper in QueryPlanUtils and a field
convention, but not the choice they actually differ on, which is the shape an entry is
decoded into. The aggregate plan decodes into the result of its select-having, whose
aggregate column exists in no record, so it could not use the helper and grew an inlined
copy instead.

IndexEntryToRecordConversion states that contract. Implementors convert one entry through
one of two named helpers, so decoding into a shape no stored record has is said out loud
rather than inferred from a differing record-type lookup and descriptor. The pieces are
passed as arguments rather than bundled into an object, since conversion runs once per
index entry read.

The Lucene spell-check plan is now an implementor too, which leaves QueryPlanUtils with
nothing in it, so it is removed. RecordQueryCoveringIndexPlan's function-returning overload
goes with it: its one caller in ComposedBitmapIndexQueryPlan converts per entry instead.

No behaviour change intended.
@hatyo
hatyo force-pushed the apple/hatyo/index-entry-conversion branch from 459503c to b78b7ff Compare September 3, 2026 10:03
@hatyo
hatyo merged commit 5767737 into main Sep 3, 2026
22 checks passed
@hatyo
hatyo deleted the apple/hatyo/index-entry-conversion branch September 3, 2026 11:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Run mixed-mode Label to add to Pull Requests to have it run mixed mode tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants