Skip to content

Give stored fields priority over doc values during fetch - #1048

Open
vim345 wants to merge 2 commits into
Yelp:mainfrom
vim345:u/mohm/profiler
Open

vim345 wants to merge 2 commits into
Yelp:mainfrom
vim345:u/mohm/profiler

Conversation

@vim345

@vim345 vim345 commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

When a retrieved field is both stored and has doc values, fetch reads it twice or from the wrong place, depending on the fetch path:

  • FillDocsTask (default fetch and parallel fetch by document) reads it twice per hit. fetchSlice fills it from doc values, then the stored-document read fills it again and overwrites the doc-values result.
  • FillFieldsTask (parallel fetch by field) checks doc values first, so it returns doc values while the other paths return stored values.

This PR makes stored fields take priority everywhere, as discussed in review. Stored values keep the original order and duplicates, while doc values come back sorted (and de-duplicated for SORTED_SET). The changes:

  • fetchSlice skips the doc-values read for stored fields. They are filled once, from the stored document.
  • FillFieldsTask.getFieldForHit checks the stored case before the doc-values case.

Fields that are only stored or only have doc values are unaffected.

Behavior change

Default fetch and parallel fetch by document return exactly what they did before, minus the wasted doc-values read. Parallel fetch by field (parallelFetchByField: true) now returns stored values for these fields. For multi-valued fields that means original order instead of doc-values order, matching the other paths.

Tests

  • FillDocsTaskTest:
    • stored-only and stored + doc-values fields are both selected for the stored read;
    • fetchSlice returns the stored values (b, a, b) for a stored + doc-values field and never calls getDocValues.
  • StoredDocValuesFetchTest: a multi-valued ATOM field that is stored + doc-valued returns stored values (b, a, b), and a stored-only field returns its values. It runs three ways: the default fetch, parallel fetch by document, and parallel fetch by field. Each parallel variant first asserts that its fetch config took effect. Against main, only the by-field run fails (it returned a, b).
  • ./gradlew :test passes, including spotless.

Side note: MultiSegmentParallel*FetchTest set parallel fetch options as YAML keys that nothing reads, so MultiSegmentParallelFieldsFetchTest never runs FillFieldsTask. Worth a follow-up.

🤖 Generated with Claude Code

FillDocsTask filled fields that are both stored and doc-valued twice per
hit: fetchSlice read the value from doc values, then
getStoredFieldFetchContext also selected the field for the stored
document read, which decompressed a stored fields block and overwrote
the doc-values value. Only collect stored fields without doc values, so
these fields are read from doc values once, as FillFieldsTask already
does.

Multi-valued fields that are stored and doc-valued are now returned in
doc-values order for the default and parallel-by-document fetch, the
same as with parallel fetch by field.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@vim345
vim345 marked this pull request as ready for review October 9, 2026 17:08
@vim345
vim345 requested a review from aprudhomme October 9, 2026 17:10
Comment on lines +796 to +797
&& indexableFieldDef.isStored()
&& !indexableFieldDef.hasDocValues()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing that is useful about stored fields is that it preserves ordering, where doc values get sorted. It may be better to give stored fields priority when both are enabled, to allow retrieval of the unsorted values.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, thanks. I changed it so stored fields take priority whenever both are enabled, although not a single field enables them both right now.

Stored values keep the original order and duplicates, while doc values
are sorted (and de-duplicated for sorted set), so a field that is both
stored and doc-valued should return its stored values. FillDocsTask
already did, but also read the doc values first and discarded them, and
FillFieldsTask (parallel fetch by field) returned doc values instead.

Keep the stored read in FillDocsTask, skip the redundant doc values read
in fetchSlice, and check the stored case before doc values in
FillFieldsTask, so every fetch path returns stored values.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@vim345 vim345 changed the title Skip stored field reads for doc-valued fields in FillDocsTask Give stored fields priority over doc values during fetch Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants