Skip to content

[DO NOT MERGE] Reproduce flaky IVF-RaBitQ build_host_input_search (dim = 1) - #2791

Closed
bdice wants to merge 4 commits into
NVIDIA:mainfrom
bdice:repro-flaky-ivf-rabitq-dim1
Closed

bdice wants to merge 4 commits into
NVIDIA:mainfrom
bdice:repro-flaky-ivf-rabitq-dim1

Conversation

@bdice

@bdice bdice commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

Reproduces the flaky NEIGHBORS_ANN_IVF_RABITQ_TEST failure IvfRabitq/f32_f32_i64.build_host_input_search/1 (dim = 1) from the 2026-09-30 nightly (job):

CUDA error encountered at: .../ivf_rabitq/gpu_index/quantizer_gpu.cu line=538 function=...data_transformation_batch_opt(...): call='cudaPeekAtLastError()', Reason=cudaErrorInvalidValue:invalid argument

CI is reduced to the failing configuration (13.3.0, arm64, ubuntu26.04, GH200), 4 copies, running the IVF-RaBitQ test repeatedly with extra diagnostics.

Part of #2783 (workstream G).

@copy-pr-bot

copy-pr-bot Bot commented Oct 10, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@bdice

bdice commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test ea47e50

bdice added 2 commits October 10, 2026 17:56
k-means can leave lists empty, for example when the dataset has fewer distinct rows than n_lists. Building from device data, or from host data that fits on the device, then launched the quantization kernels with zero blocks and failed with cudaErrorInvalidValue. The streaming build skipped empty lists, which left their rotated centroids uninitialized.

Compute only the rotated centroid for an empty list and skip the kernels that have no rows to process. Add a regression test with fewer distinct rows than lists.
@bdice

bdice commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 8aff73e

@bdice

bdice commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

Closing: this was a reproducer/diagnostic PR for workstream G of #2783. The fix is in #2798.

@bdice bdice closed this Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant