Repository navigation
Conversation
Pathological queries in the warming corpus (match all scans, very large result sets, slow queries) are cheap enough under warm caches to go unnoticed in live traffic, but replaying them against a cold index on startup can stall warming and leave replicas in a restart loop. Add four layers of protection, all disabled by default: * Collection filter (SearchHandler, SearchStreamHandler): do not collect match all queries (warmer.filterMatchAllQueries), queries with startHit + topHits above warmer.maxQueryTopHits, or queries slower than warmer.maxQueryLatencyMs. * Backup-time filter (Warmer): re-apply the match all and top hits checks to the corpus before upload, including padding from the downloaded corpus. Downloaded queries that timed out or failed during replay are not used for padding. * Per-query timeout (Warmer): warmer.queryTimeoutMs sets the replayed request's collection timeout and timeoutCheckEvery so Lucene stops the query cooperatively, plus a request deadline covering later phases when deadline cancellation is enabled. Parallel warming is unaffected. * Total warming cap (ShardState): with warmer.maxWarmingDurationMs, the warmer stops starting new queries at the deadline and startup continues even if a query is still blocked. The warming thread is not interrupted, since interrupting Lucene IO can close shared index file channels. A replay query that throws is now logged and skipped rather than failing startup. Rejections are counted in nrt_warming_query_rejected_count, and cap hits in nrt_warming_duration_cap_exceeded_count. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Emit metrics for alerting on stale or failing warming query backups and for comparing startup warming across replicas: * nrt_warming_backup_success / nrt_warming_backup_failure (by reason: below_uptime_threshold, below_query_threshold, upload_error, other), counted in BackupWarmingQueriesHandler. * nrt_warming_queries (by type full/basic and outcome success/error/timeout) and the nrt_warming_query_latency_seconds histogram, recorded for every replayed query. Rename the guardrail counters to match: nrt_warming_query_rejected_count is now nrt_warming_queries_filtered, and nrt_warming_duration_cap_exceeded_count is now nrt_warming_duration_cap_reached. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The help text was copied from plugin_initialization_time_seconds. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Close gaps in the warming query filter: * topHits already includes startHit, so the top hits check no longer adds them. A query is rejected when topHits or startHit exceeds warmer.maxQueryTopHits, which still filters deep pagination without rejecting pages that end within the limit. * With warmer.filterMatchAllQueries, a match phrase query anywhere in the query tree with zeroTermsQuery ALL_ZERO_TERMS is rejected, since it matches all documents when analysis removes every term. * Multi retriever requests are checked per retriever: each text retriever query for match all and ALL_ZERO_TERMS, and each text retriever topHits against maxQueryTopHits. * New warmer.maxVectorCandidates (default 0, disabled) rejects queries with a knn query, top level or in a knn retriever, whose num_candidates exceeds the limit. Vector search runs during query rewrite, so the replay timeout cannot stop it. * Stored queries are checked before replay, so a corpus backed up before the filter was configured is not replayed unfiltered. Basic queries are also checked after simplification, since removing a top level function filter query leaves a match all query. New reason labels for nrt_warming_queries_filtered_total: all_zero_terms and vector_candidates. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pathological queries in the warming corpus (match all scans, very large result sets, slow queries) are cheap enough under warm caches to go unnoticed in live traffic, but replaying them against a cold index on startup can stall warming and leave replicas in a restart loop. Add four layers of protection, all disabled by default:
A replay query that throws is now logged and skipped rather than failing startup.
Metrics (registered in
WarmingMetrics, served on/status/metrics; the scraper addsservice/instancelabels):nrt_warming_backup_success_totalindexBackupWarmingQueriesHandler, after the upload succeedsnrt_warming_backup_failure_totalindex,reason=below_uptime_threshold|below_query_threshold|upload_error|otherBackupWarmingQueriesHandler, for each rejected or failed backup (other: index or warmer not found)nrt_warming_queries_totalindex,type=full|basic,outcome=success|error|timeoutWarmer, for every query replayed at startupnrt_warming_query_latency_secondsindex,typeWarmer, timer around each replayed querynrt_warming_queries_filtered_totalindex,stage=collection|backup|replay,reason=match_all|top_hits|latency|timeout|errornrt_warming_duration_cap_reached_totalindexShardState, when startup stops waiting atmaxWarmingDurationMsand the replica becomes ready with warming incompleteThe existing
warming_time_secondsgauge (BootstrapMetrics) still records total warming time per index. Its help text, which was copied fromplugin_initialization_time_seconds, is corrected.