Revisit Solr JVM GC flags: pinned young generation disables the G1 pause-time goal; collect GC logs under load

[gaurav]

Summary

The Solr JVM GC flags used by the NameRes Helm chart (helm/name-lookup/values.yaml) look cargo-culted and are internally contradictory: they pin the G1 young generation to a fixed size and set G1's adaptive young-gen percentages, which cannot both apply. Pinning the young gen also silently disables the MaxGCPauseMillis pause-time goal — so the one latency knob in the config does nothing.

Before changing flags I'd like to collect GC logs under representative load, since the current production picture can't be assessed from a status snapshot alone.

Current config

heap_mem: "-Xmx12G"     # note: no -Xms
gc: "-XX:NewSize=4G -XX:MaxNewSize=4G -XX:+UseG1GC -XX:MaxGCPauseMillis=1000 \
     -XX:+UnlockExperimentalVMOptions -XX:G1MaxNewSizePercent=40 -XX:G1NewSizePercent=5 \
     -XX:G1HeapRegionSize=32M -XX:InitiatingHeapOccupancyPercent=90"

Observations

1. Pinned young gen contradicts adaptive sizing and disables the pause goal.
-XX:NewSize=4G -XX:MaxNewSize=4G fixes the young generation, which (a) overrides -XX:G1NewSizePercent=5 -XX:G1MaxNewSizePercent=40 (they're effectively dead), and (b) per the HotSpot G1 docs, setting an explicit young-gen size disables the -XX:MaxGCPauseMillis pause-time heuristic. So MaxGCPauseMillis=1000 is inert, and we've opted out of G1's main strength (adaptive young-gen sizing to hit a pause target).

2. InitiatingHeapOccupancyPercent=90 disables adaptive IHOP (default since JDK 9) and starts concurrent marking very late (90% occupancy). Combined with a large heap this is usually fine, but it removes a safety margin against evacuation failures during allocation spikes.

3. The deployed heap doesn't match the chart default. Live status from https://name-lookup.ci.transltr.io/status?full=true (sampled 2026-06-02) reports heap.max ≈ 30 GB, not the -Xmx12G in values.yaml. We should reconcile what the CI/test/prod instances actually run. A 30 GB heap in a 32 GiB container leaves almost nothing for the OS page cache, which matters because the 142 GB index is mmap'd (this is the same page-cache concern tracked in #265 / #267).

4. Live heap is heavily underutilized — heap.used ≈ 3.86 GB / 30 GB (12.9%) — but the instance was idle at sampling (SystemCpuLoad 0.0, ~17K cache lookups since a 2026-05-15 start), so this only tells us the idle live set is small, not how the JVM behaves under load.

For reference, the same snapshot confirms the queryResultCache problem from #266 is real and worse than first reported: 12.99% hit ratio, 14,665 evictions, size 512, while filterCache is healthy (99.92% hit, 15 entries).

Proposed direction (pending GC-log validation)

• Stop pinning the young generation — drop -XX:NewSize/-XX:MaxNewSize and let G1 size it adaptively against the pause target. Keep G1NewSizePercent/G1MaxNewSizePercent as bounds or drop them and let G1 self-tune.
• Lower MaxGCPauseMillis toward the 200 ms default once it actually takes effect (search is latency-sensitive).
• Let IHOP go adaptive (remove =90) or set a more conventional value.
• Set -Xms == -Xmx to avoid runtime heap resizing.
• Reconcile heap size with the page-cache goal — the chart's 12 GB default leaves ~20 GB for page cache and is almost certainly better than a 30 GB heap; live data shows the live set is only a few GB.

Data to collect to settle this

Enable G1 GC logging on a Solr pod and capture under representative query load:

-Xlog:gc*,gc+heap=info,gc+age=trace:file=/var/solr/logs/gc.log:time,uptime,level,tags:filecount=10,filesize=20M

Then compare current flags vs. the de-pinned config on:

• GC pause distribution (p50/p99) and pause frequency
• young vs. mixed GC counts; any Full GCs or to-space/evacuation failures
• allocation rate and % time in GC
• effect of heap size on OS page-cache occupancy (ties into Surface OS page-cache stats (Buffers/Cached) in /status?full=true #267's /proc/meminfo exposure)

Related: #265 (heap reduction), #266 (queryResultCache), #267 (OS memory visibility).