Speed up and simplify Solr data loading with self-contained backups#278
Open
gaurav wants to merge 3 commits into
Open
Speed up and simplify Solr data loading with self-contained backups#278gaurav wants to merge 3 commits into
gaurav wants to merge 3 commits into
Conversation
… guard The Solr schema was defined twice -- in setup_solr.sh at load time and again in the Helm restore script at serve time -- and had already drifted. Make the checked-in configset (configsets/name_lookup/conf) the single source of truth, generated from a real Solr so behaviour is unchanged, with the malformed `types` field fixed, a larger index buffer, autoSoftCommit off, and a grown queryResultCache (closes #266). Rewrite setup-and-load-solr.sh to stream files to Solr in parallel with a single deferred commit (no per-file commit, no sleeps), and guard against dropped data by counting input documents before the load and comparing against Solr's count afterward, with curl --fail on every upload. The core is now created from the configset (see the Makefile / CI), so this script no longer sets up the schema and setup_solr.sh is deleted. Convert the test fixture to JSON-lines to match the Babel production format so CI exercises the line-counting guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Run Solr in standalone mode everywhere (no ZooKeeper for a one-node job): drop -cloud from the loader Dockerfile and -DzkRun from docker-compose and CI. The Makefile now creates the core from the checked-in configset, loads in parallel, optimizes the index before export (closes #256), and tars the whole core -- config, schema and index -- into a self-contained snapshot.backup.tar.gz. Restoring is now just "untar into the Solr home and start Solr", so the old replication-backup targets and the solr-restore/ helper are removed. CI creates the core from the configset and runs the parallel loader end to end. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rewrite data-loading/README.md around the self-contained standalone-core flow, with an "Options considered" section explaining why we chose it over the status quo and PR #249, and the list of issues it closes. Update the test-run commands in CLAUDE.md to create the core from the configset first. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Building a NameRes instance loads ~130 GB of Babel synonyms into Solr and ships a backup a worker restores. The old pipeline had three problems: a slow serial load (per-file hard commit +
sleep 60), a Solr schema defined twice (loader and the Helm restore script) that had already drifted, and ZooKeeper running for what is a one-shard, one-node job.This PR reworks the whole flow around a self-contained standalone core: the backup is a whole Solr core — config + schema + index — so restoring it is just "untar into the Solr home and start Solr". No ZooKeeper, no collection creation, no restore-time schema setup. This supersedes #249.
Key changes
data-loading/configsets/name_lookup/conf, generated from a real Solr (so behaviour is unchanged), with the malformedtypesfield fixed, a larger index buffer,autoSoftCommitoff, and a grownqueryResultCache.setup-and-load-solr.shstreams files to Solr in parallel with a single deferred commit (no per-file commit, no sleeps). Because a parallel load of a huge dataset is scary, it counts input documents before loading and compares against Solr's count afterward, withcurl --failon every upload — a drop or a bad file aborts with a non-zero exit.Dockerfile,docker-compose.yml, and CI drop cloud mode; Solr bumped to 9.10.0. The Makefile creates the core from the configset, optimizes the index, and tars the core intosnapshot.backup.tar.gz.setup_solr.shandsolr-restore/(obsolete).translator-devops, edited copy underdata/name-lookup/): standalone Solr, idempotentdownload.sh,restore.shgutted to blocklist-only, heap lowered for page cache.Validation
pytest: 12/12 pass against a Solr loaded via the new pipeline (this is what CI now runs end to end).Issues
Closes #238, closes #185, closes #256, closes #266. Addresses helxplatform/translator-devops#609. Progresses #265 (heap knob + smaller index; final heap/GC tuning and #272 still need load testing). Out of scope: #267, running natively on SLURM/HPC.
Note for reviewers
The
data/name-lookup/Helm chart is gitignored here (it's a working copy); its edits land intranslator-devopsseparately. PointdataUrlat a backup built by the new loader — old backups won't restore under the new flow.🤖 Generated with Claude Code