merge_semantic_meta_data runs once per loader Job instead of once per load

[gaurav]

mode: load in the loader Helm chart creates one Kubernetes Job per compendium file, and each Job's load_all calls merge_semantic_meta_data() at the end. So the merge — which reads every file-* key and re-sums all source-prefix counts — runs dozens of times per load, redundantly, and is what duplicates the semantic_types list (see companion issue).

It should run exactly once, after all compendium Jobs finish. Options: a separate final Job/step in the chart, or a loader entrypoint that only does the merge. Coordinate with the mode: load sharding in the translator-devops chart.

Context: node_normalizer/loader/loader.py merge_semantic_meta_data / load_all; documentation/Development.md.

[gaurav]

This is also a correctness race, not just redundancy

Because each Job runs merge_semantic_meta_data() concurrently, the per-Job runs race against each other. The function is an unguarded read-modify-write — no lock, no WATCH/MULTI, no Lua script:

KEYS file-*  →  GET each  →  sum into sources_prefix  →  SET <bl_type> <summed counts>

• Per-type counts (SET <bl_type>) are last-writer-wins and can undercount. Job A reads while 5 file-* keys exist and computes a sum; Job B reads while 8 exist and computes a larger sum. If A's SET lands after B's, the final biolink:Disease count reflects only the 5 files A saw. There's no guarantee the Job that SETs last is the one that read the complete set.
• semantic_types (LPUSH) accumulates duplicates — the same per-Job-run behavior that Store semantic_types as a Redis SET instead of a LIST #379 is about.

The file-<filename> keys themselves don't collide (each Job writes its own distinct key), and the aggregates feed only the statistics endpoints (/get_curie_prefixes, /get_semantic_types), not the normalization path — so it's cosmetic and tolerated for now.

Note the fix is the same one already proposed here (run the merge once, after all compendium Jobs finish): that removes the concurrency entirely and is cleaner than trying to make the read-modify-write atomic with WATCH/Lua. Documented in documentation/Development.md.