fix(backtest,multi-vm): handle MT5 relaunch, per-VM healthchecks, orphan sweep

[Marinski]

Five bug fixes that surfaced while running backtests on a multi-terminal deployment: backtests on terminals that had in fact run them were being reported as failed, and the container healthcheck was permanently red, masking a genuine outage.

What and why

Backtest reliability

• MT5 self-relaunch on update. When a LiveUpdate is pending, the launched terminal process exits 0 within seconds and MT5 relaunches itself to run the real test. The job was failed as "Report not generated" even though a valid report was written minutes later. The runner now waits for the replacement process and its run, bounded by the existing job timeout; a run long enough to be a real backtest is not an update restart and still fails immediately.
• INI values treated as literals. ConfigParser interpolation rejected bare % characters in symbols, report names and percentage inputs, causing a spurious 400 on submission. RawConfigParser is now used on both the read and write sides, matching how MT5 treats these values.

Multi-VM healthchecks

• check_health.py probed every port in config.yaml. On a multi-VM install each VM reported the other VM's terminals DOWN and the container sat permanently unhealthy, making a genuine failure indistinguishable from standing noise (observed: a 25,000+ failing streak while all local terminals were serving). It now applies the same per-VM group filter the launcher uses, with a fallback to no-filter if the import fails.
• healthcheck.sh had the same bug — it grepped every port: in config.yaml. It now filters by the per-VM group file docker-compose already bind-mounts, using awk since the container image has no python. No group file means no filter, preserving single-VM behaviour.

Startup performance

• sweep_orphans() parsed every job file at boot. On a large shared backtest-jobs dir this took ~2.5 minutes per process, multiplied by the number of processes starting at boot — all to mark in-flight jobs failed. It now inspects only state files touched within BACKTEST_SWEEP_LOOKBACK, and a new prune_old_jobs() retires completed/failed jobs older than BACKTEST_JOB_RETENTION, gated by a shared marker so only one process scans per interval. Both run in a background daemon thread so boot never blocks, and writes are atomic (tmp+rename) so concurrent sweepers never see truncated files.

Testing

• Unit suite updated to cover relaunch handling, INI literals, per-VM healthcheck filtering, and job sweep/prune. 241 tests pass.
• Lint clean.

[psyb0t]

Approving. Nice work — the diff is ~1:1 code to tests, and the comments explain
why rather than restating the code, which made this reviewable.

Verified rather than eyeballed:

• All four suites green on the PR head: make lint, make test-unit (244
  passed), make test-integration, and make test-go (the Go client is
  byte-identical to master, so it's unaffected).
• Mutation-tested the fixes — each one broken on purpose fails the test that
  claims to cover it, so the tests aren't decorative:
  • relaunch wait disabled → test_fast_clean_exit_without_report_waits_for_a_relaunch
  • RawConfigParser back to ConfigParser → both INI tests, with exactly
    invalid interpolation syntax in 'EUR%USD'
  • sweep mtime filter disabled → test_sweep_skips_stale_files and
    test_sweep_honors_custom_lookback
• parse_duration_to_seconds is shared — it also parses utc_offset, the
  wickworks timeout, and the user-supplied backtest timeout — so adding a unit
  to it deserved proof rather than reasoning. Diffed the old and new parser
  over 50 inputs: 42 identical, 8 additive (the new d forms), 0
  regressions.
• Ran the new healthcheck.sh awk against a real multi-terminal config: no
  group file → all ports; a two-entry group → exactly those two; an empty
  group file → falls back to all (the case you called out in the comment);
  missing file → falls back to all; and instance clones sharing a
  broker/account are distinguished correctly.

Four follow-ups, none blocking — happy for these to be a separate PR:

1. The two health-filter tests assert source text, not behaviour.
   test_check_health_probes_only_this_vms_terminals checks
   "_in_group(t, allowed)" in src, and
   test_container_healthcheck_probes_only_this_vms_ports asserts an exact awk
   line, 'if (!have_group) { print port; return }' in src. They catch
   deletion but would pass on a filter that is present and wrong, and a
   whitespace-only reformat breaks them while the behaviour is fine. Running
   the awk against a fixture config plus a fixture group file and asserting the
   emitted ports is only a few lines and covers the thing that actually
   matters. This is the one place where the rest of the PR's test quality
   dips.

2. _remove_job_state builds a path from the jobId read out of the state
   file, then shutil.rmtrees it. Safe today because jobId is always a
   server-generated uuid.uuid4().hex, but it is an unbounded delete keyed on
   a value read back from a directory several processes write to. Rejecting
   anything that isn't 32 hex chars before the join would make that
   independent of the writer.

3. The ImportError fallback in check_health.py silently disables the
   filter, which reinstates exactly the permanently-unhealthy behaviour this
   PR fixes — with nothing in the output to say so. A single line on that path
   would make the degradation visible instead of looking like normal
   operation.

4. The 24h sweep lookback changes recovery semantics. A job left in-flight
   by an outage longer than BACKTEST_SWEEP_LOOKBACK is no longer marked
   failed — it stays running for good, where the old full scan caught it.
   Realistically the caller has given up by then and the window is
   configurable, but it's a behaviour change worth a line in the docs.

Separately, two pre-existing issues surfaced while reading this — both predate
the PR, no action needed here, just flagging them:

• parse_duration_to_seconds("inf") (also "1e400") passes the bare-number
  fast path and then raises OverflowError from int(round(inf * 3600)). The
  submission handler only catches ValueError, so it escapes as a 500 rather
  than the intended 400. Reachable through the timeout form field.
• timeout_seconds is never clamped, so a request can set an arbitrarily long
  timeout and hold RUN_LOCK on that terminal for as long as it likes.

[psyb0t]

Merged — thanks a lot for this one. 🙏

Genuinely good work, and I want to name the specific things rather than just say
"nice PR":

• You found five real bugs from actually running the thing in anger, and wrote
  each one up with the evidence — the 25,272-long failing streak, the build 6090
  timings (exit T+3s, replacement T+9s, report at T+5m34s). That is the
  difference between a fix and a guess, and it made the review possible.
• Roughly one line of test per line of code, and the tests hold up: I broke each
  fix on purpose and the matching test went red every time.
• The comments explain why, not what. The note about avoiding the FNR==NR
  idiom because an empty group file would misread the config as the group list
  is the kind of thing most people find in production six months later, not
  while writing it.
• Keeping RawConfigParser symmetrical across both the read and write sides,
  and making the writes atomic in the same PR that introduces a background
  sweeper, shows you were thinking about how the pieces interact rather than
  just fixing them one at a time.

The follow-ups in my review are all small and none of them held this up. The
main one, if you feel like it: those two health-filter tests assert substrings
of the script source, so they'd pass on a filter that exists and is wrong.
Everything else in the PR is tested behaviourally — worth bringing those two up
to the same bar.

Cheers.