fix(recipe): harden Hermes auto-heal recovery

[mkhaas]

Fixes NVIDIA/NemoClaw#7671

What changed

• require three consecutive failed Hermes health probes, two seconds apart, before recovery
• select exactly one Docker container by the OpenShell managed-by and sandbox-name labels
• stop the disabled local recovery launcher, terminate only known Hermes runtime processes inside that exact container, and keep the OpenShell container and supervisor running
• require the existing sandbox to remain Ready and OpenShell exec to run as the non-root sandbox user before relaunch
• repair only Hermes writable state and known legacy runtime files, without following symbolic links or modifying immutable configuration
• launch recovered Hermes through a disabled user service whose command is openshell sandbox exec --name ${SANDBOX_NAME} -- /usr/local/bin/nemoclaw-start
• require a new gateway PID and process start time, the sandbox UID, health, and the exact Slack policy before reporting recovery
• normalize Slack authorization exactly as sandbox creation does and reject a process that carries both allowlist and allow-all settings
• allow one recovery for a stale Slack policy, then require the documented tear-down and bring-up flow instead of creating a loop
• parse the Slack Socket Mode response through stdin so its temporary WebSocket ticket never appears in process arguments
• stop classifying functional Slack API errors such as missing_scope and not_in_channel as transport failures

Root cause

A single transient health-probe failure triggered recovery. The original watchdog then launched the full Hermes entrypoint with raw docker exec. OpenShell's Docker container runs its outer supervisor as root, so that path used the wrong identity and could leave a root-owned runtime tree.

OpenShell v0.0.72 does not persist or replay the trailing sandbox-create command when its Docker container restarts. Recovery must therefore leave the container and its OpenShell supervisor running, stop only known Hermes runtime processes, verify a non-root OpenShell exec session, repair bounded writable state, and relaunch Hermes through that managed session.

Security and attribution

• provider, proxy, and CA values come from the persisted OpenShell sandbox environment for the new exec session
• provider credentials and host runtime values are never added to recovery command arguments
• the runtime launcher refuses to start when a Hermes entrypoint or gateway is already present
• root ownership repair uses fixed paths and chown --no-dereference --recursive
• immutable /sandbox/.hermes configuration is excluded
• Meenakshi Kaushik's original signed commit remains intact; the first maintainer hardening commit also credits her as co-author
• all maintainer commits use Apurv Kumaria's NVIDIA identity and DCO sign-off

Validation completed

• bash scripts/autoheal/tests/test_watchdog.sh
• python3 -m unittest discover -s scripts/autoheal/tests -p 'test_*.py' (9 tests)
• Python compilation with a temporary bytecode cache
• Bash syntax checks on all affected shell scripts
• ShellCheck v0.11.0 with deliberate cross-file and literal-command exclusions only
• shfmt v3.12.0 with the repository style on all affected shell scripts
• systemd-analyze verify on the rendered runtime unit in disposable Linux
• disposable Docker lifecycle contract: exact label selection, unchanged container ID/start time, targeted Hermes termination, bounded ownership repair, no symlink-target traversal, and a live supervisor/PID 1
• full SPDX header check (252 files)
• maintainer taxonomy and governance tests
• git diff --check

Remaining external operational validation

Before merge, run the documented recovery matrix on a disposable Linux host with Docker-backed OpenShell and configured integrations:

• OpenShell exec runs as sandbox
• the runtime launcher starts exactly one new Hermes tree with a new gateway PID and process start time owned by the sandbox UID
• Hermes health returns HTTP 200
• Slack auth.test and apps.connections.open succeed, and the exact mutually exclusive Slack policy is present
• configured Outlook access and provider/CA trust still work
• tear-down and uninstall do not relaunch or duplicate the runtime
• logs contain no credentials or temporary Socket Mode ticket

Attribution and DCO

Original contribution by Meenakshi Kaushik (@mkhaas). Her original signed commit remains intact, and the maintainer hardening commit preserves her co-authorship credit.

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

[apurvvkumaria]

Maintainer hardening is ready for independent review at 5bc9414a01ae4dc6543ca069e61b66b0a676fb1b.

Local evidence is green: the focused watchdog command-contract test, 9 auto-heal Python tests, Bash syntax, ShellCheck v0.11.0, shfmt v3.12.0, SPDX, governance, and diff checks. Meenakshi's original signed commit remains intact, and the maintainer commit explicitly credits her as co-author.

@wscurran @ericksoa @cv: please review the managed OpenShell supervisor lifecycle and exact mutually exclusive Slack policy handling. This PR still needs disposable Docker/OpenShell live evidence before merge: observed container lifecycle change, non-root workload identity, health 200, Slack and configured Outlook provider continuity, and the documented ownership/mode checks. Please do not merge based on local mocks alone.

[apurvvkumaria]

Maintainer correction at e93e25935e920c98d0339304484c8dd4ec37b628 supersedes my earlier lifecycle description.

The prior head incorrectly assumed that OpenShell v0.0.72 would replay the sandbox-create command after a Docker restart. It does not. The corrected recovery now:

1. selects exactly one container by OpenShell labels;
2. stops the local recovery launcher and restarts that exact container to clear legacy root and non-root process trees;
3. waits for the OpenShell supervisor and non-root sandbox exec identity;
4. repairs only known writable paths;
5. launches Hermes with openshell sandbox exec through the disabled runtime unit; and
6. verifies a new gateway PID/start time, sandbox UID, health, and exact Slack policy.

The command-contract test, 9 Python tests, Bash syntax, ShellCheck 0.11.0, scoped shfmt 3.12.0, Python compilation, SPDX, governance, and diff checks pass. An independent static audit found no remaining code-level blocker.

This PR is still not ready to merge until the disposable Linux/OpenShell v0.0.72 validation matrix in the PR description passes, including systemd-analyze verify, one-runtime-tree proof, Slack and configured Outlook continuity, CA/provider trust, ownership/modes, and tear-down/uninstall behavior.

@wscurran @ericksoa @cv: please review the corrected lifecycle at this exact head, but do not approve it for merge based on mocked validation alone.

[apurvvkumaria]

New cross-PR lifecycle evidence must be resolved before this exact head can merge.

PR #91 at 901a6285d34b28efe90acb26f7b625f224602732 documents that raw docker restart reuses a static one-hour OpenShell bootstrap JWT, can fail with ExpiredSignature, crash-loop the container, and leave the sandbox in Provisioning. This PR at e93e25935e920c98d0339304484c8dd4ec37b628 relies on raw docker restart --time 15 as the first recovery step.

Please treat this as part of the existing disposable OpenShell v0.0.72 live gate. The validation must demonstrate the exact restart path after the bootstrap token is older than one hour, show that the supervisor returns without ExpiredSignature, and prove the sandbox reaches Ready before the Hermes relaunch. If the stale-token behavior reproduces, the restart-based recovery must be replaced with an OpenShell-owner-supported lifecycle. Do not approve or merge until the two operational contracts are reconciled with captured live evidence.

[apurvvkumaria]

The stale-bootstrap-token recovery defect has now been addressed at d2aa95a: raw container restart is removed, the existing OpenShell supervisor remains in place, recovery stops only the known Hermes runtime, verifies non-root sandbox execution, repairs bounded writable paths, and relaunches through the managed exec path. Focused local tests, SPDX, governance, and whitespace checks pass.

The following external gates still block merge:

1. Please add your own DCO declaration to the PR description exactly as follows so the current required check can attest the contribution:
   Signed-off-by: Meenakshi Kaushik <1818024+mkhaas@users.noreply.github.com>
2. Update the PR description to remove the now-stale raw-container-restart and lifecycle-marker claims.
3. Provide disposable Linux/OpenShell end-to-end evidence for the updated recovery path: non-root relaunch, exactly one new Hermes gateway/runtime tree, health 200, Slack Socket Mode and configured Outlook/provider/CA continuity, bounded ownership/mode repair, tear-down/uninstall behavior, and logs free of credentials or temporary Socket Mode tickets.

This remains changes-requested until the contributor DCO declaration and live operational proof are present. The original contributor commit and attribution remain intact.

Superseded by maintainer follow-up at 7ecd97a: the approved maintainer DCO/attribution approach, PR description correction, and local/disposable Linux validation are complete. A narrower current review records the sole remaining external operational gate.

[apurvvkumaria]

Maintainer follow-up at 7ecd97a: the PR now uses Apurv Kumaria’s valid DCO sign-off while preserving Meenakshi Kaushik’s original signed commit and explicit attribution. The stale raw-container-restart claims are removed. All required GitHub checks pass, the rendered systemd unit verifies on Linux, and a disposable Docker contract proves exact container selection, unchanged container lifecycle, targeted Hermes termination, bounded ownership repair without symlink-target traversal, and supervisor survival. No remaining code-level change is identified.

One external operational gate still blocks merge: run the configured recovery matrix on a disposable Linux/OpenShell host and attach evidence for non-root relaunch, exactly one new Hermes tree, health 200, Slack Socket Mode, configured Outlook/provider/CA continuity, teardown/uninstall behavior, and secret-safe logs. This environment has no registered OpenShell gateway or Slack/Outlook/provider credentials, so that live integration evidence cannot be produced here.