Skip to content

nodeos: NODEOS-001-LAB observability layer, tty1 console and evidence bundle - #4

Merged
Hazy142 merged 9 commits into
mainfrom
codex/nodeos-001-lab
Oct 7, 2026
Merged

nodeos: NODEOS-001-LAB observability layer, tty1 console and evidence bundle#4
Hazy142 merged 9 commits into
mainfrom
codex/nodeos-001-lab

Conversation

@Hazy142

@Hazy142 Hazy142 commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Summary

This PR implements the NODEOS-001-LAB pre-boot observability, telemetry, and evidence verification layer for the Haswell GTX 1070 Ti compute appliance.

Included Ergänzungen / Features:

  1. NodeOS Lab Console (
    odeos-console on ty1):

    • Lightweight POSIX-sh + ANSI full-screen TUI on ty1 (zero heavy dependencies, works without GPU drivers/network).
    • Real-time display of system build identity, granular gate progress, hardware metrics (CPU, RAM, GPU, VRAM, PCIe Gen/Width, Temp, Power, Clocks), fabric link status, and live log tail.
    • Interactive key shortcuts: [R]\ rerun gates, [N]\ iperf3 throughput test, [D]\ diagnostics, [E]\ export evidence bundle, [L]\ full log, [S]\ shell.

  2. Fine-Grained State Machine & Reason Codes:

    • Expanded state sequence: \BOOTING\ -> \KERNEL_READY\ -> \NIC_DISCOVERED\ -> \LINK_READY\ -> \GPU_DISCOVERED\ -> \NVIDIA_PAYLOAD_VERIFIED\ -> \DRIVER_READY\ -> \CUDA_PROBE_PASS\ -> \NODE_READY.
    • Distinct failure states: \BLOCKED_NO_NIC, \BLOCKED_NO_CARRIER, \BLOCKED_NO_IP, \BLOCKED_GPU_MISSING, \BLOCKED_DRIVER, \BLOCKED_CUDA, \DEGRADED_LINK.
    • Structural gate report emitted at /run/nodeos/gate-report.json.

  3. Hardware & PCIe Telemetry:

    • PCIe current vs maximum speed & width sampling (\current_link_speed, \current_link_width, \downtrained_now).
    • Remembers \�est_observed\ PCIe link to differentiate idle GPU power-down from physical slot/bus degradation.
    • Comprehensive VRAM, temperature, power draw vs limit, clocks, NIC driver, MAC, MTU, RX/TX bytes & errors.

  4. Direct-LAN Network Gate (
    odeos-netgate):

    • Measures carrier, link speed/duplex, link errors, ICMP ping reachability, RTT, packet loss %, and optional \iperf3\ throughput test against peer (\192.168.77.1).
    • Integrated \S56nodeos-iperf\ service daemon and Buildroot \iperf3\ package.

  5. Key-Only SSH over Direct-LAN (\S55nodeos-ssh):

    • Password login disabled; fail-closed unless explicit \NODEOS_SSH_PUBKEY\ is supplied at build time.
    • Bound strictly to \192.168.77.2:22. Added --ssh-pubkey=\ flag to \scripts/build-nodeos.sh.

  6. Reproducible Evidence Bundle & Host Verifier (
    odeos-evidence):

    • Collects sealed evidence bundle under /run/nodeos/evidence//\ with \manifest.json, \SHA256SUMS, telemetry files, logs, and dmesg tail.
    • Evidence ID = SHA-256 of \manifest.json.
    • Export tool
      odeos-evidence-export\ generates
      odeos-evidence-.tar.
    • Host verifier \ ools/verify-evidence.py\ verifies bundle hash integrity and \PASS\ verdict.

  7. Test Coverage & Contracts:

    • 50 unit tests in \ ests/test_lab.py\ covering state transitions, failure modes, telemetry, network gates, evidence bundles, and console rendering.
    • Updated contracts:
      ode-capability-v1,
      odeos-gate-report-v1,
      odeos-evidence-manifest-v1.
    • Extended \scripts/verify-layout.py\ and .github/workflows/nodeos-001-contract.yml.

aelse
nodeos: NODEOS-001-LAB observability layer, tty1 console and evidence…
225816a 
… bundle

- Add POSIX-sh/ANSI tty1 lab console (nodeos-console) with hardware, fabric link, gate status, and interactive key actions
- Add fine-grained 9-stage state machine and reason codes (BOOTING -> NODE_READY) in nodeos-agent & nodeos-lib.sh
- Add PCIe link telemetry (current/max gen & width, downtrain detection, best-observed tracking) and hardware metrics
- Add direct-LAN network gate (nodeos-netgate) measuring link, ICMP ping, RTT, packet loss, and iperf3 throughput
- Add key-only SSH listener (S55nodeos-ssh) for direct-LAN remote debugging
- Add automated hash-sealed evidence bundle generator (nodeos-evidence) and host verification tool (tools/verify-evidence.py)
- Add 50 unit tests covering lab state machine, telemetry, network gate, evidence roundtrips, and console rendering
Copilot AI balanced review requested due to automatic review settings October 7, 2026 19:03
Copilot AI reviewed Oct 7, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

aelse added 8 commits October 7, 2026 21:26
aelse
fix(nodeos-agent): enforce fail-closed ready flag, sm_61 CC pinning, …
4baf0a3 
…strict evidence verification, PCIe link pair tracking, and compact TUI
aelse
fix(review-followup): harden evidence transfer, host verifier, PCIe s…
da1deb9 
…coring & compact console

- P0 Host Evidence Transfer: strict LASTEXITCODE checks, isolated timestamped subdirectories, explicit TCP listener check for iperf3, Remote Evidence ID binding, clean stdout/stderr separation.
- P0 Host Verifier: safe JSON parsing without unhandled tracebacks, expected profile CLI parameters (--expected-cc, --expected-profile) with local KNOWN_PROFILES mapping, regex/parser validation for PCI BDF, PCIe source, IPv4 interface, GPU UUID, and network cross-file consistency.
- P1 PCIe Scoring: replaced Gen*Width heuristic with theoretical per-lane MB/s capacity table (Gen1=250, Gen2=500, Gen3=985, Gen4=1969, Gen5=3938) multiplied by lane count, with tie-break favoring higher generation; non-numeric string validation for pcie-best.env.
- P1 iperf & SSH: listener address/port validation with timeout, known_hosts pinning support, and strict host key checks.
- P1 Compact Console: top-level sourcing guard for testability, max 22 lines / 80 cols compact view, explicit 'waiting for nodeos-agent' fallback when status.env missing.
- Gate preservation: fail-closed ready logic requiring STATE=NODE_READY & VERDICT=PASS.
aelse
fix(lab): resolve P0/P1 review findings for evidence verification
9803e65 
- Replace Start-Process with native execution in dell-directlan-host.ps1 to fix InvokeMethodOnNull exceptions and guarantee $LASTEXITCODE.
- Enforce strict 64-char regex matching and exactly 1 exported Evidence ID.
- Migrate Python test mocks from batch scripts to robust inline sys.executable wrappers to prevent False Positives.
- Secure verify-evidence.py tar extraction against path traversal and filter data strictly for Python 3.12+.
- Implement explicit jsonschema validation for manifest, capability, and gate-report in verify-evidence.py.
- Harden PCIe Best-Observed scoring by adding a counterexample test case.
aelse
test: Fix and complete negative regression tests for NodeOS verification
c63bab8
aelse
test: Add strict bundle filename validation and fix test isolation
7b7c7f7
aelse
test: close cross-platform evidence gate regressions
47289a3
aelse
test: make POSIX scp mock argument parsing portable
2b0bdba
aelse
fix: make evidence scp destination cross-platform
2d822be
@Hazy142
Hazy142 merged commit 0c9e53e into main Oct 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Hazy142