Plan 2: Situation controller and Acute Triage coordination - #84
Merged
Conversation
Signed-off-by: ernescz <ernescz@gmail.com>
…shape
Two review findings on the controller contracts:
- Fact.EvidenceRefs, ReasonCandidate.EvidenceRefs,
SufficientReason.EvidenceRefs, ActionContract.NextUpdateOn,
Assessment.Limitations, and AssessmentProposal.Limitations were plain nil
slices that marshaled as JSON null, contradicting spec.md's Assessment
contract ground truth (next_update_on/limitations always arrays). Added a
shared canonicalizeSlice helper plus custom MarshalJSON on each affected
type so a nil-constructed value always serializes [] rather than null.
- Assessment.Validate never cross-checked Cadence against
a.Lifecycle.Terminal(): a nonterminal Assessment with Cadence("") and a
terminal Assessment with a non-empty Cadence both silently passed,
contradicting the Cadence doc comment's claim that "" is the only legal
value for a terminal Assessment. Added the parallel terminal/nonterminal
check already done for ActionContract's NextUpdateAt/NextUpdateOn.
Tests: TestNilSlicesCanonicalizeToEmptyJSONArray covers all six fields
(standalone and nested); TestAssessmentCadenceTerminalConsistency covers
both new error cases (terminal + non-empty cadence, nonterminal + empty
cadence).
Signed-off-by: ernescz <ernescz@gmail.com>
Add migrations 0015_situation_controller.sql and 0016_incident_triage_controller.sql: immutable Situation facts, L2 provider call/attempt ledgers, bounded per-Incident Assessment coverage tuples, the guarded current-Assessment/controller-retry projection on situations, and the controller-gated incident_triage rebuild (awaiting_decision phase, Situation/decision metadata, both coverage digests, fenced lease fields) plus its incident_triage_attempts ledger. Also updates the pre-existing v0.13.4 upgrade test: migration 0016's own awaiting_decision backfill now closes the gap ListLegacyReadyIncidents used to reconcile, so that legacy staleness heuristic is correctly superseded by the controller's own B+ gate judgment. Signed-off-by: ernescz <ernescz@gmail.com>
…ort/tests The v0.13.4 upgrade test's doc comment claimed ListLegacyReadyIncidents is now vacuous/superseded in general. It isn't: reconcileUnscheduledTriage still calls it every Correlator tick to catch an ongoing, non-transactional race between MarkIncidentReadyWithSituationInput and SeedIncidentTriage, unrelated to schema migration. Narrow the comment to what the test actually proves — migration 0016 closes the one-time, v0.13.4-vintage gap — and note explicitly that the function remains live production code. No SQL, schema, or test assertion changes. Signed-off-by: ernescz <ernescz@gmail.com>
internal/situation (input_worker.go, reconstruct.go) imported internal/store for SituationClaim, LeaseRecovery, UpgradeIncident, DeadLetterCounts, ErrNotFound, and ErrSituationLeaseLost. Plan 2's Task 8 needs internal/store to import internal/situation for controller-facing store methods, which cycles as long as the old direction holds. Relocate the five plain data structs and the two sentinel errors these files depend on into internal/situation/model, a leaf package both sides already import safely. internal/store now aliases its own SituationInput/SituationClaim/LeaseRecovery/DeadLetterCounts/UpgradeIncident types and ErrNotFound/ErrSituationLeaseLost values to the relocated originals, so every existing store.X call site (internal/store itself, its tests, cmd/alertint's test fakes) keeps compiling unchanged. internal/situation no longer imports internal/store at all. Zero behavior change: same field shapes, same error message text, same error identity for errors.Is comparisons across both packages. Signed-off-by: ernescz <ernescz@gmail.com>
Implement Task 3 of Plan 2: the Situation controller's fact/dispatch/
outcome persistence layer, interrupted-call recovery, and bounded read
views, plus the forward-declared transport-neutral types (internal/
situation/controller.go, snapshot.go) Task 8's ControllerStore
interface will structurally satisfy against.
- internal/store/situation_controller.go: LoadReconciliationInput (one
coherent read transaction over Situation/members/deliveries/Triage/
prior terminals/current Assessment), the idempotent claim-fenced
AppendSituationFacts and RecordAssessmentCall appends, the
lease-independent AppendAssessmentOutcome for non-authoritative
call-backed history, RecoverInterruptedAssessmentCalls (turns an
outcome-less dispatch under an expired claim into one immutable
process_interrupted failed attempt and merges retry_due), and
CommitController's fenced-transaction skeleton (Task 8 completes the
decision/projection commit).
- internal/store/situation_views.go: GetSituationControllerView — the
bounded, sanitized current Assessment/contract/hash/due-reason
projection plus up to 20 recent sanitized attempts and current
per-Incident Triage state, never raw proposal content or provider
bodies.
- internal/situation/{controller,snapshot}.go: Claim, AssessmentCall,
AssessmentAttempt, AuthoritativeAssessment, TriageDecision,
ControllerCommit, SnapshotInput (+ Delivery/IncidentState/TriageState/
CompletedSituation). Two fields deviate from plan.md's literal
Cross-Task Contracts snippet for schema fidelity against migration
0015 (AssessmentCall.MaterialFactHash/ProviderProfile replace a
nonexistent AssessmentBasisHash column; AssessmentAttempt.
UsageInputTokens/UsageOutputTokens replace a single ModelUsage blob,
and ValidationAdjustments is dropped for having no backing column) —
documented in both files' doc comments and the Task 3 report.
- internal/situation/input_worker_test.go: mechanical fix, no behavior
change — references internal/situation/model directly instead of via
internal/store's now-redundant type aliases.
- internal/situation/reconstruct_test.go: moved to the external
situation_test package. internal/store now imports internal/situation
(situation_controller.go) for its controller-facing types; this file's
own store-backed integration fixtures made it the second (after
input_worker_test.go) internal test file whose internal/store import
completed an "import cycle not allowed in test" that only exists for
test binaries, not production code — Go forbids a package's own
internal test files from reaching back to a package that imports the
package under test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BXfMEy12bH1TEkN7PoyM9b
Signed-off-by: ernescz <ernescz@gmail.com>
Adds Task 4's pure Snapshot/fact/hash/reason layer over Task 3's SnapshotInput: DeriveStoreFacts (the 8 Plan 2 fact kinds), BuildSnapshot, MembershipDigest/IncidentInputDigest, DurationClass, MaterialFactHash, AssessmentBasisHash, and EligibleReasons. Everything is a deterministic function over already-provided data — no I/O, no clock reads beyond SnapshotInput.Now. Extends TriageState with LatestAttempt (*TriageAttemptResult, nil until Task 6 wires the store side) so acute_finding has a normalized Finding source to read, per Task 3's own forward note. Of the four Sufficient-reason catalog codes Plan 2 can eventually reach, only duration_outlier is provable from this task's actual inputs today. critical_anchor, novel_symptom, and terminal_uncertainty are wired into the catalog but always return false, each documented with the specific missing data source (severity, persisted symptom history, and lifecycle- deadline machinery respectively) rather than guessed at. Signed-off-by: ernescz <ernescz@gmail.com>
Task 4 review found 4 Important findings, all traceable to Delivery missing data that already exists immutably in alert_deliveries: - criticalAnchorEligible was always false (no severity signal reached the pure layer), so critical_anchor — the only reachable deterministic urgent floor in Plan 2 — could never fire. - MembershipDigest conflated Alert identity with delivery identity, churning MaterialFactHash on every routine Alertmanager re-fire of an unchanged alert (a new alert_deliveries row, same Alert). - Reason candidate IDs hashed only code+situation+input version, omitting predicate version/typed result/evidence refs, so a predicate-version bump silently collided with the old ID. - MaterialFactHash's Triage contribution dropped output_digest, disagreeing with the acute_finding fact about what's material. Fix: extend loadSituationDeliveriesTx's SELECT/scan to read alert_deliveries.alert_id and parse labels_json for severity and the Drill marker; add Delivery.AlertID/Severity/Drill. MembershipDigest now groups by AlertID and picks each Alert's chronologically-earliest delivery for FirstDeliveryIDs. IncidentInputDigest's drillParity placeholder is replaced with real per-Incident Drill data (any-drill- delivery treated as drill-true). criticalAnchorEligible now checks firing status + severity.Rank >= 4. Reason candidate IDs are now a canonical hash over code/catalog version/predicate version/ deterministic floor/sorted evidence refs (plus situation/input version scoping), mirroring the DTO-then-hash pattern used elsewhere. materialIncidentDTO now carries TriageOutputDigest; FindingID is deliberately excluded (storage identity, not decision content). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXfMEy12bH1TEkN7PoyM9b Signed-off-by: ernescz <ernescz@gmail.com>
…sends Round-2 review findings on Task 4: - criticalAnchorEligible latched permanently: it scanned the whole delivery history for any firing critical+ delivery, so an Alert that had since resolved still confirmed an active critical severity forever. Fixed to evaluate, per distinct AlertID, only that Alert's latest delivery (by deliveryLess' total order, mirroring MembershipDigest's per-Alert grouping but taking the maximum instead of the minimum element). - MaterialFactHash embedded Situation.InputVersion directly, so it could never equal itself across two input versions of the same Situation even when nothing material changed -- defeating the reuse guarantee on every reconciliation, not just routine re-fires. InputVersion removed from materialFactHashDTO; SituationID kept. - materialIncidentDTO embedded the full delivery-level IncidentInputDigest, which still churns on every routine Alertmanager re-send (a new alert_deliveries row, same AlertID) -- reintroducing one level up the same class of problem round 1 fixed for MembershipDigest itself. Replaced with MembershipDigest, which answers the actual "did Incident membership change" question this hash needs. Bumped materialFactHashSchemaVersion, assessmentBasisHashSchemaVersion, membershipDigestSchemaVersion, and incidentInputDigestSchemaVersion from 1 to 2 per the file's own stated convention -- the last two were left unbumped in round 1 despite round 1 also changing their included-field semantics. Signed-off-by: ernescz <ernescz@gmail.com>
Implement Task 5 of Plan 2: ValidateAssessmentProposal, DeriveAssessment, DeriveActionContract, DeriveCadence, RevalidateReuse, DeterministicAssessment, DeterministicFallback, prompt construction, and typed L2 outcome classification (ClassifyL2Outcome) — all pure, no I/O, no provider calls. Also fixes the carried-forward AssessmentBasisHash bug flagged in Task 4's report: assessmentBasisReasonDTO hashed each eligible ReasonCandidate's own ID, which bakes in InputVersion (and, transitively, input-version-scoped fact IDs via EvidenceRefs) via reasonCandidateID. That made the basis hash change on every input version whenever a reason was eligible, defeating RevalidateReuse before it could ever fire in the interesting cases. Fixed by hashing each candidate's stable semantic identity (code, catalog/predicate version, own deterministic-floor result) instead of its opaque ID; bumped assessmentBasisHashSchemaVersion to 3 and added regression tests proving stability across input versions for both a floor (critical_anchor) and a non-floor (duration_outlier, whose EvidenceRefs are themselves input-version-scoped) reason. Signed-off-by: ernescz <ernescz@gmail.com>
…ds, fix reuse freshness check Three Important review findings on Task 5's Assessment validation/reuse: - trustworthy() called prior.Assessment.Validate(now) using the CURRENT clock, so a prior's stale next_update_at (always true for a later reconciliation - the exact case reuse exists to serve) made every reuse candidate look untrustworthy forever. Split model.ActionContract.validate into a shape-only check and a separate requireFreshNextUpdateAt, and added Assessment.ValidateShape (skips only the next_update_at-vs-now comparison, keeps every other enum/consistency rule). trustworthy() now calls ValidateShape instead of Validate(now). - ValidateAssessmentProposal never checked a proposed Limitation.Code against any known set, so a model could fabricate a capability code (e.g. "prometheus_confirmed_healthy") and have it stored as authoritative. Added knownLimitationCode, checked against plan2UnsupportedCapabilities plus the fallback's own semantic_assessment_unavailable code. - forbiddenTopLevelKeys was a 3-key blacklist (lifecycle/action_contract/cadence) checked against the raw proposal JSON, so a model response that flattened an Operator-contract field to the top level (e.g. a bare next_update_at) passed through silently. Replaced with an allowlist of AssessmentProposal's actual 8 legitimate top-level keys. Covering tests added for all three; full suite green. Signed-off-by: ernescz <ernescz@gmail.com>
Task 6 of Plan 2: every ready Incident now begins in awaiting_decision and requires a versioned controller request/skip decision before Acute Triage may dispatch it, with every decision and attempt fenced by both membership_digest and incident_input_digest. - internal/situation/triage.go: DecideTriage, a pure B+ gate decision. Skip is legal only when a trustworthy current Assessment's own material_fact_hash and per-Incident coverage tuple exactly match the current snapshot; a changed membership/Incident-input digest on an already-decided pending/backoff row triggers a "refresh" that is always request, never a newly discovered skip. Never reads Attention/EligibleReasons, so a deterministic urgent floor can never shortcut or block the decision. - internal/store/triage_controller.go: the unexported applyTriageDecisionsTx (tested directly; Task 8's CommitController is its only future production caller) plus the fenced attempt lifecycle over the new incident_triage_attempts ledger -- ClaimIncidentTriageAttempt (freezes digests and bounded member delivery IDs, leases the schedule, appends triage_retry_changed), CompleteIncidentTriageAttempt (one atomic/idempotent boundary that recomputes current digests and branches between a current-compatible Finding and a sanitized stale_membership/stale_incident_input record), BackoffIncidentTriageAttempt/ExhaustIncidentTriageAttempt, ExtendIncidentTriageLease, RecoverExpiredIncidentTriageAttempts, and BackfillUpgradedIncidentTriageSchedule (startup-only, tags retained pre-Plan-2 schedulable rows decision_origin=upgrade_existing_schedule without retroactively deciding them). - internal/situation/controller.go, internal/store/situation_controller.go: added AuthoritativeAssessment.MaterialFactHash (a genuine Task 3 gap DecideTriage's skip condition needs) sourced from the existing situation_assessment_attempts.material_fact_hash column. - internal/store/deliveries.go: MarkIncidentReadyWithSituationInput now creates the awaiting_decision schedule row in the same transaction as the ready transition and its incident_ready Situation input. - internal/store/triage.go: SeedIncidentTriage becomes a tolerant upsert (promotes a fresh zero-attempt awaiting_decision row straight to pending, or inserts fresh if none exists) rather than a bare insert, so Correlator's still-unmodified shipped five-attempt dispatch path and its full existing test suite keep passing unchanged against the new atomic ready+schedule commit. This is a deliberate, documented compatibility shim, not a controller decision -- Task 7/8/9 own replacing it once real dispatch ownership moves off Correlator (per the Task 2 report's own note on that boundary). 37 new tests (15 pure DecideTriage, 22 store-level decision/claim/ completion/backoff/exhaust/recovery/upgrade); full repo suite and internal/store + internal/correlator race tests green; zero changes to internal/correlator. Signed-off-by: ernescz <ernescz@gmail.com>
ClaimIncidentTriageAttempt's doc comment claimed it only claims "due" pending/backoff rows, but the claim query never read or checked next_at, so a backoff row still an hour from due was fully claimable right now. Add a next_at read + check after the phase gate, returning the new ErrTriageNotDue sentinel (distinct from ErrNotFound/ ErrTriageNotDecided) when a row exists and is otherwise claimable but not yet due. Also add missing test coverage for ExtendIncidentTriageLease (the lease heartbeat), which previously had zero tests: a successful extend pushes lease_expires_at forward without disturbing phase/owner/ attempt identity; a wrong-owner extend, a wrong-attempt-id extend, and an extend against a row that already moved off in_flight (backed off) all fail with ErrTriageAttemptLeaseLost. Fixes 2 Important findings from the Task 6 review (review-967ad1e..bb5b363.diff). Signed-off-by: ernescz <ernescz@gmail.com>
Task 7 of Plan 2: separates Acute Analysis (Skill.Analyze) from its
durable dispatch, and moves due-Triage dispatch out of the Correlator
tick entirely into a new internal/situation.TriageWorker that polls
Task 6's gated schedule independently, claim by claim.
- internal/situation/triage_worker.go: the situation-native contract
(TriageAttemptClaim/AcuteResult/PostCommitData/AcuteAnalyzer per
plan.md, with one documented field-shape fix -- TriageAttemptClaim's
MemberIDs/DeliveryIDs collapse into the single MemberDeliveryIDs
Task 6 actually built) plus TriageWorker itself: claim -> heartbeat
the lease while Analyze runs -> complete/backoff/exhaust against the
real result, AfterCommit only on a genuine success (never on a stale
completion). Deliberately no internal/store import -- store already
imports situation, so the reverse would cycle -- so
TriageAttemptStore/TriageScheduleLister are phrased in this
package's own types; a thin *store.Store adapter for the three
methods whose store.* types this package can't reference is Task
9's runtime-wiring job.
- skills/acutetriage/{skill.go,result.go}: extracted analyzeCore (a
pure extract-method of what pipeline did before persist -- rules ->
evidence -> LLM -> verification round -> caps; no analysis logic
changed, only where it lives) shared by Rejudge's unchanged pipeline
and the new Analyze. Analyze does no durable write and no outward
call; AfterCommit applies the held role/memory/notify/audit effects
best-effort, once, only after the store's own completion commits.
Run is now a compatibility composition (Analyze -> SaveIncidentOutput
-> AfterCommit) for existing direct/rejudgment tests only -- not used
by the worker.
- internal/correlator/correlator.go: removed the SeedIncidentTriage +
dispatchTriage + dispatchDueTriage + recoverTriageState calls.
MarkIncidentReadyWithSituationInput (atomic since Task 6) is the only
per-incident triage-adjacent write flushExpired does now -- a
Correlator tick makes zero model calls (TestFlushExpired_
MakesNoModelCalls). Deleted triage_retry.go/_test.go (the whole file
is dead code once its two entry points have no callers); IncidentSink
stays for API compatibility but nothing invokes it anymore.
Necessary collateral fixes: 7 correlator_test.go tests that observed
readiness via the now-never-called sink, rewritten against a
store.ListRecentIncidents-based wait; export_test.go's two dangling
exports removed; one verify_integration_test.go fixture that built its
reference prompt from a stale (pre-insertTestAlert) Incident value the
new Analyze path correctly no longer trusts.
Four known gaps flagged (not silently patched) in the Task 7 report:
Analyze loads Alert data via the legacy alerts/incident_alerts tables
rather than the claim's frozen delivery ids (the store's completion
digest fence is the real backstop); clean-skip uses
ExhaustIncidentTriageAttempt (no dedicated skip-completion primitive
exists yet, so a clean skip lands the Incident on "failed" rather than
the old schedule's "ready"); failure classification is coarser than
the removed llmhealth-aware classifyTriageError (importing llmhealth
here would cycle through internal/store the same way internal/store
itself would); no equivalent to the old one-hour startup-backlog
horizon.
go build ./..., go vet ./..., and go test ./... (whole repo) all
clean; the three brief-specified -race commands (skills/acutetriage,
internal/situation -run TestTriageWorker, internal/correlator) all
green.
Signed-off-by: ernescz <ernescz@gmail.com>
…store exhaustion signal and error granularity Fixes 4 Important review findings on Task 7's triage cutover: - Analyze now loads member content through the frozen TriageAttemptClaim.MemberDeliveryIDs (new store.GetAlertDeliveries), never GetIncidentAlerts' current mutable projection -- closes the re-fire divergence where a claimed Incident's own digests never change but the shared alerts row does. Run keeps loading current state (it has no claim to freeze against) but now re-reads the Incident fresh, matching Analyze's own contract. - Clean skip now closes through a dedicated store primitive, CompleteIncidentTriageAttemptAsCleanSkip: schedule -> skipped, Incident -> ready (never failed), one triage_skipped input -- instead of misusing ExhaustIncidentTriageAttempt, which broke later-occurrence collapse and emitted the wrong Situation input kind. - TriageWorker gained an ExhaustionNotifier hook, called once after a genuine 5-attempt exhaustion (never on a clean skip or stale completion); skills/acutetriage.Skill.OnTriageExhausted is the concrete implementation. cmd/ wiring is left to Task 9. - classifyAttemptError now checks for an optional ClassifiedError interface before falling back to its coarse context.*-only classification; skills/acutetriage.classifyAnalyzeError restores the deleted classifyTriageError's llmhealth-aware granularity (capability-aware codes, schema/malformed-response, LLM-origin timeout vs. ambiguous non-LLM shapes). Signed-off-by: ernescz <ernescz@gmail.com>
…embers loadFrozenClaimAlerts treated an EMPTY claim.MemberDeliveryIDs as proof of zero member alerts, but memberDeliveryIDsTx freezes it from incident_alert_deliveries alone — empty both for a genuinely membership-less Incident and for one that predates the delivery ledger (pre-migration-0013) or was reconstructed via UnrepresentedOperationalIncidents/ ReconstructSituation with incident_alerts rows but no incident_alert_deliveries rows. The latter case was terminally clean-skipped with a misleading "below the minimum member alert count" reason, even though the Incident has real members. Fall back to GetIncidentAlerts (the pre-Task-7 current-state read) when MemberDeliveryIDs is empty: every current ingestion path inserts incident_alerts and incident_alert_deliveries together in one transaction, so a nonempty GetIncidentAlerts result with an empty frozen delivery set can only be a legacy/pre-ledger row, never a currently-active Incident racing this claim. Only when GetIncidentAlerts also returns nothing is this genuinely a zero-member Incident, which still reaches ErrCleanSkip exactly as before. Signed-off-by: ernescz <ernescz@gmail.com>
Implement Task 8, the fenced Situation controller: Controller.Reconcile orchestrates coherent load, fact derivation, Snapshot/hash/reason reduction, lifecycle timing, pure Triage decisions, deterministic/reuse checks, work-bearing L2 dispatch with durable before-I/O call recording, and one fenced CommitController commit, per spec.md's runtime-ownership order. - internal/llm: add RequestStartStatus/OneShotCompletion/CompleteOnce to both provider clients (zero internal retries, ClassifyRequestStart maps true/false/unknown); Complete's existing retry behavior is unchanged. - internal/situation/lifecycle.go: source-aware recovery-grace and lifecycle-observation-deadline derivation (RecoveryGraceDuration, ObservationDeadlineAt, ClosedUnknownReason, AnyFiring). - internal/situation/controller.go: ControllerStore, AssessmentClient, AuditSink, ControllerConfig, Controller, NewController, Reconcile, and the L2 outcome-matrix dispatch/retry/park machinery. - internal/situation/controller_worker.go: ControllerWorker (claim, concurrent worker pool, global L2 semaphore, lease heartbeat, Start/Wake/RunOnce/Drain/Stop, optional dependency-recovery wake hook). - internal/store/situation_controller.go: complete CommitController's fenced commit body, BeginControllerAttempt, LastTrustworthyAssessment, ClaimControllerWork/ExtendControllerLease/ReleaseControllerWork, and WakeDependencyRecoveredSituations. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXfMEy12bH1TEkN7PoyM9b Signed-off-by: ernescz <ernescz@gmail.com>
golangci-lint pass over the fenced controller: remove dead code (subtractDueReasons, unused since CommitController's own subtraction is store-side), fix a duplicated-word error message, make two closed-enum switches exhaustive instead of relying on default, reorder dispatchWorkBearing's return values so error is last, and small test hygiene (t.Helper(), drop an unused param, avoid an unlambda). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXfMEy12bH1TEkN7PoyM9b Signed-off-by: ernescz <ernescz@gmail.com>
…cy park, add worker backoff, resolve floor/L2 interaction Two Critical fixes: revalidated-reuse commits now pass the schema's nominal work_attempt=1 instead of 0 (migration 0015's CHECK rejected 0); recovery_pending->closed_unknown-via-deadline now carries GraceUntil forward alongside RecoveryObservedAt (migration 0014's unconditional pairing CHECK rejected the mismatch). Both were completely hidden by controller_test.go's fake store, which never enforces either CHECK - real-SQLite regression + end-to-end tests now cover both call paths. Three Important fixes: a policy/capability park is now actually enforced - Reconcile checks it before BeginControllerAttempt/dispatch, skipping work-bearing L2 entirely while parked against the unchanged basis, and naturally lifting once the basis changes. ControllerWorker.processOne now writes a bounded typed backoff (mirroring input_worker.go's own retry pattern) when releasing a lease after a failed Reconcile, so a persistently-failing Situation can no longer spin Drain at 100% CPU. The deterministic urgent floor no longer short-circuits Reconcile before consulting L2 - the floor's only effect is Task 5's existing Attention-raising adjustment; a Situation with no prior trustworthy Assessment goes through the same reuse-check/work-bearing-dispatch path as any other Situation, so critical_anchor Situations eventually receive real semantic judgment instead of being stuck on conservative defaults reused forever. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXfMEy12bH1TEkN7PoyM9b Signed-off-by: ernescz <ernescz@gmail.com>
…ot just fresh ones fallbackOrPreserve's PRESERVE branch called DeriveAssessment directly on a stale prior Assessment with no floor check and no validateProposalContent pass, so a critical_anchor floor becoming newly eligible in the same cycle L2 fails (or the cycle is park-blocked) silently committed the stale pre-floor Attention instead of urgent. Route the preserved proposal through the same rebindSufficientReason + validateProposalContent pattern RevalidateReuse already uses to revalidate a prior against a newer Snapshot, extracted into a shared rebindSufficientReason helper. On acceptance the (possibly floor-adjusted) proposal is preserved; on rejection (e.g. the SufficientReason grounding a stale urgent Attention is no longer eligible) it falls through to DeterministicFallback instead of committing a proposal no longer valid against current state. Also corrects the I3 ruling comment in Reconcile, which incorrectly claimed every remaining path already guaranteed Attention=urgent when the floor is active. Signed-off-by: ernescz <ernescz@gmail.com>
Wire Task 7/8's Situation controller and Acute Triage workers into the actual running binary: a new controllerRuntime (mirroring foundationRuntime) composes them with startup recovery/backfill and drain-then-stop shutdown, main.go builds the one-shot L2 AssessmentClient and wraps it with the installation LLM-health observer, and the Acute Triage skill drives the Triage worker directly. Renames Task 8's ad-hoc controller audit events to spec's exact taxonomy and adds the four missing Triage-worker events. Closes the one-hour startup-horizon gap Task 7 deferred (ADR-0045) as a new store primitive. Extends the Situation controller view and alertint_get_situation with real Assessment/derivation, hashes, bounded recent attempts, Triage digests, and retry/park state. Adds the assessment LLM-health capability (migration 0017) and fixes the config/logic cadence-default mismatch. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXfMEy12bH1TEkN7PoyM9b Signed-off-by: ernescz <ernescz@gmail.com>
…on, fix runServe lint, thread real duration and request-started into audit Task 9 review fix round, 4 Important findings: 1. TriageWorker.completeFailure appended incident.triage_exhausted directly AND called the configured ExhaustionNotifier (skills/ acutetriage.Skill.OnTriageExhausted), which appends the same event again into the same audit sink in production -> double emission. The worker's own direct append now only fires as a fallback when no ExhaustionNotifier is configured; the notifier owns the row otherwise. 2. The new ExhaustOverdueUnclaimedIncidentTriage startup-horizon sweep (internal/store/triage_controller.go) terminally failed Incidents with zero audit trail. It now returns per-Incident identifying content (ExhaustedTriageIncident) instead of a bare count, and controllerRuntime.RecoverAndBackfill emits one incident.triage_exhausted row per exhaustion, outside any transaction, with reason "startup_retry_window_expired" matching the deleted pre-Plan-2 applyStartupHorizon's own convention. 3. cmd/alertint/main.go:runServe exceeded golangci-lint's gocyclo threshold (32 > 30). Extracted the controller-runtime construction (buildControllerRuntime) and two closures with their own branches (runFoundationReconstruction, runControllerRecovery) into named functions, since Go's gocyclo counts a closure literal's branches against its enclosing function. 4. duration_ms was structurally always zero (CreatedAt/CompletedAt set from the same clock read) and provider_request_started never appeared in any audit payload. buildAuthoritativeAttempt/ buildOutcomeAttempt now thread the real measured L2 call latency (llm.OneShotCompletion.Latency) through to backdate CreatedAt, and provider_request_started is now included in every call-backed audit payload (assessment_authoritative/reused/fallback, rejected, failed, stale). Tests: TestTriageWorkerAuditsExhaustionOnFifthAttempt now wires a real audit-emitting notifier and asserts exactly one row (plus a new fallback-path test); TestExhaustOverdueUnclaimedIncidentTriage* assert the returned identifying content, TestSituationControllerRuntimeRecoverAndBackfillAuditsStartupHorizonExhaustion proves a real end-to-end audit row; four new Reconcile-level tests use a fake client with a controlled non-zero Latency to prove duration_ms/ provider_request_started reflect real measurements, not fixtures. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXfMEy12bH1TEkN7PoyM9b Signed-off-by: ernescz <ernescz@gmail.com>
Round 1's dedupe fix made TriageWorker.completeFailure's own audit append fire only when no ExhaustionNotifier was configured, on the theory that the notifier (skills/acutetriage.Skill.OnTriageExhausted) could build a richer payload instead. That was backwards: OnTriageExhausted's signature (incidentID, code, detail) never receives situation_id/attempt_id/attempt_number/input_version, so every production deployment (which always configures a real notifier) ended up with the one surviving incident.triage_exhausted row missing those four identity fields - the only sibling event (triage_attempt/_stale_input/_completed) not carrying full identity. Make TriageWorker the single, unconditional owner of the audit row: it holds the full identity via claim, so it appends the row itself regardless of whether a notifier is also configured. OnTriageExhausted no longer audits at all - it is now a pure notification hook (its real remaining behavior, the notify.TriageFailureSink call, is kept). This also removes the dependency on a notifier's own choice to audit, closing the "nil-auditor notifier yields zero rows" fragility flagged in round 1's review. Signed-off-by: ernescz <ernescz@gmail.com>
Real-store replay fixture entering through the real Receiver HTTP path and production foundation/controller/Triage assembly (never a hand-seeded Situation, fact, Assessment, or Triage envelope), with the eight crash-boundary subtests the spec names: after Plan 1 input application; after Situation claim; after L2 dispatch record before response; after rejected/failed attempt persistence; after authoritative insert before projection commit; after Triage attempt begin before result; after Finding persistence before worker return; and a concurrent input raising a new due reason. Only the L1/L2 clients are deterministic fakes; every store, worker, and pipeline component around them is real. Each subtest closes and reopens the same on-disk database file and runs the real startup recovery/replay sequence before asserting convergence. The replay went red twice for genuine reasons, not test-infra gaps: 1. AppendSituationFacts (internal/store/situation_controller.go) compared a fact's observed_at as part of its idempotent-replay conflict check, but DeriveStoreFacts sets observed_at fresh to "now" on every call. Any retry of an unchanged input past that point (a transient L2 failure, a stale-claim race, or a crash before CommitController) always fails closed with ErrImmutableConflict forever, permanently wedging the Situation - reachable in ordinary operation, not just a crash. Fixed by excluding observed_at from the conflict check; the row's first-recorded value stands. 2. incident_triage_state and acute_finding facts (internal/situation/ facts.go) keyed their identity on (situationID, inputVersion, subject) alone, but both read live controller/Triage state that can legitimately advance within one unchanged input_version (a controller's own request/skip decision landing, or an independent Triage attempt completing). Fixed by folding the fact's own content digest into its id (factIdentityWithContent) so a genuinely distinct observation gets its own row instead of colliding with the prior one. Also updates public docs to say precisely what is wired on state-controller (durable Situation foundation, the fenced controller, local Store facts, the B+ Triage gate) versus what is not yet (connector prep, Assessment/Triage artifacts beyond bounded history, Transition/Episode history, Situation Slack, the v0.14 cutover), documents every real situations.* config key against config.go's current struct, and seeds lab-acceptance.md's evidence-table structure for the follow-on lab run to fill in. Signed-off-by: ernescz <ernescz@gmail.com>
…ity and make idempotent-reconverge assertions non-vacuous Two Important review findings on the crash-boundary replay suite (72bcde8): 1. factIdentityWithContent (Bug 2's fix) had zero test coverage anywhere. Added two direct unit tests in facts_test.go pinning that a different Phase/LatestAttempt at the same (situationID, inputVersion, subject) produces a different fact ID — verified red on a reverted fix, green on current. Also added a systemic assertNoReconcileFailed check inside the replay fixture's own convergence helpers (after every controller drain round): asserts no Situation ever carries a non-nil last_error_class, catching any reconcile cycle that silently failed and got swallowed by ControllerWorker.processOne — not just this one bug, but the whole class. 2. assertIdempotentReconverge (and boundary 6's own re-run) proved nothing: every re-run's clock advance (1 minute) never reached the committed slow-cadence next_assessment_at checkpoint (+15m), so zero controller work ever executed and "no new L2 call"/"hashes unchanged" held vacuously. Fixed by aging each tested Situation into DurationClassLong (unbounded above, so no later class-boundary crossing can force a fresh L2-calling derivation) before its first tested convergence, then advancing past the checkpoint before reconverging, and asserting the controller genuinely claimed the Situation again. This also corrected an over-assertion: current_assessment_id was required to stay byte- identical across a reuse commit, but a reuse commit always mints its own new authoritative attempt row (already pinned by TestControllerReconcileEndToEndReuseCommitsAgainstRealSchema) — that requirement was never exercised before this fix and turned out to contradict already-established production behavior the moment a real second cycle ran. Replaced with a hash-stability + fresh-attempt + derivation=revalidated_reuse check. Fixing #1's systemic assertion surfaced a genuine, previously-undetected production bug (not caused by, or specific to, either fix above): a crash landing after RecordAssessmentCall durably records a Situation's first-ever L2 dispatch but before that cycle's own CommitController ever runs leaves current_material_fact_hash unset, so the next reconcile's BeginControllerAttempt resets work_attempt to 1 again and collides with the immutable pre-crash situation_assessment_calls row on its own UNIQUE(situation_id, input_version, retry_epoch, work_attempt, call_number) index — silently wasting one cycle on a degraded deterministic_fallback Assessment before self-healing on the next cycle. Reproduces boundaries 3, 4, and 5 identically with the clock-advancement fix removed, so it is not a side effect of either fix; documented at length in the test file and left red (not weakened or scoped away) pending a dedicated production fix, per this task's own explicit instruction not to paper over a genuine discovery. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXfMEy12bH1TEkN7PoyM9b Signed-off-by: ernescz <ernescz@gmail.com>
…rrupted-call recovery A crash landing after RecordAssessmentCall's durable dispatch record but before that cycle's own CommitController commit left situations.controller_work_attempts/current_material_fact_hash behind the immutable pre-crash situation_assessment_calls row. The next real BeginControllerAttempt call then recomputed the EXACT SAME (retry_epoch, work_attempt, call_number) coordinates, colliding on that table's own UNIQUE index and silently degrading to a deterministic_fallback commit for a whole cycle (boundaries 3/4/5 of TestControllerRealStoreReplay). RecoverInterruptedAssessmentCalls now also catches up each touched Situation's controller basis: - loadStaleControllerBasisTx finds every un-claimed Situation whose projection has fallen behind its own most-recently-dispatched call — deliberately broader than "outcome-less": a call can already have a real rejected/failed outcome (AppendAssessmentOutcome's own separate commit) while its owning cycle's CommitController still never ran. - advanceControllerBasisForRecoveryTx writes current_material_fact_hash/ controller_work_attempts to match that call (so an unchanged basis correctly continues at work_attempt+1), unconditionally bumps controller_retry_epoch (so a basis that has since drifted — e.g. a duration-class boundary crossed during the crash-to-restart gap — still lands on a fresh, never-colliding coordinate lane, mirroring the existing dependency-recovery wake primitive), and clears any controller_parked_at/ controller_parked_reason (provably always stale whenever an interrupted call exists, since controllerParkBlocksDispatch would otherwise have prevented that call from ever being dispatched). Regression coverage: TestRecoverInterruptedAssessmentCallAdvancesBasisSoNextAttemptDoesNotCollide, TestRecoverInterruptedAssessmentCallsCatchUpStaleBasisAfterResolvedRejectedOutcome, TestRecoverInterruptedAssessmentCallClearsStalePolicyParkOnBasisAdvance (internal/store/situation_controller_test.go). All 8 TestControllerRealStoreReplay subtests now pass; shrank the resolved root-cause comment above boundary 3 to a short pointer at this commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXfMEy12bH1TEkN7PoyM9b Signed-off-by: ernescz <ernescz@gmail.com>
…close crash-free epoch gap Fix round 3 on internal/store/situation_controller.go (Task 10 Part 1). C1 (Critical regression from 2e6fa67): loadStaleControllerBasisTx's "controller_work_attempts < c.work_attempt" disjunct matched a Situation freshly reset by wakeOneDependencyRecoveredSituationTx (work_attempts=0, same basis) as if it were a crash gap, and advanceControllerBasisForRecoveryTx restored work_attempts back up to the old exhausted call's value - silently, permanently re-stranding a woken Situation. Removed the disjunct; the query now matches only current_material_fact_hash NULL-or-mismatch, which is both necessary and sufficient. The now-redundant MAX(controller_work_attempts, ?) write became a plain assignment, since work_attempts is always already caught up under the narrowed query. I1 (original bug's crash-free variant): BeginControllerAttempt now advances controller_retry_epoch by 1, in the same transaction as work_attempts, when a Situation transitions from one previously-committed basis to a genuinely different one (never on a first-ever dispatch) - closing the same UNIQUE-index collision 2e6fa67 fixed for crashes, but reachable with no crash at all when MaterialFactHash drifts between two ordinary reconcile cycles at the same input_version. Two new regression tests added; all pre-existing assertions unchanged. Signed-off-by: ernescz <ernescz@gmail.com>
Mechanical lint pass, no behavior change: named interface parameters, a fixed audit actor constant instead of a repeated literal, #nosec annotations with rationale on the bounded shift/jitter in retryBackoff, thelper/unparam annotations on test fixtures, the multi-value fake snapshot() replaced by a struct, an unused test helper removed, and the nested fallback-or-preserve branch split into a guard-clause helper. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
A Situation whose controller lease is lost mid-cycle with no restart (heartbeat ExtendControllerLease error cancelling the reconcile, a stall outliving the lease, or a CommitController error) leaves a durable situation_assessment_calls row whose cycle never committed. The projection cache on situations then still describes the previous commit, so the next claimant re-minted the same (retry_epoch, work_attempt, call_number) and collided on the UNIQUE index; RecoverInterruptedAssessmentCalls is startup-only and never ran. BeginControllerAttempt now reconciles against this input version's most recently dispatched call inside its own fenced transaction: if the projection hash is NULL or differs from that call's basis, it repairs the projection to what that commit would have written (hash, work_attempt, park cleared) before choosing a coordinate. The startup pass's basis catch-up is defense in depth. Tests drive the exact worker-A-dispatch / lease-expired / worker-B-claim sequence against the real schema for five stranded shapes, the stale policy park, the older input version, and the woken dependency park, and pin the three crash-recovery probes that used to live in a scratch file. The lab-acceptance note records the accepted edge (a stranded fifth attempt leaves the basis exhausted with no park marker). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
…ervability Eight findings from the branch review, each fixed at the seam that owns it: - LLM health observed the transport, not the outcome: a parseable but malformed or policy-invalid Assessment proposal was recorded as a healthy call, and every observation used an empty subject so the two-Situation corroboration rule could never fire. The controller now exposes AssessmentHealthObserver and reports each dispatch's final ClassifyL2Outcome with the Situation ID; cmd/alertint maps it onto the llmhealth tracker (content vs dependency class, stale = success). The CompleteOnce decorator is gone. - A post-claim clean skip consumed a Triage attempt, contradicting the Part 2 invariant. The Triage worker now resolves minimum-member eligibility BEFORE claiming, through a store primitive that counts members and closes the schedule as skipped in one transaction with no attempt row and no attempt increment; the skill exposes MinimumMemberAlerts. Analyze's ErrCleanSkip path stays as defense in depth. - Shutdown order matches the plan: Receivers, Correlator (so its fixed-window ticker cannot mint fresh durable work after the drain), then foundation and controller/Triage drains in rounds until a round handles nothing, then the workers. - Cadence is live configuration: ControllerConfig.Cadence reaches the contract derivation through ControllerState.Tempo, defaults are the plan's 60/300/900s, and the loader enforces fast < normal < slow. - MCP exposes the eligible Sufficient-reason candidate set: the commit persists it on situations.current_eligible_reasons_json, the Store view reads it, alertint_get_situation renders eligible_reasons. - Migration ownership: the llm_health_capabilities widening that shipped as 0017 is folded into unapplied 0015; MaxSchemaVersion is 16 again and 0017/0018 stay free for Plan 3. - The Correlator is constructed with NopIncidentSink; the Skill-backed sink and its callback are removed, with a wiring test. - OpenTelemetry spans for the controller cycle, each consumed L2 dispatch slot, and each consumed Triage attempt carry stable identity, digest, count, result-class, and duration attributes and never payloads, pinned by a span-recorder test. Spans go through the global provider; no exporter is wired in this build, and the docs and lab-acceptance notes say so. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
SeedIncidentTriage was a production Store method with no production caller — every ready Incident begins awaiting_decision and only a fenced controller decision moves it — and a doc comment describing the pre-Plan-2 Correlator dispatch chain. It now lives in internal/store/storetest as a test-only fixture over *sql.DB (so the store's own in-package tests can use it without an import cycle), the two stale comments naming the removed dispatch chain are reworded, and the repo-side lab-acceptance structure points at the canonical private evidence record. Task 10's forbidden- leftover sweep now returns only the documented absence of max_l1_llm_calls. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
…o LLM dispatch The Skill-backed IncidentSink was already gone, but production still handed the Acute Triage skill to cor.SetRejudger, and attachOccurrence still called Rejudge on an escalation trigger. Dormant on the durable Delivery path or not, that is an analyzer/LLM dispatch dependency the Correlator must not carry (Plan 2 Task 7). - Delete Rejudger, the rejudger field, SetRejudger, and the Rejudge call; an escalation now only stamps its trigger_kind on the occurrence row (actionRejudge renamed actionEscalate to say what it does). - cmd/alertint no longer passes the skill to the Correlator. - The wiring proof now also checks the Correlator type for any re-judgment seam and that no non-test correlator source imports internal/llm, internal/llmhealth, or skills/*. - Correlator tests assert the durable trigger instead of a fake rejudger. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
…bort when it cannot be Two review findings on the controller's L2 dispatch loop: - A failed AppendAssessmentOutcome was logged and the cycle went on to audit, correct, and commit as if the row existed. dispatchWorkBearing now returns its own infrastructure error, distinct from the provider transportErr the caller classifies; reconcile aborts on it, and the assessment_rejected/_failed audit event is emitted only after the outcome row committed. Prompt-build and call-record failures take the same path instead of being classified as transport outcomes. - A cancellation while a call was still waiting for the worker's L2 semaphore returned a zero-valued completion, so the outcome row carried provider_request_started "" (rejected by the store CHECK) on a context that was already dead. The semaphore now reports RequestStartStatusFalse (no request was attempted), and the outcome row plus its audit event are written on a short detached context once the cycle's own context is done, mirroring the Triage worker's detachedWriteContext — a real-store test drives two due Situations through one slot and reads the 'false' row back. Every span site now also writes one structured log line with the same identities and the span's trace_id/span_id (reconcile, dispatch slot, Triage attempt), pinned by tests against a span recorder, so logs, spans, audit, and the store reconcile by identity. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
…ller and Triage spans Plan 2's lab gate reads consumed L2 dispatch slots and Triage attempts from OTel next to MCP, audit, SQLite, and logs, and the lab runs an OpenTelemetry Collector — but the binary installed no exporter, so the spans never left the process. - New config section telemetry.otlp (enabled, endpoint, protocol grpc|http, insecure, service_name, timeout_seconds); off by default, validated under the strict loader, documented in the configuration reference and config.example.yaml. - New internal/telemetry: builds the OTLP exporter and a batching tracer provider (resource from config + OTEL_RESOURCE_ATTRIBUTES), installs it as the global, and returns a flush/shutdown; never dials eagerly, so an unreachable collector is a logged export error, not a startup failure. - cmd/alertint starts it after config load and flushes it after the foundation stop sequence; the receivers stop step moved into receiverShutdown to keep runServe within the complexity gate. - lab-acceptance.md: check 10's OTel columns are filled from the agent's per-cycle log lines (trace/span IDs) and the collector-derived span-metrics series in the lab Prometheus, reconciled against the SQLite ledgers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
…on codes in the L2 prompt
First live lab run of Plan 2 (2026-09-04): every L2 call was rejected as
invalid_shape and the controller burned all five work attempts into
deterministic fallbacks. Reproduced against the same model from the lab:
the schema instructions showed sufficient_reason as null and limitations
as [], and the model answered with a bare candidate-ID string and
free-text strings. The validator's closed shapes were never stated.
The prompt now spells out sufficient_reason as {code, candidate_id,
summary, evidence_refs} copied verbatim from eligible_reasons, limitations
as {code, detail} objects, the bounded text length, and the allowed
limitation codes rendered from the same list the validator checks. Two
live calls with the new prompt produced the documented shape.
Tests pin the nested-shape wording, every allowed code, the observed lab
shape as an invalid_shape regression, and the documented shape passing the
shape and capability gates.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs
Signed-off-by: ernescz <ernescz@gmail.com>
…n the Situation view The 2026-09-04 lab acceptance run (check 10) caught the MCP Situation read disagreeing with SQLite, audit, logs and OTel on "consumed Triage attempts": listIncidentTriageViews projected the count and phase from the incident_triage schedule row, which CompleteIncidentTriage deletes once a Finding is persisted, so a just-judged Incident rendered triage_attempts 0 and an empty phase while incident_triage_attempts held the consumed attempt. The view now reads attempts as MAX(schedule counter, ledger row count) and renders phase "completed" when the schedule row is gone and a successful attempt exists. Two store tests pin the post-completion state and the never-below-ledger rule. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
…ucture The lab step ran on 2026-09-04 (two fresh-database runs, Slack disabled). The repo copy stays a structure without IDs; its status now says which checks passed on live evidence, which are pending, and what the two runs found (prompt shapes, MCP Triage-attempt projection). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
…ass, one controller defect open Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
A work attempt whose durable slot was reserved but whose cycle never committed (the process died mid-dispatch, e.g. a drain-time L2 call cut off by SIGKILL) left the Situation exhausted with no park recorded. The stranded repair skipped it (unchanged material hash), every cycle re-entered the exhausted branch and re-committed the bounded fallback on cadence with a healthy LLM, and WakeDependencyRecoveredSituations could never re-arm it because it only selects dependency parks. Only a material hash change (a duration-class boundary) ever recovered it; in the long class it never would. Reconcile's ErrControllerAttemptsExhausted branch now records a ParkedReasonDependency park when the loaded input carries no park, so the next dependency recovery generation opens one new epoch. A park already on record is left untouched, as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
…path, and the reuse hot loop Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
…ckpoint (lab F8) A Situation whose lifecycle-observation deadline had already passed but whose alert kept firing stayed active, as it must, yet the past deadline was still folded into next_update_at as a "reconsider at" candidate. nextUpdateAt clamps an already-due candidate to now + 1 s, so the worker re-claimed the Situation on every 2 s poll for as long as the alert fired: one revalidated_reuse attempt row and one audit row every two seconds, without end (lab run 4: a week-old ContainerMemoryHigh alert, ~30 rows a minute). buildControllerState now carries the deadline only while it is still ahead of now. The lifecycle already reconsiders a passed deadline every cycle in resolveLifecycle, so nothing is lost: closure still happens the moment firing truth is gone, and a firing Situation idles at its cadence like any other. A still-future deadline is promised and scheduled exactly as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
…e lab acceptance note Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
…ified live, one design gap open Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
… others' dependency failures (lab F10) The dependency-recovery wake waits for the rolled-up LLM state to be healthy, and the assessment capability could heal only on an assessment success. While every Situation is parked on that very outage no assessment can run, so in the lab the wake waited five minutes for an unrelated lease to expire instead of following the first Triage draft that proved the provider was back. triage_draft, assessment, verification_rejudge and query_repair are served by the one configured primary client and model, so a real success on any of them now withdraws a dependency-class failure on the others and on the probe. Content-class failures stay with their own capability, the memory classifier is outside the rule in both directions, a probe success still clears nothing but the probe, and a capability that was not called keeps its own last_success_at (the old probe clearing fabricated one). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
…mmit (lab F5) The correlator plans a retry attach after reading that the Incident's Acute Triage schedule is in backoff, but the commit re-checked only Incident failure and terminal Situation ownership. A clean skip committed in between is a first judgment that closes membership, so a later alert could attach behind it without any decision seeing it. The attach now carries the exact precondition it was planned under: the schedule row must still be in backoff inside the transaction (a missing row counts as not in backoff). Otherwise the plan is rejected with the existing not-collapsible sentinel and the correlator re-plans the delivery through its next attachment path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
…ace fix in the lab acceptance note Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
…rified live Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
…rts still fires deriveSymptoms took the status of the Incident's most recently received delivery, so one Alert resolving while a sibling still fired read as a resolved Incident and drove active -> recovery_pending -> recovered under a still-firing Alert (external review of the finished branch, P1). The rationale in Symptom's doc comment — no Alert identity reaches the pure layer — had been stale since Delivery.AlertID landed with the Task 4 review fixes; criticalAnchorEligible already reduced per Alert. Reduce per Alert: each distinct AlertID contributes only its chronologically latest delivery (deliveryLess, the same order MembershipDigest and criticalAnchorEligible use) and the Incident is firing while any Alert's latest delivery is firing. Key stays the Incident ID; FirstObservedAt is unchanged. MaterialFactHash and the symptom facts change only for mixed-status Incidents, which were exactly the wrong cases. Tests: the same-Alert resolution case keeps its resolved reading; new cases for a still-firing sibling, all-Alerts-resolved, a re-fire after a resolution, and two Reconcile-level cases (partial resolution stays active across cycles; full resolution reaches recovery_pending). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
…son-predicate scope revision External review of the finished branch raised two plan-versus-code discrepancies. Both resolve toward the spec, with the plan amended. Replay boundary 9: a crash strictly after CommitController's transaction committed but before Reconcile observed the result. The verbatim replay of the same commit against its now-stale claim fails closed with ErrSituationLeaseLost and changes nothing (spec: the commit lands only while owner and claim token still match); convergence comes from the next claim, which finds the committed Assessment and projects it without redispatch. The store-level test's doc comment now says why fail-closed is the intended contract. novel_symptom and terminal_uncertainty stay unreachable, now for the right reasons: novel_symptom needs persisted per-Situation symptom identity and a per-Alert symptom key that Plan 2 does not have; terminal_uncertainty has its deadline half (ObservationDeadlineAt) but no definition of the spec's "actionable uncertainty" conjunct, and the old comment claiming the deadline did not exist was stale. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
…seventh lab run Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
… landed commit Boundary 9's companion, from the external review's follow-up: age the duration class before the pre-crash commit and leave it alone after the restart, so the basis is unchanged and the proof is direct — startup replay plus one converge pass dispatch no L2 call, grow no dispatch rows, and produce exactly one revalidated_reuse Assessment bound to the new claim with the same hashes. Boundary 9 ages the class after the restart, which forces a fresh derivation and cannot prove this half on its own. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfKEvYSqoFFmL81L1Dsexs Signed-off-by: ernescz <ernescz@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One fenced controller commit per Situation cycle; Acute Triage moves onto a durable, bounded schedule under controller decisions.
Lab acceptance: seven runs recorded in the private Plan 2 lab-acceptance record (doctopus).