Plan 4: Bounded evidence preparation and semantic profiles - #86
Draft
ernescz wants to merge 13 commits into
Draft
Conversation
Freeze the transport-neutral shapes Plan 4 builds on: internal/observation/model (Fence, closed Capability/Phase/ResultStatus/ Freshness vocabularies, Plan/Cycle/Run/Fact shapes, CanonicalPlanID, MaterialDigest, ValidatePlan/ValidateRun) and internal/semanticprofile (advisory Signature/Profile/Version/JobClaim model, BuildSignature's proven-id/proven-id-only/fallback precedence, ParseProfile's duplicate-key-rejecting + strict-decode + bounds pipeline). Add the six situations.preparation/semantic_profiles config defaults, their range validation, and reference documentation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZWdCMiwFv57w2auUMWXUi Signed-off-by: ernescz <ernescz@gmail.com>
Add migration 0022 and internal/store/situation_preparation.go implementing the six Task 2 store APIs: BeginPreparation (freeze-or-reuse a cycle keyed by situation/input-version/generation, sealing-driven generation counter), ReserveObservationRequest (cycle+plan physical-request budget enforcement, investigation-credit debit for the protected optional plan), CompleteObservationRequest (fence-free durable outcome recording), CommitObservationRun (idempotent replay, conflict detection, reference-lifecycle bookkeeping), ListObservationRuns (cursor pagination, archive-safe detail_state), and PruneUnusedObservationDetails (the ten-day unused-detail retention sweep, ADR-0051). Wire cycle sealing into CommitController's existing fenced transaction via sealPreparationCycleTx, and add the two new ControllerCommit fields it reads. Found and fixed along the way: a SetMaxOpenConns(1) self-deadlock from querying while an outer rows cursor was open in ListObservationRuns, and a reference- lifecycle gap where a run's open_cycle protection never lifted (blocking retention indefinitely) -- sealing now transitions it to a current_cycle reference, superseded only when a later cycle begins. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZWdCMiwFv57w2auUMWXUi Signed-off-by: ernescz <ernescz@gmail.com>
Add internal/observation's deterministic planner (BuildPlans), the investigative-fairness budget allocator (allocateFairly: time-sensitive lifecycle work first without limit, one protected optional plan admitted only when both remaining cycle capacity and durable credit cover its full MaxRequests, routine lifecycle filling what remains, least-recently-served first within each tier), and the Runner that executes a frozen cycle's plans through registered capability executors -- reusing any plan with an existing committed run, committing a vocabulary_unresolved run for a capability with no registered executor (no I/O), and a failed run for an executor error, so one plan's failure never blocks the rest of the phase. Add the store_read capability's executor (internal/observation/connectors): bounded prior-Situation summaries and durable findings from AlertINT's own local store, never spending the RequestRecorder's physical-request budget (a local SQL read, not a physical network slot). Add the corresponding LocalSituationSummary/LocalFinding/LocalChange model DTOs (LocalChange is change_events' own local-read shape, added here so internal/store's two new bounded local-read methods share one leaf dependency). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZWdCMiwFv57w2auUMWXUi Signed-off-by: ernescz <ernescz@gmail.com>
Add the Task 4 connectors and their bounded upstream client APIs: - prometheus.Client.QueryRangeBounded (server-side series cap on range queries, mirroring QueryInstant's existing limit param). - loki.Client.FetchRecentBounded (existing selector/label-map translation and filtered+fallback logic, with each physical request -- including the fallback -- individually reserved/recorded rather than hidden behind one apparent call). - sentry.Client.doGETInstrumented + ListIssuesBounded (per-physical-attempt instrumentation covering 429/5xx retries, plus Link-header pagination metadata ListIssues previously discarded). - store.ChangesInScopeWindow (a new scoped, limited query against the local change ledger -- never the unbounded, installation-wide ChangesInWindow). Each observation.Executor builds a deterministic query from the frozen plan's scope, persists bounded normalized summaries (never a raw matrix, full log dump, or raw exception text unless IncludeMessage is configured), and reports the closed nine-value result vocabulary including truncation and withheld-by-budget. store_read and change_events never spend the RequestRecorder's physical-request budget (local reads); the other three account for every physical dispatch individually. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZWdCMiwFv57w2auUMWXUi Signed-off-by: ernescz <ernescz@gmail.com>
Add internal/zabbix/problem_history.go (ProblemHistory, EventLifecycle) using Zabbix 7.0's event.get for historical resolved events (problem.get only retains unresolved/recently resolved problems): exact host/trigger event timeline, paired recovery event times via a batched secondary lookup, ongoing/unknown state, severity/acknowledgement/suppression, tags, and cause-event id. A failing or budget-withheld recovery lookup degrades the affected episodes to RecoveryUnknown rather than losing the already- fetched primary evidence. Add callInstrumented/MetricHistoryBounded so every physical request this package makes -- including secondary item/ recovery lookups -- is individually reservable. Add the zabbix_metric_range and zabbix_problem_history observation.Executor adapters, reading exact host/item-key or host/trigger-ID identifiers from each plan's typed Parameters (never a model-invented metric key or fuzzy cross-host fallback), with host falling back to the plan's own Scope SubjectID when omitted. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZWdCMiwFv57w2auUMWXUi Signed-off-by: ernescz <ernescz@gmail.com>
Wires EvidencePreparer into Reconcile's control flow: a lifecycle-phase prepare+reload runs first (guarded by a nonterminal Situation), source lifecycle is reduced across every expected member Alert, and an assessment-phase prepare+reload runs only while that reduction leaves the Situation active. A nil preparer (every pre-Task-6 fixture) skips both phases entirely, keeping local-only Reconcile behavior unchanged. resolveLifecycle now branches on whether a preparation cycle exists for this input: prepared evidence (source acquisition mode/interval, never Delivery.StartedAtBasis alone) becomes authoritative, replacing the local Delivery/Symptom inference only when it is actually present. Store-side, LoadReconciliationInput now reloads the current preparation cycle (Runs/Facts, frozen profile guidance, and source_lifecycle facts decoded into SourceObservation) inside its own coherent transaction — SnapshotInput.Prepared, never an in-memory EvidencePreparer receipt. Also fixes a Task 3 naming bug: store_read's prior-situation fact was mislabeled Kind "source_lifecycle" (migration 0022's reserved kind for this task's own per-Alert evidence), colliding with genuine lifecycle facts; it now uses the same generic "capability_result" kind findingsFact already does. Bumps fact schema to 2, material hash to 3, Assessment-basis hash to 4, and validator to 2; replaces the blanket Plan 2 limitation catalog with the closed, still-reserved Plan 5 policy set. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZWdCMiwFv57w2auUMWXUi Signed-off-by: ernescz <ernescz@gmail.com>
Adds migration 0023: the append-only delivery-to-signature mapping, immutable Profile versions with a CAS-protected head, the deduplicated inference-job/call/outcome ledger, and the head-change fan-out outbox (spec.md "Source identity and advisory signatures" and "Durable semantic-profile inference"). ApplySituationInput now attaches a delivery's deterministic advisory signature (and, only for a genuinely new signature, enqueues its pending inference job with a frozen semantic input) inside its own existing transaction. BackfillActiveSemanticMappings recovers any missed attachment for active/recovery_pending followers in bounded batches, skipping terminal-only episodes. CorrectSemanticProfile applies a CAS-protected operator override; ClaimSemanticInferenceJob/ReserveSemanticInferenceCall/ CompleteSemanticInference give Task 8's worker its dispatch surface, downgrading a late accepted inference to stale when a correction (or a sibling job) already advanced the head first. RecoverSemanticInference releases stranded leases, recovering a crash on the final attempt directly as exhausted. DeliverSemanticProfileChanges fans a head change out to every matching nonterminal Situation, paginated 100 per transaction, merging the existing semantic_profile_changed due reason. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZWdCMiwFv57w2auUMWXUi Signed-off-by: ernescz <ernescz@gmail.com>
Adds llm.InferenceLimiter: bounds concurrent Assessment (L2) and semantic-profile (L0) provider calls to one shared capacity, capping profile dispatch at one concurrently-held slot regardless of total capacity and granting a queued Assessment first whenever both are waiting. internal/situation.ControllerWorker now gates its own L2 dispatch through this SAME shared limiter when SetInferenceLimiter wires one in (falling back to its private per-worker semaphore otherwise, unchanged from before). internal/llmhealth learns CapabilitySemanticProfile: a shared-primary capability (a real success on it, or on triage_draft/assessment, clears the others' dependency-class failures) that drives Degraded rather than Unavailable when unhealthy — advisory guidance going dark must not declare the core loop down. Migration 0024 widens the capability CHECK constraint (SQLite cannot ALTER one in place). internal/semanticprofile gains BuildInferencePrompt (renders the frozen semantic input and the closed eight-field schema, no tools, no model-authored queries) and Worker: claims one durable inference job at a time, reserves exactly one dispatch before the request, heartbeats the job's own lease while the call is in flight, and commits a typed outcome (accepted/malformed/failed) through the shared limiter with llmhealth observation — a store-side CAS loss downgrading a late accepted result to stale is recorded as healthy transport, since the model itself answered correctly. internal/store gains ClaimSemanticInferenceJob/ReserveSemanticInferenceCall/ CompleteSemanticInference/ExtendSemanticInferenceJobLease, reusing semanticprofile.ErrAttemptsExhausted directly so *Store structurally satisfies Worker's own ProfileStore interface with no adapter shim. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZWdCMiwFv57w2auUMWXUi Signed-off-by: ernescz <ernescz@gmail.com>
Wire the real EvidencePreparer adapter, semantic-profile inference workers, and bounded sweeps into cmd/alertint (buildPreparationRuntime), inject the preparer into every controller before workers start, and recover/drain/stop it alongside the rest of the Situation runtime. Add the three spec-named MCP tools: alertint_list_observation_runs, alertint_get_semantic_profile, alertint_correct_semantic_profile (the only new write path this plan adds). Add audit rows for cycle begun/profile dispatch/outcome/head advance/change delivery/correction, and OTel spans situation.preparation, situation.observation, and semantic_profile.inference on their owning packages' existing tracer scopes. Fix a shared-preparation-wall gap found while wiring: Prepare() now applies MaxWallSeconds as its own sub-deadline from the shared req.Now anchor, so a phase that already spent part of the cycle's wall correctly leaves less of it for the other. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZWdCMiwFv57w2auUMWXUi Signed-off-by: ernescz <ernescz@gmail.com>
…ed-list upgrade reservedUnsupportedCapabilities is build-scoped, not input-derived, so an old still-nonterminal Situation reconciled by a newer binary observes a different limitation list at the SAME (situationID, inputVersion) an older binary already committed a capability_limitation fact for. Discovered live during Plan 4 lab acceptance: reconciling four pre-Plan-4 Situations against the upgraded candidate failed AppendSituationFacts' own immutable-conflict check (same ID, different Value) on every cycle -- not a crash-only edge case, ordinary post-upgrade operation. Fix mirrors the existing factIdentityWithContent precedent (incident_triage_state, acute_finding): fold the fact's own content digest into its ID so a genuinely distinct observation gets its own row. MaterialFactHash computes the reserved-capability contribution independently from the package var, never from this fact's ID, so the fix has no effect on material-hash stability or Assessment reuse. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZWdCMiwFv57w2auUMWXUi Signed-off-by: ernescz <ernescz@gmail.com>
The synchronous alertint_correct_semantic_profile write response echoed CreatedAt as the zero value even though the row it just durably committed carries the real correction instant -- an MCP caller reading the write's own response (not a follow-up alertint_get_semantic_profile read) saw "0001-01-01T00:00:00Z" for a correction that had genuinely just landed. Discovered live during Plan 4 lab acceptance. The persisted row and every subsequent read were always correct; only the write's own return value was affected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZWdCMiwFv57w2auUMWXUi Signed-off-by: ernescz <ernescz@gmail.com>
Real end-to-end proof that the production preparation/semantic-profile wiring actually works together: real store, real ingress HTTP receiver, real correlator/input workers, real ControllerWorker with the real productionPreparer/observation.Runner/semanticprofile.Worker this package builds for production, hitting a fake Prometheus HTTP server. Covers a full pipeline from HTTP alert to committed run and profile head, a crash that reuses the same frozen cycle, a stranded inference-job lease that startup recovery resumes, and a connector outage that commits an honest failed run without blocking the Situation's own lifecycle commit. Deliberately a smaller, hand-written set of end-to-end tests rather than a generic JSON-fixture-driven scenario corpus -- a scope reduction agreed with the user to spend the freed budget on real lab SSH acceptance instead (doctopus lab-acceptance.md; two real bugs found and fixed live during that run, in separate commits on this branch). Update public docs with the actual now-wired capabilities: bounded evidence preparation, advisory semantic profiles, the three new MCP tools, the shared L0+L2 limiter, retention, and source lifecycle timestamp semantics. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZWdCMiwFv57w2auUMWXUi Signed-off-by: ernescz <ernescz@gmail.com>
4 tasks
* fix(situation): apply assessment stability hotfix to Plan 4 lab Port e41b63b onto the deployed Plan 4 baseline. Preserve observation hashing and advance material/basis schema versions to 4/5 because Plan 4 already uses 3/4. Signed-off-by: ernescz <ernescz@gmail.com> * fix: prevent assessment evidence churn and bound shared LLM usage Signed-off-by: ernescz <ernescz@gmail.com> * docs: record assessment reuse and LLM budget hotfix Signed-off-by: ernescz <ernescz@gmail.com> --------- Signed-off-by: ernescz <ernescz@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP: Bounded evidence gathering and advisory semantic profiles for the Situation controller.
Implementation and validation are ongoing. Not ready to merge.