[v2] Cognitive Lattice: Build Miscalibration Detector

[SHAURYASANYAL3]

Phase 4: Cognitive Lattice

The Context

Overconfidence is a major LLM flaw. An agent might state "I am 95% confident this will fix the bug" but fail repeatedly. We need to track calibration mathematically.

Tasks

• Implement MiscalibrationDetector.
• Maintain a history of the agent's stated confidence per step vs actual success/failure outcomes.
• Calculate the Brier Score for the session (mean squared difference between stated probability and actual outcome).
• Enforce an invariant to trigger a warning or block if the Brier Score indicates severe miscalibration (e.g., score > 0.25).

Part of Epic #651

[prasiddhi-105]

Hiii please assign this issue to me under SSOC'26!!!

[SHAURYASANYAL3]

@prasiddhi-105 Love the enthusiasm, but we have a strict limit of 2 active issues per contributor to make sure everyone gets a chance. Let's focus on the first two you requested for now!

[SHAURYASANYAL3]

Hey @prasiddhi-105, we just assigned you to two other issues. Since we have a strict limit of 2 active issues per contributor, we can't assign you to this one right now. Once you finish your current tasks, feel free to come back!

[rohitkumarnaidu]

Hey maintainer, I'd love to work on this issue and submit a PR. Could you please assign it to me?

[rohitkumarnaidu]

Hi! Could you please add the ELUSOC label to this so it gets tracked properly by the dashboard? Thanks!

[arcgod-design]

Hi @SHAURYASANYAL3! I'd love to build the MiscalibrationDetector for AgentWatch v2 under SSoC '26. I'm @arcgod-design.

My Approach:

• Implement MiscalibrationDetector that maintains a session-level history of the agent's stated confidence score per step alongside the actual success/failure outcome
• Compute the Brier Score (mean squared error between stated probability and binary outcome) after each step
• Enforce a configurable invariant: emit WARNING when Brier Score exceeds a soft threshold (e.g., 0.15), and BLOCKED when it exceeds a hard threshold (e.g., 0.25)
• Write unit tests covering well-calibrated sessions (no block), gradual drift (warn), and severe miscalibration (block)

Could you please assign this to me (@arcgod-design)? Ready to start immediately!

[Aqsa30nz]

Hi @SHAURYASANYAL3! I'd like to work on this issue.

I've reviewed the v2 Invariant Lattice architecture and understand that this detector should remain fully deterministic, lightweight, and require zero LLM calls.

My implementation plan is to:

• Implement a dedicated MiscalibrationDetector with a single responsibility.
• Compute the session's Brier Score by comparing the agent's stated confidence against actual binary outcomes.
• Support configurable warning and blocking thresholds while keeping the implementation deterministic.
• Handle edge cases such as empty histories and invalid confidence values.
• Add comprehensive unit tests covering calibrated, warning, blocking, boundary, and invalid-input scenarios.

I'll keep the PR focused, well-tested, and aligned with the existing project architecture and coding style.

If this approach aligns with the intended design, I'd be happy to take this issue. Thank you!

[sreerevanth]

@Aqsa30nz please make an pr asap

[sreerevanth]

@Aqsa30nz if u havent joined the discord please join the discord https://discord.gg/eCDag2dG

[Aqsa30nz]

@sreerevanth Thanks! I'll start working on it right away and will open a PR as soon as possible