Skip to content

feat(lattice): add miscalibration detector using Brier score - #673

Open
Aqsa30nz wants to merge 1 commit into
sreerevanth:mainfrom
Aqsa30nz:feat/miscalibration-detector
Open

feat(lattice): add miscalibration detector using Brier score#673
Aqsa30nz wants to merge 1 commit into
sreerevanth:mainfrom
Aqsa30nz:feat/miscalibration-detector

Conversation

@Aqsa30nz

@Aqsa30nz Aqsa30nz commented Jul 31, 2026

Copy link
Copy Markdown

Summary

Implements the session-level MiscalibrationDetector for the v2 Cognitive Lattice.

Fixes #661

Changes

  • Added CalibrationEntry to store confidence/outcome pairs.
  • Added MiscalibrationDetector for tracking session-level calibration.
  • Implemented Brier Score calculation.
  • Added configurable warning and blocking thresholds.
  • Added evaluate() returning a MiscalibrationResult.
  • Exported the new lattice components via agentwatch.lattice.
  • Added unit tests covering:
    • empty history
    • recording predictions
    • invalid confidence values
    • well-calibrated sessions
    • warning threshold
    • blocking threshold
    • sample size reporting

Testing

pytest tests/test_miscalibration.py -v
pytest tests/test_shadow_filesystem.py tests/test_miscalibration.py -v

Both test suites pass successfully.

@SHAURYASANYAL3

Copy link
Copy Markdown
Collaborator

This miscalibration detector looks like a solid addition to the V2 Cognitive Lattice. However, there are merge conflicts. Please resolve them and update the PR so I can review the final diff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v2] Cognitive Lattice: Build Miscalibration Detector

2 participants