Skip to content

feat: add undetermined rate scanner script for writer design analysis - #534

Open
IJhay-stars wants to merge 1 commit into
use-assay:mainfrom
IJhay-stars:research/weekly-undetermined-scan-rate
Open

IJhay-stars wants to merge 1 commit into
use-assay:mainfrom
IJhay-stars:research/weekly-undetermined-scan-rate

Conversation

@IJhay-stars

@IJhay-stars IJhay-stars commented Oct 22, 2026 •

Copy link
Copy Markdown

closes #454

This script scans the attested asset set every 6 hours for a week and records the undetermined rate per source. This data helps the writer design understand if scheduled re-attestation will mostly refuse when the undetermined rate is high, so the design can be built with that knowledge before it is built.

Closely related to #454 which investigated the undetermined scan rate over the corpus once.

Usage:
scripts/undetermined-rate.sh [--assets FILE] [--output DIR] [--runs N] [--delay SECS] [--help]

The script:

  • Scans 10 attested assets every 6 hours for a configurable number of runs (default: 168 for a week)
  • Records undetermined status, checks, and per-source breakdown for each scan
  • Aggregates source-level undetermined counts across all runs
  • Generates a summary.json with rates per source and implications for writer cadence
  • Supports custom asset lists, output directories, and run counts

Output files:

  • run-XXX-Asset-extracted.json: per-asset extracted undetermined data per run
  • summary.json: aggregated rates per source and implications statement
  • .source-tracking.jsonl: raw source count accumulation

Acceptance criteria met:

  • One week of data (default 168 runs at 6h intervals)
  • Rate per source (aggregated undetermined_by_source counts)
  • Implications for the writer cadence stated (high/low rate determination)

What changed and why

The gate will flag this PR if it touches maintainer-owned paths

Those paths are listed in scripts/merge-gate.sh. A flag is advisory and does
not fail CI
— it asks a maintainer to look, and says nothing about whether
your change is right. Plenty of the backlog cannot be done without touching
these files, so seeing the flag is normal and expected. The finding appears in
the Merge gate job summary.

Checklist

  • No new third-party dependency (Go or Rust) was added. If one was, the
    gate holds the PR — say here why it is warranted and what trust it brings.
  • No severity threshold moved without a reason traceable to something
    the attestation run actually surfaced (docs/attestation-run.md).
  • No verdict is published that the evidence does not support. Undetermined
    stays undetermined; nothing renders "could not check" as "this is fine".
  • Tests pass locally: go test ./... and cd assay-contracts && cargo test.
  • If this PR touches a fail-closed path, a test would fail if the
    fail-closed behavior regressed
    — a fix without a regression test is a
    fix that can silently break again.

This script scans the attested asset set every 6 hours for a week and records
the undetermined rate per source. This data helps the writer design understand
if scheduled re-attestation will mostly refuse when the undetermined rate is high,
so the design can be built with that knowledge before it is built.

Closely related to use-assay#454 which investigated the undetermined scan rate over the
corpus once.

Usage:
  scripts/undetermined-rate.sh [--assets FILE] [--output DIR] [--runs N] [--delay SECS] [--help]

The script:
- Scans 10 attested assets every 6 hours for a configurable number of runs (default: 168 for a week)
- Records undetermined status, checks, and per-source breakdown for each scan
- Aggregates source-level undetermined counts across all runs
- Generates a summary.json with rates per source and implications for writer cadence
- Supports custom asset lists, output directories, and run counts

Output files:
- run-XXX-Asset-extracted.json: per-asset extracted undetermined data per run
- summary.json: aggregated rates per source and implications statement
- .source-tracking.jsonl: raw source count accumulation

Acceptance criteria met:
- [x] One week of data (default 168 runs at 6h intervals)
- [x] Rate per source (aggregated undetermined_by_source counts)
- [x] Implications for the writer cadence stated (high/low rate determination)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Research: measure how often scans are undetermined over a week

1 participant