Add generic probe data model and private storage - #131
Open
Ely-S wants to merge 3 commits into
Open
Conversation
Introduce the shared, probe-agnostic contracts for runs, cohort members, source windows, units, provider attempts, evidence anchors, and claims. Add a transactional per-probe SQLite store that keeps raw responses, source text, and quotes private while preserving the identity, cache, accounting, and resume fields required by the second-pass engine. Co-authored-by: Cursor <cursoragent@cursor.com>
Record what the shipped data model and private schema actually contain, and where they deliberately diverge from the full design: probe_run stays in the private database, source_window is a table so claim membership is a foreign key rather than an engine check, and attempt carries no variant column. State plainly what V1 leaves out so a reader does not mistake the design for the build. Remove unfinished_unit_keys, which no caller reaches -- resume derives pending work from unit status. Correct the cohort-order docstring: the store persists whatever order it is handed and does not impose one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Eight review findings against the storage layer, none of which the existing callers would have surfaced on their own. Whitespace stripping applied to every string field, so a stored response body no longer matched the checksum written beside it and a stored source window no longer matched the corpus text it was selected from. Both are audit records, so add a non-stripping base for the models that carry private payloads. The default database path was relative to the working directory, and the WAL sidecars were not ignored. A probe run from a subdirectory wrote quote-bearing uncommitted pages to a tracked location. Anchor the default to the repository root and ignore the sidecars, so the L11 boundary holds structurally rather than by convention. Cache lookup ordered on ISO-8601 text, which is not a time ordering across offsets: an older attempt could win and serve a stale body. Order on a normalized UTC column instead. Also: run inserts now roll back instead of leaving the connection in an open write transaction; status updates on an unknown key raise rather than silently no-op, which would strand a unit as running and re-dispatch it on every resume; the connection is shared across threads under a lock, ahead of the engine's concurrency; and two indexes that restated their table's primary key are dropped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The second-pass extraction prototype combines probe-specific pharmacology logic, execution state, and persistence in one large script. That makes the next engine difficult to reuse and risks leaking quote-bearing source material into shared analysis artifacts.
Approach taken
data/probes/boundary.User-facing changes
This PR adds the reusable data and persistence layer only. It does not change the existing psychedelic study, invoke an LLM, or add CLI behavior; the planning/execution engine and probe migration are intentionally deferred to later stacked PRs.
Detailed test plan
uv run python -m compileall -q probesand verify success.ProbeRun,CohortMember,Unit,Attempt, andClaimobjects; persists them throughProbeStore; verifies accepted-response cache lookup; verifies unfinished-unit discovery; then marks the unit complete and verifies resume state is empty.LLM_PROVIDER=anthropic uv run pytest variable_extraction/tests -q; the current repository package suite reports 316 passing tests.git diff --name-only study/psychedelics...HEADcontains onlyprobes/__init__.py,probes/models.py, andprobes/store.pyfor this PR.The full repository test command is currently blocked during collection by an unrelated environment mismatch: an existing test imports a utility that rejects the pre-existing
LLM_PROVIDER=openaienvironment value.Made with Cursor