Repository navigation
Share the question-set validator and add v2 acceptance records - #53
Merged
Merged
Conversation
Preserve runtime validation paths, bounds, answer parsing and outcome rules. Freeze 1,049 invalid v1 cases from the original validator. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Require strict public hashes and aggregate evaluation counts for v2, retain EN/FR notices and v1 validation, and expose the band map for tuning. Evaluation and digest verification remain the evaluator and export gate responsibility. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Review: 1/6/5; fixes 1. Own the canonical digest preimages and minimum acceptance policy. Preserve compound v1 issue paths, refuse prototype-related keys, and ignore inherited author data. V2 carries explicit question order and a record per provider and language. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
aharlap
force-pushed
the
feat/shared-question-set
branch
from
October 20, 2026 01:42
60a9d83 to
fb66d19
Compare
Review: 1/6/5, 0/2/4; fixes 2. Share exact question-order validation and JSON request ordering, bind Choice option order in the digest, reject invalid course bars loudly, and require eight ordinary holdout rows alongside adversarial coverage. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review: 1/6/5, 0/2/4; fixes 2.
Share the dependency-free question-set owner with v1 migration support and v2 acceptance bound to each provider, model and language. Both versions retain the documented reserved-key hardening.
The final fix preserves authored question order in JSON requests, including numeric IDs, and binds Choice option order as ordered pairs in the public digest. The validator and request builder share exact-permutation validation. Invalid course bars throw; holdouts require eight ordinary rows in addition to gaming and injection minimums. Updated literal digest/fingerprint vectors cover ordered options.
Checks on Node 22.23.3: 17 focused shared tests, typecheck and generated-declaration checks pass. The runtime wire-order regression uses the actual provider request bytes. Full/browser suites remain in CI.
Runtime integration: https://github.com/Praxity/praxity-runtime/pull/10. No merge or deployment.