Docs: full-corpus prompt eval polish (ship A) - #2448
Closed
monicasmith463 wants to merge 275 commits into
Closed
Conversation
api part 2: port over remaining endpoints for exam, exam attempts
…rplate Frontend: new boilerplate containerized
…rate Feature: frontend Dropzone uploader page
Feature: frontend Take Exam form
Internal: delete old admin dashboard code
Feature: final frontend polish
…PI URL for frontend
Infra: update docker compose for prod deploy
Docs: new README and LICENSE
Readme: add a "try in 3 minutes" section
Route production through a neutral generate_questions helper and keep prompt A’s any-mix behavior, so eval can reuse the same path later. Co-authored-by: Cursor <cursoragent@cursor.com>
Return short fixed errors to callers and log only exception types so raw model output cannot leak through API responses. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep baseline A unchanged, make B difficulty-calibration-only, make C distractor-quality-only, and drop the grounding variant. Co-authored-by: Cursor <cursoragent@cursor.com>
Call shared generate_questions over a local .txt corpus and write timestamped JSON artifacts with recorded generation controls. Co-authored-by: Cursor <cursoragent@cursor.com>
Scope exams, documents, and users to the current owner only, and drop unimplemented password-reset and test-email routes that were privilege hazards. Co-authored-by: Cursor <cursoragent@cursor.com>
Tests now create documents/exams under the authenticated bootstrap user instead of assuming superuser bypass. Also remove the FastAPI org add-to-project action that fails without PROJECTS_TOKEN. Co-authored-by: Cursor <cursoragent@cursor.com>
…leges Remove superuser data access and unused auth endpoints
Add report summaries, a small corpus and rubric, ignore run artifacts, and document methodology aligned with measurable prompt evaluation. Co-authored-by: Cursor <cursoragent@cursor.com>
Cast content-check fields to int/float so _final_contract_valid returns a bool instead of Any. Co-authored-by: Cursor <cursoragent@cursor.com>
Drop the unused structured question LLM at import time and branch HTTP details on error_kind so API messages cannot drift with error wording. Co-authored-by: Cursor <cursoragent@cursor.com>
Cursor: shared prompt registry
* Enforce answer ∈ options before persisting generated questions. Rejects T/F and MC items whose answer is not a selectable option so bad structured output fails validation instead of breaking grading. Co-authored-by: Cursor <cursoragent@cursor.com> * Expand eval corpus and record ship/no-ship takeaway. Documents the answer∈options bug→fix story, grows the lecture corpus to 13 docs, and scores prompt A as the production choice from the existing smoke artifacts. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
Document the post-validator 13-doc run, sampled manual rubric, and align README takeaway with ship-A evidence. Co-authored-by: Cursor <cursoragent@cursor.com>
Author
|
Opened by mistake against upstream template; closing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
backend/evals/rubric.mdwith sampled manual scores from full-corpus run2026-08-21T01-52-36Z(4 docs / ~20 questions per prompt) and ship A.backend/evals/DEVELOPMENT.mdwith post-validator results (13 docs; success ~77/77/85; answer∈options 100% on successes).Test plan
.envandresults/are not in the PRMade with Cursor