Benchmark follow-ups: unsupported assessment capabilities and learner access boundaries #26

Activity

1. rabesss

   added

   enhancementNew feature or requestNew feature or request

   on Sep 18, 2026
2. pullfrog commented on Sep 18, 2026

   

   pullfrogBot

   on Sep 18, 2026 – with Pullfrog

   Last edited by pullfrog

   More actions

   Evaluation

   Actionable, with a split: checkboxes 1, 3, 4 are buildable CLI capabilities; checkbox 2 is blocked by the upstream D2L API; checkbox 5 is an external provisioning/permissions block, not code. Duplicate check: the similarity lookup service errored (500), so I verified manually against the issue list — no duplicates; this builds on the closed #16/#17/#18 line and interacts with open #24 (preview-start P1) and #25 (submit site binding).

   Key evidence from the codebase:

   • submit (lighthouse_cli/cli.py:504 → submit.py) is file-only (-f/--file required); LighthouseClient.submit_file (api.py:1047) posts multipart to .../dropbox/folders/<id>/submissions/mysubmissions/. Instructors can already create text folders (assignment-create --submission-type text), so only the learner write is missing → checkbox 1 is a real gap.
   • quiz-create help and docs/assessment-coverage.md both record that the public Valence quiz API exposes question definitions read-only — no documented question-authoring route exists → checkbox 2 is upstream-blocked.
   • AssessmentAPI (assessment_api.py) has reads (questions, attempts, submissions) and creation writes, but no feedback/grading write, even though docs/assessment-coverage.md lists "instructor feedback/rubric grading and publication" as open work and the documented dropbox feedback routes exist → checkbox 3 is buildable.
   • Root courses/semesters/content (commands.py:1308/1340/1424) hardcode LighthouseClient() (lighthouse site); every instructor/student subcommand requires a known COURSE_ID, so --site trial has no discovery path → checkbox 4 is buildable and small.
   • The coverage doc's evidence section already establishes that no genuine learner identity exists (no impersonation control, stale sealed session 403s, D2L confirms no REST learner-quiz endpoints) → checkbox 5 stays blocked externally.

   Plan

   Keep this issue as the tracker (per its own Boundary note: one checkbox per track). Each code milestone lands as its own branch/PR (pullfrog/26-<kebab>) and checks off its box when merged.

   M1 — Role-scoped discovery (checkbox 4) — smallest, unblocks trial work

   1. Add a site parameter to cmd_courses, cmd_semesters, cmd_content (default "lighthouse"; root commands unchanged).
   2. Register courses, semesters, content in the instructor and student groups (extend the register_course_reads wiring in course_read_commands.py) so instructor --site trial courses works without a known course ID.
   3. Keep the role-group variant a plain enrollment listing — no local tracking-config/semester-filter semantics on the trial site.
   4. Tests: extend tests/test_assessment_workflows.py and tests/test_cmd_semesters.py; keep --json contract assertions (stdout-only JSON, stderr diagnostics).

   M2 — Learner text submission (checkbox 1)

   1. Fix or fold in Bind assignment submission to an explicit site or session #25 first: the new command should be site-scoped from the start — build it in the assessment module (student assignment-submit --text COURSE_ID FOLDER_ID) rather than extending the site-unbound root submit.
   2. Transport: add submit_text() beside submit_file() (api.py) — same documented route, JSON RichText part only, no file part; reuse _submission_response_result verification, cached-CSRF semantics, and the never-replay POST contract.
   3. Gates: confirm/--yes (stderr prompt in JSON mode), --dry-run, text length bound mirroring assignment_payload's 65536 limit.
   4. Known risk: the coverage doc records trial file submits returning 403 even with CSRF, so live validation likely needs a genuine learner (M5). Land code + unit tests; mark live validation blocked rather than claiming e2e.

   M3 — Instructor feedback/grading writes (checkbox 3)

   1. Verify the exact feedback verb/shape against the D2L reference (res/dropbox.html route family .../submissions/<submissionId>/feedback) before coding.
   2. Generalize AssessmentAPI.write() routing to nested paths (it currently maps only quiz/folder roots), reusing the CSRF bootstrap and AssessmentWriteUnknownError outcome-unknown semantics.
   3. Command: instructor feedback COURSE_ID FOLDER_ID SUBMISSION_ID --score N --text ... with confirm/--yes/--dry-run; pre-state is already readable via instructor submissions.
   4. Tests: outcome-unknown branches, CSRF handling, JSON contract, tests/test_secret_absence.py extension if any new credential-adjacent output appears.

   M4 — Quiz question authoring (checkbox 2) — document, don't build

   The public Valence API is read-only for quiz questions (confirmed by quiz-create help, the coverage doc, and D2L's own forum response). Deliverable: update docs/assessment-coverage.md and the checkbox decision — close as upstream-blocked unless an experimental web-route spike is explicitly wanted (that would be a separate issue and must honor the doc's guardrail: no guessed replay protocols).

   M5 — Genuine learner e2e (checkbox 5) — external block, define acceptance

   1. Prerequisites to record: admin-provisioned genuine learner identity (or authorized sample-student impersonation) on the trial; identification of which permissions caused the observed 403s (quiz question/attempt reads, assignment history).
   2. With that identity: re-run the 403'd reads, then evaluate generalizing the checkpointed preview driver into a student quiz flow — strictly after [P1] Trial preview start can create a remote attempt while CLI reports unknown and blocks resume #24 is fixed, and following the driver guardrails in docs/assessment-coverage.md (verify saves by readback, no blind retries, verified receipts or unknown outcomes).
   3. Until then: keep the checkbox open as an environment boundary, not a CLI defect.

   Cross-cutting (every milestone)

   • Mirror existing test patterns (CliRunner + unittest.mock, hermetic config dirs); pytest -q green is the gate; python3 -m ruff check --select F for the pyflakes subset.
   • --json contract: one JSON document on stdout, prompts/diagnostics on stderr, exit 0/1.
   • No secrets in output or exceptions; from __future__ import annotations, X | None, no bare Callable; no new dependency ranges.
   • Remote-write commands keep explicit confirmation, --yes for non-interactive use, and dry-run where feasible.

   Suggested order: M1 → M2 → M3 (independent but increasing size), M4 as a docs-only PR anytime, M5 parked on external provisioning.

   Task list (4/5 completed)

   • Gather context: issue comments, learnings, codebase structure
   • Map existing command surface (student/instructor groups) against the 5 checklist items
   • Retry duplicate detection
   • Draft structured plan (milestones per checklist item)
   • Post plan via report_progress

     ^{｜ Implement plan ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using openai-compatible/glm-5.3 ｜ 𝕏}
3. rabesss commented on Sep 18, 2026

   

   rabesss

   on Sep 18, 2026

   OwnerAuthor

   More actions

   The passive learner-side observation adds protocol evidence to this checklist, but does not complete learner E2E validation.

   Observed, without controlling the graded attempt: multipart POSTs to the legacy quiz save endpoint, then a confirmation GET, URL-encoded submission RPC, and receipt GET with non-preview mode. The retained responses were HTTP 200. No request/response bodies, question text, answers, tokens, or real course/quiz/attempt identifiers were retained.

   Capture limitations: the event buffer reported truncation, so the startup handoff is not established; payload fields and action semantics were not captured. This does not by itself diagnose #24, implement assignment submission (#25), supply a genuine trial learner, or verify the CLI's learner write behavior. Keep those acceptance criteria open. Follow-up execution remains restricted to synthetic trial fixtures; no graded requests will be replayed.
4. rabesss commented on Sep 22, 2026

   

   rabesss

   on Sep 22, 2026

   OwnerAuthor

   More actions

   Sequenced capability work packages, 2026-09-22

   Execution map: #31. Keep #26 as the umbrella; do not execute its old multi-feature outline as one branch.

   Existing checkbox	Concrete next issue	Completion boundary
   Instructor course/semester/content discovery	#33	Site-scoped read-only commands, lazy help, no production fallback
   Student text-assignment submission	#34	Supported protocol + explicit site + verified/unknown outcome; depends on #25
   Teacher grading / feedback	#35	First draft-only assignment-feedback slice; broader grading/rubrics/publication remain open
   Genuine learner E2E access	#36	Human-authorized provisioning + capability matrix; not completed by instructor previews or mocks
   Quiz question authoring	#37	Bounded supported-route feasibility; no guessed endpoint or promise to implement absent API

   First clear #32/#30 and finish #25/#28 + #27/#29. #24 is the next P1 correctness fix before expanding workflows. Discovery is the next low-risk capability; text and grading share modules and should be implemented serially. Learner access can be arranged independently but no account action or LMS write is authorized by this plan.

   Corrections / stop conditions

   • Do not invent a text field on the file-upload endpoint. [P2] Add learner text assignment submissions #34 starts by verifying a documented contract.
   • Do not assume a feedback endpoint saves a draft rather than publishing. [P2] Verify and implement draft-only instructor assignment feedback #35 has a hard protocol/visibility gate.
   • No public question-authoring route is currently established. [P3] Establish supported quiz question-authoring capability and blockers #37 is a decision record, not a request to replay private forms.
   • Genuine learner quiz driving is not implemented. Passive traffic evidence and instructor preview do not close that gap; no graded replay or learner-answer automation belongs in these tickets.
   • Preserve each checkbox until its full named scope is delivered. Closing a feasibility task or a first grading slice does not satisfy the broader capability.

   These handoffs include source entry points, dependency order, acceptance tests and external blockers so the next agent can skip historical PR rediscovery.
5. rabesss commented on Oct 15, 2026

   

   rabesss

   on Oct 15, 2026

   OwnerAuthor

   More actions

   Status of each box (2026-10-15)

   None is ticked yet; #31 has the current roadmap.

   • Student text-assignment submission: not supported by the CLI. Brightspace documents no REST route for a text-only learner submission; text entry exists only in the browser ([P2] Add learner text assignment submissions #34, closed). submit and submit --dry-run read the folder's settings and refuse text-only, on-paper and observed folders before reading or uploading the file (fix: refuse submit to a folder that takes no file #73).
   • Instructor quiz question authoring: on hold ([P3] Establish supported quiz question-authoring capability and blockers #37). LE 1.100 documents question create and update routes; Lighthouse reports LE 1.99.
   • Teacher grading and feedback writes: on hold ([P2] Verify and implement draft-only instructor assignment feedback #35).
   • Instructor course, semester and content discovery: covered by the top-level courses, semesters and content reads now that the CLI is Lighthouse-only (chore: make the CLI Lighthouse-only; drop the trial connection #46; [P2] Add role course, semester, and content discovery #33, closed).
   • Genuine learner end-to-end access: the learner quiz flow now exists (student attempt start, page, answer, next, previous, images, submit, verify, forget and status), so the note above that the student command tree has no quiz flow is out of date. It has been checked on a disposable sandbox and on public learner sites, not yet on a Lighthouse quiz. The trial-learner route ([Blocked: access] Authorize synthetic trial learner and record CLI E2E evidence #36) is closed: the trial could not provide a learner identity.