Skip to content

Add preregistered Hamlet replication - #14

Merged
almondsun merged 2 commits into
mainfrom
agent/hamlet-external-replication
Jul 13, 2026
Merged

almondsun merged 2 commits into
mainfrom
agent/hamlet-external-replication

Conversation

@almondsun

Copy link
Copy Markdown
Owner

What changed

  • preregister a Hamlet external-distribution comparison before corpus access in commit d4a17dc
  • add frozen character and ByteBPE512 80/10/10 configurations
  • execute both training jobs before one-shot sealed-test access
  • record exact corpus/checkpoint provenance, baselines, validation behavior, test results, limitations, and commands
  • extend the theory notes with preregistration, effect heterogeneity, and chronological difficulty
  • update reviewer paths and current experiment status

Why

Milestone 026 confirmed ByteBPE512 on two narrative-prose books but left distributional breadth as the main uncertainty. Hamlet introduces dramatic dialogue, verse, speaker labels, and stage directions under the unchanged sealed protocol.

Result

ByteBPE512 reaches test BPC 2.2546152063 versus character 2.3218808622, for the preregistered contrast -0.0672656559 BPC. The directional hypothesis is supported. Both Hamlet test results are better than validation, reversing milestone 026's harder-tail pattern and showing that chronological difficulty is corpus-dependent.

Validation

  • deterministic Project Gutenberg marker extraction and SHA-256 provenance
  • train-only tokenizer fitting and validation-only baselines
  • both frozen training runs completed before either test evaluation
  • both one-shot artifacts report full target-token coverage
  • make PYTHON=.venv/bin/python check — 163 tests, 91.00% coverage
  • make PYTHON=.venv/bin/python audit — no known third-party vulnerabilities
  • git diff --check

Ignored corpora, tokenizers, checkpoints, and run directories are not included.

@almondsun
almondsun marked this pull request as ready for review July 13, 2026 07:09
Copilot AI review requested due to automatic review settings July 13, 2026 07:09
@almondsun
almondsun merged commit 3f2a4dc into main Jul 13, 2026
5 checks passed
@almondsun
almondsun deleted the agent/hamlet-external-replication branch July 13, 2026 07:09

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants