diff --git a/AGENTS.md b/AGENTS.md index ab83552..5545f4b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -90,10 +90,11 @@ If a relevant check cannot be run, state why and what remains unverified. ## Current Technical Status -Milestone 027 is the latest modeling evidence. It was preregistered before Hamlet corpus access and -confirms ByteBPE512 over character by `0.0673` sealed-test BPC on a dramatic play. Hamlet test BPC -is better than validation for both models, reversing milestone 026's harder-tail pattern. Alice, -Peter Pan, and Hamlet tests are consumed and must not guide further tuning. +Milestone 028 is the latest modeling evidence. Its preregistered 2-corpus × 2-tokenizer × 3-seed +panel finds ByteBPE512 wins all six sealed-test pairs, averaging `0.1134` BPC better on Art of War +and `0.1543` on Lincoln. Both new terminal regions are harder than validation, and effect magnitude +remains corpus-dependent. All previously reported sealed-test segments are consumed and must not +drive later selection. ## Safety And Security diff --git a/README.md b/README.md index 4efd872..c8ec2e2 100644 --- a/README.md +++ b/README.md @@ -17,8 +17,8 @@ For a fast technical review, inspect: 1. [`docs/architecture.md`](docs/architecture.md) for boundaries and the end-to-end pipeline. 2. [`docs/experiments.md`](docs/experiments.md) for the milestone index. -3. [`experiments/027-hamlet-external-distribution.md`](experiments/027-hamlet-external-distribution.md) - for the latest preregistered external-distribution evidence and its limitations. +3. [`experiments/028-preregistered-external-corpus-panel.md`](experiments/028-preregistered-external-corpus-panel.md) + for the latest preregistered multi-corpus, multi-seed evidence and its limitations. 4. [`src/smallm/model/`](src/smallm/model/) for the from-scratch GPTiny model. 5. [`src/smallm/training/`](src/smallm/training/) for training and run artifacts. 6. [`src/smallm/data/`](src/smallm/data/) for corpus and tokenizer contracts. @@ -52,6 +52,7 @@ chronological 90/10 split unless noted otherwise. | [025](experiments/025-corpus-by-seed-matrix.md) | 2 tokenizers × 2 corpora × 3 seeds | Paired ByteBPE512 minus character BPC | `-0.0619` Alice; `-0.0252` Peter Pan | ByteBPE512 wins all six pairs; effect magnitude is corpus-dependent. | | [026](experiments/026-sealed-test-evaluation.md) | Frozen decision on terminal 10% test segments | ByteBPE512 minus character test BPC | `-0.0614` Alice; `-0.0258` Peter Pan | The tokenizer decision survives one-shot full-coverage tests on both books. | | [027](experiments/027-hamlet-external-distribution.md) | Preregistered Hamlet play replication | ByteBPE512 minus character test BPC | `-0.0673` | The direction replicates on drama, while validation/test difficulty and effect size remain corpus-dependent. | +| [028](experiments/028-preregistered-external-corpus-panel.md) | 2 new corpora × 2 tokenizers × 3 seeds | Mean paired ByteBPE512 minus character test BPC | `-0.1134` Art of War; `-0.1543` Lincoln | ByteBPE512 wins all six preregistered pairs; magnitude and chronological-tail difficulty remain corpus-dependent. | Token-level loss and perplexity are not directly comparable between character and BPE tokenizers because they predict different units. The tokenizer @@ -86,7 +87,7 @@ shapes link directly to the implementation and its tests. | Corpus preparation | Normalized corpus output, stats, checksums, source metadata, and manifest files. | | Tokenization | Character, educational character-BPE, and lossless boundary-aware UTF-8 byte-BPE tokenizers. | | Model | Decoder-only GPT-style Transformer with causal self-attention. | -| Evaluation | Uniform, unigram, and add-one bigram baselines; optional sealed chronological test evaluation. | +| Evaluation | Uniform, unigram, and add-one bigram baselines; sealed chronological tests and complete factorial-matrix aggregation. | | Training | Config-driven training with validation loss, optional early stopping, progress logging, checkpoints, metrics, summaries, and samples. | | Run records | Preserved run directories with copied dataset manifests and selected provenance fields in `summary.json`. | | Generation | `max_new_tokens`, `temperature`, `top_k`, `seed`, and greedy decoding. | diff --git a/configs/gptiny_artofwar_bytebpe512_sealed_seed1337.yaml b/configs/gptiny_artofwar_bytebpe512_sealed_seed1337.yaml new file mode 100644 index 0000000..1cbbef0 --- /dev/null +++ b/configs/gptiny_artofwar_bytebpe512_sealed_seed1337.yaml @@ -0,0 +1,40 @@ +data: + input_path: data/raw/art_of_war_body.txt + prepared_path: data/processed/art_of_war_corpus.txt + manifest_path: data/processed/art_of_war_corpus_sealed_manifest.json + tokenizer_path: data/processed/art_of_war_tokenizer_bytebpe512_sealed.json + tokenizer_type: byte_bpe + bpe_vocab_size: 512 + bpe_min_frequency: 2 + block_size: 37 + train_split: 0.8 + validation_split: 0.1 + +model: + vocab_size: 512 + block_size: 37 + n_layer: 4 + n_head: 4 + n_embd: 128 + dropout: 0.1 + +train: + run_name: gptiny_artofwar_bytebpe512_sealed_seed1337 + runs_dir: runs + batch_size: 27 + max_steps: 5000 + learning_rate: 0.001 + weight_decay: 0.0 + log_interval: 100 + eval_interval: 250 + eval_batches: null + early_stopping_patience: 3 + early_stopping_min_delta: 0.0 + sample_prompt: "The art" + sample_max_new_tokens: 100 + sample_temperature: 1.0 + sample_top_k: null + sample_seed: 1337 + sample_greedy: false + seed: 1337 + diff --git a/configs/gptiny_artofwar_bytebpe512_sealed_seed2027.yaml b/configs/gptiny_artofwar_bytebpe512_sealed_seed2027.yaml new file mode 100644 index 0000000..b9c193d --- /dev/null +++ b/configs/gptiny_artofwar_bytebpe512_sealed_seed2027.yaml @@ -0,0 +1,40 @@ +data: + input_path: data/raw/art_of_war_body.txt + prepared_path: data/processed/art_of_war_corpus.txt + manifest_path: data/processed/art_of_war_corpus_sealed_manifest.json + tokenizer_path: data/processed/art_of_war_tokenizer_bytebpe512_sealed.json + tokenizer_type: byte_bpe + bpe_vocab_size: 512 + bpe_min_frequency: 2 + block_size: 37 + train_split: 0.8 + validation_split: 0.1 + +model: + vocab_size: 512 + block_size: 37 + n_layer: 4 + n_head: 4 + n_embd: 128 + dropout: 0.1 + +train: + run_name: gptiny_artofwar_bytebpe512_sealed_seed2027 + runs_dir: runs + batch_size: 27 + max_steps: 5000 + learning_rate: 0.001 + weight_decay: 0.0 + log_interval: 100 + eval_interval: 250 + eval_batches: null + early_stopping_patience: 3 + early_stopping_min_delta: 0.0 + sample_prompt: "The art" + sample_max_new_tokens: 100 + sample_temperature: 1.0 + sample_top_k: null + sample_seed: 2027 + sample_greedy: false + seed: 2027 + diff --git a/configs/gptiny_artofwar_bytebpe512_sealed_seed4242.yaml b/configs/gptiny_artofwar_bytebpe512_sealed_seed4242.yaml new file mode 100644 index 0000000..8bf10d4 --- /dev/null +++ b/configs/gptiny_artofwar_bytebpe512_sealed_seed4242.yaml @@ -0,0 +1,40 @@ +data: + input_path: data/raw/art_of_war_body.txt + prepared_path: data/processed/art_of_war_corpus.txt + manifest_path: data/processed/art_of_war_corpus_sealed_manifest.json + tokenizer_path: data/processed/art_of_war_tokenizer_bytebpe512_sealed.json + tokenizer_type: byte_bpe + bpe_vocab_size: 512 + bpe_min_frequency: 2 + block_size: 37 + train_split: 0.8 + validation_split: 0.1 + +model: + vocab_size: 512 + block_size: 37 + n_layer: 4 + n_head: 4 + n_embd: 128 + dropout: 0.1 + +train: + run_name: gptiny_artofwar_bytebpe512_sealed_seed4242 + runs_dir: runs + batch_size: 27 + max_steps: 5000 + learning_rate: 0.001 + weight_decay: 0.0 + log_interval: 100 + eval_interval: 250 + eval_batches: null + early_stopping_patience: 3 + early_stopping_min_delta: 0.0 + sample_prompt: "The art" + sample_max_new_tokens: 100 + sample_temperature: 1.0 + sample_top_k: null + sample_seed: 4242 + sample_greedy: false + seed: 4242 + diff --git a/configs/gptiny_artofwar_char_sealed_seed1337.yaml b/configs/gptiny_artofwar_char_sealed_seed1337.yaml new file mode 100644 index 0000000..74af617 --- /dev/null +++ b/configs/gptiny_artofwar_char_sealed_seed1337.yaml @@ -0,0 +1,38 @@ +data: + input_path: data/raw/art_of_war_body.txt + prepared_path: data/processed/art_of_war_corpus.txt + manifest_path: data/processed/art_of_war_corpus_sealed_manifest.json + tokenizer_path: data/processed/art_of_war_tokenizer_char_sealed.json + tokenizer_type: char + block_size: 64 + train_split: 0.8 + validation_split: 0.1 + +model: + vocab_size: 256 + block_size: 64 + n_layer: 4 + n_head: 4 + n_embd: 128 + dropout: 0.1 + +train: + run_name: gptiny_artofwar_char_sealed_seed1337 + runs_dir: runs + batch_size: 16 + max_steps: 5000 + learning_rate: 0.001 + weight_decay: 0.0 + log_interval: 100 + eval_interval: 250 + eval_batches: null + early_stopping_patience: 3 + early_stopping_min_delta: 0.0 + sample_prompt: "The art" + sample_max_new_tokens: 100 + sample_temperature: 1.0 + sample_top_k: null + sample_seed: 1337 + sample_greedy: false + seed: 1337 + diff --git a/configs/gptiny_artofwar_char_sealed_seed2027.yaml b/configs/gptiny_artofwar_char_sealed_seed2027.yaml new file mode 100644 index 0000000..865c168 --- /dev/null +++ b/configs/gptiny_artofwar_char_sealed_seed2027.yaml @@ -0,0 +1,38 @@ +data: + input_path: data/raw/art_of_war_body.txt + prepared_path: data/processed/art_of_war_corpus.txt + manifest_path: data/processed/art_of_war_corpus_sealed_manifest.json + tokenizer_path: data/processed/art_of_war_tokenizer_char_sealed.json + tokenizer_type: char + block_size: 64 + train_split: 0.8 + validation_split: 0.1 + +model: + vocab_size: 256 + block_size: 64 + n_layer: 4 + n_head: 4 + n_embd: 128 + dropout: 0.1 + +train: + run_name: gptiny_artofwar_char_sealed_seed2027 + runs_dir: runs + batch_size: 16 + max_steps: 5000 + learning_rate: 0.001 + weight_decay: 0.0 + log_interval: 100 + eval_interval: 250 + eval_batches: null + early_stopping_patience: 3 + early_stopping_min_delta: 0.0 + sample_prompt: "The art" + sample_max_new_tokens: 100 + sample_temperature: 1.0 + sample_top_k: null + sample_seed: 2027 + sample_greedy: false + seed: 2027 + diff --git a/configs/gptiny_artofwar_char_sealed_seed4242.yaml b/configs/gptiny_artofwar_char_sealed_seed4242.yaml new file mode 100644 index 0000000..508d12c --- /dev/null +++ b/configs/gptiny_artofwar_char_sealed_seed4242.yaml @@ -0,0 +1,38 @@ +data: + input_path: data/raw/art_of_war_body.txt + prepared_path: data/processed/art_of_war_corpus.txt + manifest_path: data/processed/art_of_war_corpus_sealed_manifest.json + tokenizer_path: data/processed/art_of_war_tokenizer_char_sealed.json + tokenizer_type: char + block_size: 64 + train_split: 0.8 + validation_split: 0.1 + +model: + vocab_size: 256 + block_size: 64 + n_layer: 4 + n_head: 4 + n_embd: 128 + dropout: 0.1 + +train: + run_name: gptiny_artofwar_char_sealed_seed4242 + runs_dir: runs + batch_size: 16 + max_steps: 5000 + learning_rate: 0.001 + weight_decay: 0.0 + log_interval: 100 + eval_interval: 250 + eval_batches: null + early_stopping_patience: 3 + early_stopping_min_delta: 0.0 + sample_prompt: "The art" + sample_max_new_tokens: 100 + sample_temperature: 1.0 + sample_top_k: null + sample_seed: 4242 + sample_greedy: false + seed: 4242 + diff --git a/configs/gptiny_lincoln_bytebpe512_sealed_seed1337.yaml b/configs/gptiny_lincoln_bytebpe512_sealed_seed1337.yaml new file mode 100644 index 0000000..4e59f72 --- /dev/null +++ b/configs/gptiny_lincoln_bytebpe512_sealed_seed1337.yaml @@ -0,0 +1,40 @@ +data: + input_path: data/raw/lincoln_speeches_body.txt + prepared_path: data/processed/lincoln_speeches_corpus.txt + manifest_path: data/processed/lincoln_speeches_corpus_sealed_manifest.json + tokenizer_path: data/processed/lincoln_speeches_tokenizer_bytebpe512_sealed.json + tokenizer_type: byte_bpe + bpe_vocab_size: 512 + bpe_min_frequency: 2 + block_size: 37 + train_split: 0.8 + validation_split: 0.1 + +model: + vocab_size: 512 + block_size: 37 + n_layer: 4 + n_head: 4 + n_embd: 128 + dropout: 0.1 + +train: + run_name: gptiny_lincoln_bytebpe512_sealed_seed1337 + runs_dir: runs + batch_size: 27 + max_steps: 5000 + learning_rate: 0.001 + weight_decay: 0.0 + log_interval: 100 + eval_interval: 250 + eval_batches: null + early_stopping_patience: 3 + early_stopping_min_delta: 0.0 + sample_prompt: "Fellow" + sample_max_new_tokens: 100 + sample_temperature: 1.0 + sample_top_k: null + sample_seed: 1337 + sample_greedy: false + seed: 1337 + diff --git a/configs/gptiny_lincoln_bytebpe512_sealed_seed2027.yaml b/configs/gptiny_lincoln_bytebpe512_sealed_seed2027.yaml new file mode 100644 index 0000000..3c20c5f --- /dev/null +++ b/configs/gptiny_lincoln_bytebpe512_sealed_seed2027.yaml @@ -0,0 +1,40 @@ +data: + input_path: data/raw/lincoln_speeches_body.txt + prepared_path: data/processed/lincoln_speeches_corpus.txt + manifest_path: data/processed/lincoln_speeches_corpus_sealed_manifest.json + tokenizer_path: data/processed/lincoln_speeches_tokenizer_bytebpe512_sealed.json + tokenizer_type: byte_bpe + bpe_vocab_size: 512 + bpe_min_frequency: 2 + block_size: 37 + train_split: 0.8 + validation_split: 0.1 + +model: + vocab_size: 512 + block_size: 37 + n_layer: 4 + n_head: 4 + n_embd: 128 + dropout: 0.1 + +train: + run_name: gptiny_lincoln_bytebpe512_sealed_seed2027 + runs_dir: runs + batch_size: 27 + max_steps: 5000 + learning_rate: 0.001 + weight_decay: 0.0 + log_interval: 100 + eval_interval: 250 + eval_batches: null + early_stopping_patience: 3 + early_stopping_min_delta: 0.0 + sample_prompt: "Fellow" + sample_max_new_tokens: 100 + sample_temperature: 1.0 + sample_top_k: null + sample_seed: 2027 + sample_greedy: false + seed: 2027 + diff --git a/configs/gptiny_lincoln_bytebpe512_sealed_seed4242.yaml b/configs/gptiny_lincoln_bytebpe512_sealed_seed4242.yaml new file mode 100644 index 0000000..3b2285f --- /dev/null +++ b/configs/gptiny_lincoln_bytebpe512_sealed_seed4242.yaml @@ -0,0 +1,40 @@ +data: + input_path: data/raw/lincoln_speeches_body.txt + prepared_path: data/processed/lincoln_speeches_corpus.txt + manifest_path: data/processed/lincoln_speeches_corpus_sealed_manifest.json + tokenizer_path: data/processed/lincoln_speeches_tokenizer_bytebpe512_sealed.json + tokenizer_type: byte_bpe + bpe_vocab_size: 512 + bpe_min_frequency: 2 + block_size: 37 + train_split: 0.8 + validation_split: 0.1 + +model: + vocab_size: 512 + block_size: 37 + n_layer: 4 + n_head: 4 + n_embd: 128 + dropout: 0.1 + +train: + run_name: gptiny_lincoln_bytebpe512_sealed_seed4242 + runs_dir: runs + batch_size: 27 + max_steps: 5000 + learning_rate: 0.001 + weight_decay: 0.0 + log_interval: 100 + eval_interval: 250 + eval_batches: null + early_stopping_patience: 3 + early_stopping_min_delta: 0.0 + sample_prompt: "Fellow" + sample_max_new_tokens: 100 + sample_temperature: 1.0 + sample_top_k: null + sample_seed: 4242 + sample_greedy: false + seed: 4242 + diff --git a/configs/gptiny_lincoln_char_sealed_seed1337.yaml b/configs/gptiny_lincoln_char_sealed_seed1337.yaml new file mode 100644 index 0000000..b25ade3 --- /dev/null +++ b/configs/gptiny_lincoln_char_sealed_seed1337.yaml @@ -0,0 +1,38 @@ +data: + input_path: data/raw/lincoln_speeches_body.txt + prepared_path: data/processed/lincoln_speeches_corpus.txt + manifest_path: data/processed/lincoln_speeches_corpus_sealed_manifest.json + tokenizer_path: data/processed/lincoln_speeches_tokenizer_char_sealed.json + tokenizer_type: char + block_size: 64 + train_split: 0.8 + validation_split: 0.1 + +model: + vocab_size: 256 + block_size: 64 + n_layer: 4 + n_head: 4 + n_embd: 128 + dropout: 0.1 + +train: + run_name: gptiny_lincoln_char_sealed_seed1337 + runs_dir: runs + batch_size: 16 + max_steps: 5000 + learning_rate: 0.001 + weight_decay: 0.0 + log_interval: 100 + eval_interval: 250 + eval_batches: null + early_stopping_patience: 3 + early_stopping_min_delta: 0.0 + sample_prompt: "Fellow" + sample_max_new_tokens: 100 + sample_temperature: 1.0 + sample_top_k: null + sample_seed: 1337 + sample_greedy: false + seed: 1337 + diff --git a/configs/gptiny_lincoln_char_sealed_seed2027.yaml b/configs/gptiny_lincoln_char_sealed_seed2027.yaml new file mode 100644 index 0000000..039698c --- /dev/null +++ b/configs/gptiny_lincoln_char_sealed_seed2027.yaml @@ -0,0 +1,38 @@ +data: + input_path: data/raw/lincoln_speeches_body.txt + prepared_path: data/processed/lincoln_speeches_corpus.txt + manifest_path: data/processed/lincoln_speeches_corpus_sealed_manifest.json + tokenizer_path: data/processed/lincoln_speeches_tokenizer_char_sealed.json + tokenizer_type: char + block_size: 64 + train_split: 0.8 + validation_split: 0.1 + +model: + vocab_size: 256 + block_size: 64 + n_layer: 4 + n_head: 4 + n_embd: 128 + dropout: 0.1 + +train: + run_name: gptiny_lincoln_char_sealed_seed2027 + runs_dir: runs + batch_size: 16 + max_steps: 5000 + learning_rate: 0.001 + weight_decay: 0.0 + log_interval: 100 + eval_interval: 250 + eval_batches: null + early_stopping_patience: 3 + early_stopping_min_delta: 0.0 + sample_prompt: "Fellow" + sample_max_new_tokens: 100 + sample_temperature: 1.0 + sample_top_k: null + sample_seed: 2027 + sample_greedy: false + seed: 2027 + diff --git a/configs/gptiny_lincoln_char_sealed_seed4242.yaml b/configs/gptiny_lincoln_char_sealed_seed4242.yaml new file mode 100644 index 0000000..e40beed --- /dev/null +++ b/configs/gptiny_lincoln_char_sealed_seed4242.yaml @@ -0,0 +1,38 @@ +data: + input_path: data/raw/lincoln_speeches_body.txt + prepared_path: data/processed/lincoln_speeches_corpus.txt + manifest_path: data/processed/lincoln_speeches_corpus_sealed_manifest.json + tokenizer_path: data/processed/lincoln_speeches_tokenizer_char_sealed.json + tokenizer_type: char + block_size: 64 + train_split: 0.8 + validation_split: 0.1 + +model: + vocab_size: 256 + block_size: 64 + n_layer: 4 + n_head: 4 + n_embd: 128 + dropout: 0.1 + +train: + run_name: gptiny_lincoln_char_sealed_seed4242 + runs_dir: runs + batch_size: 16 + max_steps: 5000 + learning_rate: 0.001 + weight_decay: 0.0 + log_interval: 100 + eval_interval: 250 + eval_batches: null + early_stopping_patience: 3 + early_stopping_min_delta: 0.0 + sample_prompt: "Fellow" + sample_max_new_tokens: 100 + sample_temperature: 1.0 + sample_top_k: null + sample_seed: 4242 + sample_greedy: false + seed: 4242 + diff --git a/docs/architecture.md b/docs/architecture.md index dc3ab77..c62e10a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -24,6 +24,7 @@ flowchart TD J --> O[One-shot sealed test evaluator] B -. terminal split .-> O O --> P[test_evaluation_best.json] + P --> Q[Sealed matrix validator + paired contrasts] I --> L[Generation + diagnostics] J --> L K --> M[Experiment reports] @@ -90,6 +91,10 @@ Training does not encode the terminal test text. After checkpoint selection, the verifies corpus and checkpoint identity, evaluates full coverage once, and refuses to overwrite its result artifact. +For balanced panels, `sealed_matrix.py` reads only bounded run artifacts and refuses incomplete, +duplicated, incomparable, identity-invalid, or partially evaluated cells before computing paired +tokenizer effects and corpus interactions. The CLI remains a thin adapter. + ## Run Artifacts Each training run writes: diff --git a/docs/codex/architecture.md b/docs/codex/architecture.md index 0238774..c5f0a61 100644 --- a/docs/codex/architecture.md +++ b/docs/codex/architecture.md @@ -61,6 +61,8 @@ config/checkpoints, call package code, and print results. Do not put reusable domain logic in scripts when it belongs under `src/smallm/`. `scripts/evaluate_test.py` is the post-selection edge: it verifies run-local provenance and writes the one-shot sealed-test artifact without exposing test data to training orchestration. +`scripts/summarize_test_matrix.py` is the panel-analysis edge: reusable completeness, identity, +comparability, and paired-contrast logic remains in `smallm.evaluation.sealed_matrix`. ## Module Boundaries @@ -84,6 +86,7 @@ the one-shot sealed-test artifact without exposing test data to training orchest - baseline probability models - validation loss and perplexity calculations - exact full-coverage sealed-test evaluation results +- complete sealed-test matrix validation and descriptive paired effects `training/` owns: diff --git a/docs/codex/build-and-test.md b/docs/codex/build-and-test.md index 027743b..ff8317d 100644 --- a/docs/codex/build-and-test.md +++ b/docs/codex/build-and-test.md @@ -50,7 +50,7 @@ The individual commands remain canonical and are listed below. python -m pytest ``` -Current expected result after milestone 026+: at least 163 tests passing with at least 90% coverage. +Current expected result after milestone 028: at least 166 tests passing with at least 90% coverage. ### Compile Check @@ -146,6 +146,20 @@ python scripts/evaluate_test.py --run runs// --checkpoint-kind best Do not rerun, delete, or use the resulting test artifact for model selection. +For a preregistered balanced panel, aggregate only after every one-shot artifact exists: + +```bash +python scripts/summarize_test_matrix.py \ + --reference-tokenizer char --candidate-tokenizer byte_bpe \ + --cell artofwar char \ + --cell artofwar byte_bpe \ + --cell lincoln char \ + --cell lincoln byte_bpe +``` + +The command rejects missing or duplicate cells, incomparable configurations, corpus/checkpoint +identity failures, non-best checkpoints, and partial evaluation coverage. + ## Validation Matrix | Change type | Minimum validation | @@ -176,9 +190,7 @@ current task or explicitly quoted from a prior report. ## Current Milestone Status -Milestone 027 uses the milestone-026 evaluation contract: legacy two-way splits remain compatible, -while explicit validation fractions reserve a chronological test region that training leaves -unencoded and unscored. -Best-checkpoint test evaluation verifies corpus/checkpoint identity, requires full coverage, and -refuses artifact overwrite. The preregistered Hamlet replication confirms the workflow on an -external dramatic-play distribution. +Milestone 028 extends the sealed-test contract to a complete preregistered 2 × 2 × 3 factorial +panel. The matrix aggregator validates schema, coverage, identities, cell comparability, and exact +declared membership before computing paired effects and interactions. Art of War and Lincoln test +segments are consumed alongside the earlier Alice, Peter Pan, and Hamlet segments. diff --git a/docs/codex/experiments.md b/docs/codex/experiments.md index a0a3dab..dcbb4fd 100644 --- a/docs/codex/experiments.md +++ b/docs/codex/experiments.md @@ -180,3 +180,9 @@ Milestone 027 was preregistered in commit `6f130d5` before Hamlet corpus access. seed-1337 sealed protocol, ByteBPE512 beats character by `0.0673` test BPC on the dramatic-play distribution. Hamlet's terminal segment is easier than validation for both models, so gap direction and effect magnitude remain corpus-dependent. The Hamlet test is consumed. + +Milestone 028 was preregistered in commit `5ee2274` before Art of War or Lincoln corpus access. +Across three seeds per tokenizer and corpus, ByteBPE512 wins all six sealed-test pairs: mean paired +candidate-minus-character BPC is `-0.1134` on Art of War and `-0.1543` on Lincoln. Every new test +gap is positive, and the mean corpus interaction is `-0.0409`; report a robust direction with +corpus-dependent magnitude. Both terminal segments are consumed. diff --git a/docs/experiments.md b/docs/experiments.md index 6a9101c..43b4ddb 100644 --- a/docs/experiments.md +++ b/docs/experiments.md @@ -34,6 +34,7 @@ not a replacement for the original reports. | [025 Corpus-by-Seed Matrix](../experiments/025-corpus-by-seed-matrix.md) | Factorial robustness | ByteBPE512 wins all six paired comparisons; mean advantage is `0.0619` BPC on Alice and `0.0252` on Peter Pan. | | [026 Sealed Test Evaluation](../experiments/026-sealed-test-evaluation.md) | Confirmatory evaluation | On untouched terminal segments, ByteBPE512 beats character by `0.0614` BPC on Alice and `0.0258` on Peter Pan. | | [027 Hamlet External-Distribution Replication](../experiments/027-hamlet-external-distribution.md) | Preregistered external validity | ByteBPE512 beats character by `0.0673` sealed-test BPC on a dramatic play; the terminal region is easier than validation for both models. | +| [028 Preregistered External Corpus Panel](../experiments/028-preregistered-external-corpus-panel.md) | Multi-corpus confirmatory panel | ByteBPE512 wins all six same-seed sealed-test pairs, averaging `-0.1134` BPC on Art of War and `-0.1543` on Lincoln. | ## Topic Shortcuts @@ -62,7 +63,8 @@ not a replacement for the original reports. [024](../experiments/024-cross-corpus-robustness.md), [025](../experiments/025-corpus-by-seed-matrix.md), [026](../experiments/026-sealed-test-evaluation.md), - [027](../experiments/027-hamlet-external-distribution.md). + [027](../experiments/027-hamlet-external-distribution.md), + [028](../experiments/028-preregistered-external-corpus-panel.md). ## Current Status @@ -112,3 +114,8 @@ Milestone 027 preregisters Hamlet before corpus access and transfers the same fr dramatic play. ByteBPE512 reaches sealed-test BPC `2.2546` versus character's `2.3219`, an advantage of `0.0673`. Both terminal results are better than validation, reversing milestone 026's gap direction and reinforcing that chronological difficulty is corpus-dependent. + +Milestone 028 expands that confirmatory design to two new corpora and three seeds. ByteBPE512 wins +all six same-seed test comparisons, with mean paired effects `-0.1134 ± 0.0041` BPC on Art of War +and `-0.1543 ± 0.0130` on Lincoln. Every terminal segment is harder than validation, and the +`-0.0409` mean corpus interaction again rejects a universal effect-size interpretation. diff --git a/docs/training.md b/docs/training.md index 7b5bb2a..f9e666e 100644 --- a/docs/training.md +++ b/docs/training.md @@ -50,6 +50,7 @@ compileall, and Markdown links. `make smoke` writes an ignored smoke run, and | `configs/gptiny_peterpan_*_earlystop_seed*.yaml` | Additional Peter Pan matrix seeds. | | `configs/gptiny_{alice,peterpan}_{char,bytebpe512}_sealed.yaml` | Frozen 80/10/10 confirmatory runs. | | `configs/gptiny_hamlet_{char,bytebpe512}_sealed.yaml` | Preregistered 80/10/10 dramatic-play replication. | +| `configs/gptiny_{artofwar,lincoln}_{char,bytebpe512}_sealed_seed{1337,2027,4242}.yaml` | Preregistered 2 × 2 × 3 external-corpus panel. | ## Corpus Preparation @@ -236,6 +237,11 @@ Experiment 027 preregisters a structurally different Hamlet distribution before ByteBPE512 beats character by `0.0673` sealed-test BPC. Both terminal results are better than validation, demonstrating that chronological gap direction is not stable across corpora. +Experiment 028 preregisters two additional external corpora and three seeds. ByteBPE512 beats the +character control in all six paired sealed-test comparisons, averaging `0.1134` BPC better on Art +of War and `0.1543` better on Lincoln. Use `scripts/summarize_test_matrix.py` for complete balanced +panels; it refuses missing cells, identity mismatches, non-best checkpoints, and partial coverage. + ## Artifact Policy `data/raw/`, `data/processed/`, `checkpoints/`, and `runs/` are local runtime diff --git a/experiments/028-preregistered-external-corpus-panel.md b/experiments/028-preregistered-external-corpus-panel.md new file mode 100644 index 0000000..965d07f --- /dev/null +++ b/experiments/028-preregistered-external-corpus-panel.md @@ -0,0 +1,240 @@ +# 028 — Preregistered External Corpus Panel + +## Preregistration + +This section and all 12 referenced configurations are committed before downloading, inspecting, +preparing, tokenizing, training on, or evaluating either new corpus. Later observations will be +reported separately without changing this declaration. + +### Question And Hypotheses + +Does the frozen ByteBPE512 decision retain its direction across multiple seeds on two new corpus +families: a translated military treatise and a collection of political speeches and letters? + +For corpus \(c\) and seed \(s\), the paired contrast is + +\[ +\Delta_{c,s} = +\operatorname{BPC}_{\text{ByteBPE512,test},c,s}- +\operatorname{BPC}_{\text{character,test},c,s}. +\] + +The primary directional hypothesis is that the mean paired contrast is negative for each corpus. +A stronger robustness outcome requires all six paired contrasts to be negative. Ties, positive +means, positive individual pairs, and corpus interactions will be reported rather than excluded. +Population standard deviation describes only the complete three-seed panel; it is not a confidence +interval or population-level hypothesis test. + +### Corpus Contract + +1. **Nonfiction:** *The Art of War* by Sun Tzu, translated by Lionel Giles, Project Gutenberg + ebook #132, fetched from + `https://www.gutenberg.org/cache/epub/132/pg132.txt`. +2. **Speeches and letters:** *Lincoln's Inaugurals, Addresses and Letters (Selections)*, edited by + Daniel Kilham Dodge, Project Gutenberg ebook #14721, fetched from + `https://www.gutenberg.org/cache/epub/14721/pg14721.txt`. + +For each source, the complete content between the unique ordered Gutenberg START/END markers will +be extracted without character-budget truncation, then normalized only by the existing +`prepare_corpus.py` rules. Each prepared corpus receives a chronological 80% train, 10% +validation, and 10% sealed-test split. Downloads, extracted bodies, prepared text, tokenizers, +checkpoints, metrics, and runs remain ignored; exact hashes and counts will be reported. + +A source may be replaced only if its declared URL cannot be fetched, its marker contract fails, or +the marker title contradicts the declared work. Any replacement must be committed before training. +Corpus size or content style observed after a valid fetch is not grounds for substitution. + +### Frozen Factorial Design + +The panel crosses: + +- corpora: `artofwar`, `lincoln`; +- tokenizers: character, boundary-aware lossless ByteBPE512; +- seeds: 1337, 2027, 4242. + +All 12 models use 4 layers, 4 heads, width 128, dropout 0.1, AdamW at `1e-3`, zero +weight decay, full validation every 250 steps, patience 3, and a 5,000-step ceiling. Character uses +context 64 and batch 16. ByteBPE512 uses context 37, batch 27, vocabulary target 512, and minimum +merge frequency 2. No setting will change after corpus access. + +The committed configurations follow: + +```text +configs/gptiny_{artofwar,lincoln}_{char,bytebpe512}_sealed_seed{1337,2027,4242}.yaml +``` + +### Access, Selection, And Analysis Rules + +Tokenizer fitting uses train text only. Gradients use train tokens only. Validation selects +`best_checkpoint.pt` independently within each run and drives the fixed early-stopping rule. +Every one of the 12 training jobs must complete before either sealed test region is tokenized or +scored. Then each best checkpoint is evaluated exactly once with full target-token coverage. + +The report will include corpus hashes and counts, vocabularies, compression, baselines, parameter +counts, all validation and test results, stopping steps, checkpoint hashes, test support, paired +same-seed contrasts, per-corpus means and population standard deviations, corpus interaction by +seed, and validation-to-test gaps. No test result may trigger configuration changes, replacement +runs, seed selection, or additional use of these test segments. + +## Observations + +Before training, source verification found that ebook #14721's marker and front matter identify +*Speeches & Letters of Abraham Lincoln, 1832–1865*, edited by Merwin Roe, rather than the Lincoln +collection title initially declared above. This provenance correction was committed before +tokenizer fitting, baselines, training, or test access. The URL, ebook identifier, complete-body +extraction, factorial design, hypotheses, and analysis rules are unchanged. + +### Audit Trail + +The preregistration and 12 configs were committed as `5ee2274` at +`2026-07-13T02:15:26-05:00`, before either URL was accessed. The Lincoln provenance amendment was +committed as `f1bd534` at `2026-07-13T02:16:56-05:00`, before tokenizer fitting, baselines, or +training. The sealed-matrix loader and analysis were committed as `fb5d59c` before test access. + +All 12 training jobs completed before either test region was tokenized. A global pre-access audit +found exactly 12 balanced schema-v2 runs, all with `test_status: sealed_unread`, no test metric +fields, and no existing `test_evaluation_best.json`. Each best checkpoint was then evaluated once; +the evaluator's no-overwrite rule now prevents accidental reuse of these consumed segments. + +After test access, the matrix loader correctly rejected the first aggregation attempt because its +training-comparison fingerprint included `sample_seed`. The preregistered configs intentionally +pair that generation-only seed with the training seed. Commit `9d937ee` removed all sample-only +controls from the likelihood-comparison fingerprint and added a regression test. It changed no +config, model, checkpoint, evaluation artifact, or numerical result. + +### Corpus And Tokenizer Provenance + +| Corpus | Download bytes / SHA-256 | Extracted characters / SHA-256 | Prepared characters / SHA-256 | Train / validation / test characters | +| --- | --- | --- | --- | --- | +| Art of War | 342,105 / `c478f7e05a07bf190585fcea0bc4658ce22e47949ac18444eca53fb274878ffe` | 312,842 / `14b1ca4cf78693eb4f3ba24dad7143337bb76f6d041d6e57f2569908f12fe654` | 312,378 / `560195e8b717e83c4bd3a53949b71bac6962731c9ef940ed38301804c3545262` | 249,902 / 31,238 / 31,238 | +| Lincoln | 536,309 / `7a3ef6688c53c55fbc4c37fb7f283f17c48d21c3f3e5de7b9034fd28573923f2` | 507,454 / `1fef9ffbffe7f701caa1aeef2b6e689f0a119a96cd6f522c0ef13d231566a8a0` | 507,076 / `3b76831dfd33c8a738d4f68064c209e7278302541733bbfb6354fa998710e1f5` | 405,660 / 50,708 / 50,708 | + +Character vocabularies contain the train vocabulary plus ``: 97 entries for Art of War and +86 for Lincoln. ByteBPE512 reaches its declared 512 entries on both corpora. On train text it +compresses Art of War from 249,902 characters to 149,379 tokens (`1.6730` characters/token) and +Lincoln from 405,660 to 234,775 (`1.7279` characters/token). The character models have 826,465 and +823,638 parameters respectively; both ByteBPE512 models have 929,664. This vocabulary-dependent +parameter difference is part of the frozen design, not evidence of a parameter-matched comparison. + +Validation token-loss baselines were: + +| Corpus | Tokenizer | Uniform | Unigram | Add-one bigram | +| --- | --- | ---: | ---: | ---: | +| Art of War | character | 4.5747 | 3.1695 | 2.5016 | +| Art of War | ByteBPE512 | 6.2383 | 4.4276 | 3.5525 | +| Lincoln | character | 4.4543 | 3.0733 | 2.4529 | +| Lincoln | ByteBPE512 | 6.2383 | 4.3502 | 3.4119 | + +These token losses are within-tokenizer references only; their units differ across tokenizers. + +### Validation Selection And Sealed Tests + +| Corpus | Tokenizer | Seed | Actual / best step | Best validation BPC | Test BPC | Test minus validation | +| --- | --- | ---: | ---: | ---: | ---: | ---: | +| Art of War | character | 1337 | 5000 / 5000 | 2.154108 | 2.181844 | +0.027735 | +| Art of War | character | 2027 | 5000 / 5000 | 2.158726 | 2.177676 | +0.018950 | +| Art of War | character | 4242 | 5000 / 5000 | 2.164572 | 2.181226 | +0.016654 | +| Art of War | ByteBPE512 | 1337 | 5000 / 4250 | 2.018544 | 2.074171 | +0.055627 | +| Art of War | ByteBPE512 | 2027 | 5000 / 4500 | 2.018501 | 2.061350 | +0.042849 | +| Art of War | ByteBPE512 | 4242 | 4750 / 4000 | 2.020842 | 2.064947 | +0.044105 | +| Lincoln | character | 1337 | 5000 / 5000 | 2.139197 | 2.254961 | +0.115764 | +| Lincoln | character | 2027 | 5000 / 5000 | 2.110965 | 2.220802 | +0.109837 | +| Lincoln | character | 4242 | 5000 / 5000 | 2.114325 | 2.231029 | +0.116703 | +| Lincoln | ByteBPE512 | 1337 | 5000 / 5000 | 1.959578 | 2.082262 | +0.122684 | +| Lincoln | ByteBPE512 | 2027 | 5000 / 5000 | 1.971808 | 2.075354 | +0.103546 | +| Lincoln | ByteBPE512 | 4242 | 5000 / 4500 | 1.969978 | 2.086232 | +0.116254 | + +Every test gap is positive. This is consistent with harder chronological tails, but does not by +itself distinguish distribution shift from checkpoint-selection optimism. Mean gaps are `+0.0211` +character and `+0.0475` ByteBPE512 on Art of War, and approximately `+0.1141` for both tokenizers +on Lincoln. + +Full-coverage test support is 31,238 character tokens versus 18,644 ByteBPE tokens for Art of War, +and 50,708 versus 30,032 for Lincoln. The normalized likelihood denominators are 31,237 and 50,706 +target characters for ByteBPE512; the one-character differences from the character denominators +come from the first token's span, and are handled by the exact coverage contract. + +### Paired Confirmatory Result + +| Corpus | Seed 1337 | Seed 2027 | Seed 4242 | Mean ± population SD | Range | +| --- | ---: | ---: | ---: | ---: | ---: | +| Art of War | -0.107673 | -0.116326 | -0.116279 | -0.113426 ± 0.004068 | -0.116326–-0.107673 | +| Lincoln | -0.172699 | -0.145448 | -0.144797 | -0.154315 ± 0.013002 | -0.172699–-0.144797 | + +The primary hypothesis is supported for both corpora, and the stronger criterion is also met: all +six preregistered same-seed contrasts are negative. Mean cell results are `2.066823 ± 0.005399` +ByteBPE512 versus `2.180249 ± 0.001837` character on Art of War, and +`2.081283 ± 0.004494` versus `2.235597 ± 0.014314` on Lincoln. + +The Lincoln-minus-Art-of-War interaction is negative at every seed (`-0.065025`, `-0.029122`, and +`-0.028518`), with mean `-0.040889 ± 0.017069` BPC. Direction therefore transfers, but magnitude +does not: ByteBPE512's advantage is larger on this speeches-and-letters collection than on the +translated treatise. This is descriptive heterogeneity, not a population-level interaction test. + +### Checkpoint Identity + +The evaluated best-checkpoint SHA-256 prefixes, in seed order 1337 / 2027 / 4242, are: + +- Art of War character: `c16dbc44365a`, `299836416066`, `fb90f8765d80`. +- Art of War ByteBPE512: `962bbf242de2`, `2ce7024cff0f`, `dded2e5d8c09`. +- Lincoln character: `68fcd2a41f9e`, `ee8e958fdb3a`, `50444690c309`. +- Lincoln ByteBPE512: `354c7c134442`, `cb37e53e0dc1`, `c693f18fc458`. + +The complete hashes remain in each ignored `test_evaluation_best.json`; the report uses prefixes +for readability while corpus/checkpoint identity checks used all 64 hexadecimal characters. + +### Commands And Validation + +The protocol used the repository-native commands below for each corpus/config/run. Angle-bracketed +arguments vary only by the committed config or the hashed local artifact described above: + +```bash +python scripts/prepare_corpus.py --input --output \ + --stats --manifest --train-split 0.8 \ + --validation-split 0.1 --source-name --source-note +python scripts/prepare_data.py --config configs/.yaml +python scripts/evaluate_baselines.py --config configs/.yaml +python scripts/train.py --config configs/.yaml +python scripts/evaluate_test.py --run runs// --checkpoint-kind best +``` + +The exact final matrix command was: + +```bash +python scripts/summarize_test_matrix.py \ + --reference-tokenizer char --candidate-tokenizer byte_bpe \ + --cell artofwar char runs/gptiny_artofwar_char_sealed_seed1337/2026-07-13_02-25-10 \ + --cell artofwar char runs/gptiny_artofwar_char_sealed_seed2027/2026-07-13_02-33-27 \ + --cell artofwar char runs/gptiny_artofwar_char_sealed_seed4242/2026-07-13_02-41-51 \ + --cell artofwar byte_bpe runs/gptiny_artofwar_bytebpe512_sealed_seed1337/2026-07-13_02-51-59 \ + --cell artofwar byte_bpe runs/gptiny_artofwar_bytebpe512_sealed_seed2027/2026-07-13_03-00-35 \ + --cell artofwar byte_bpe runs/gptiny_artofwar_bytebpe512_sealed_seed4242/2026-07-13_03-09-17 \ + --cell lincoln char runs/gptiny_lincoln_char_sealed_seed1337/2026-07-13_03-17-06 \ + --cell lincoln char runs/gptiny_lincoln_char_sealed_seed2027/2026-07-13_03-27-18 \ + --cell lincoln char runs/gptiny_lincoln_char_sealed_seed4242/2026-07-13_03-36-41 \ + --cell lincoln byte_bpe runs/gptiny_lincoln_bytebpe512_sealed_seed1337/2026-07-13_03-47-12 \ + --cell lincoln byte_bpe runs/gptiny_lincoln_bytebpe512_sealed_seed2027/2026-07-13_03-55-25 \ + --cell lincoln byte_bpe runs/gptiny_lincoln_bytebpe512_sealed_seed4242/2026-07-13_04-04-20 +``` + +The aggregator verifies the complete declared matrix, cell comparability, corpus and checkpoint +identity, best-checkpoint selection, full coverage, balanced seeds, and unique run paths before +reporting any contrast. Focused Ruff, mypy, and 24 evaluation tests passed after the fingerprint +repair. The final frozen quality gate passed Ruff formatting/lint, mypy over 44 source files, +166 tests at 90.74% coverage, compileall, and Markdown link validation. `pip-audit` found no known +vulnerabilities; the local `smallm` package is correctly skipped because it is not a PyPI release. + +### Conclusion And Limits + +Across two preregistered external corpora and three preregistered seeds, ByteBPE512 robustly beats +the character control on one-shot sealed-test BPC. Together with milestones 025–027, this makes the +direction a repeated empirical result in this small lab rather than a one-corpus or one-seed +accident. It does not establish universality, a causal mechanism, or a production tokenizer claim. + +The panel has only two new English public-domain sources; both are single chronological documents, +their sizes differ, Gutenberg bodies include editorial material, the Lincoln metadata required a +transparent correction, and vocabulary-dependent embeddings give ByteBPE512 roughly 103k–106k +more parameters. Corpora are fixed rather than sampled, population standard deviations are purely +descriptive, and both new terminal regions are now consumed. The cleanest next modeling question +is a preregistered parameter-matched architecture ablation on new sealed data, not further tuning +against any test segment reported here. diff --git a/notes/06-reproducibility.md b/notes/06-reproducibility.md index 739eee8..c67e6f5 100644 --- a/notes/06-reproducibility.md +++ b/notes/06-reproducibility.md @@ -212,3 +212,34 @@ milestone 027 finds \(G_c<0\) for both Hamlet models. Textual position can alter chapter or scene structure, punctuation, verse density, and intrinsic entropy. A professional interpretation therefore reports both the model contrast and the regional difficulty shift rather than labeling every positive gap "overfitting" or every negative gap "improved generalization." + +### Balanced confirmatory panels and analysis contracts + +A multi-corpus confirmatory panel adds two forms of blocking. Same-seed pairing within corpus +removes the seed component shared by candidate and control, while repeating the pair across corpus +families exposes tokenizer-by-corpus heterogeneity. For corpora (c=1,\ldots,C), tokenizers +(t\in\{A,B\}), and fixed seeds (s=1,\ldots,S), completeness is part of the estimand: + +\[ +\mathcal{D}=\{y_{tcs}:t\in\{A,B\},c\in\{1,\ldots,C\},s\in\{1,\ldots,S\}\}. +\] + +Silently dropping a failed, inconvenient, or missing cell changes \(\mathcal{D}\) after outcomes +are partially known. A professional aggregator therefore validates the exact Cartesian product, +unique paths, common seed sets, artifact schema, checkpoint kind and hash, corpus hash, and full +coverage before calculating any effect. Validation failure is an outcome of the analysis contract, +not an invitation to weaken it. + +Comparability fingerprints must encode causes of the likelihood result—data, model, optimizer, +training budget, and selection policy—but exclude downstream presentation controls such as sample +prompt, decoding temperature, and generation seed. Including an irrelevant sample seed creates a +false mismatch; excluding a learning rate or split creates a false match. Milestone 028 encountered +the former after test access, repaired it transparently without changing artifacts, and added a +regression test. This illustrates why preregistering the analysis rule is necessary but executable +schema checks are still fallible software that require audit trails. + +Milestone 028's complete 2 × 2 × 3 panel finds all six ByteBPE512-minus-character test contrasts +negative. The result strengthens a directional claim, not a universal effect-size claim: the mean +interaction differs across Art of War and Lincoln, the corpora are fixed rather than sampled, and +three population-SD summaries are descriptive. Every accessed terminal region is now historical +evidence and cannot serve as a clean test set for a parameter-matched follow-up. diff --git a/scripts/summarize_test_matrix.py b/scripts/summarize_test_matrix.py new file mode 100644 index 0000000..ff38eed --- /dev/null +++ b/scripts/summarize_test_matrix.py @@ -0,0 +1,72 @@ +from __future__ import annotations + +import argparse +from pathlib import Path + +from smallm.evaluation.robustness import summarize_values +from smallm.evaluation.sealed_matrix import analyze_sealed_test_matrix + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument( + "--cell", + action="append", + nargs=3, + metavar=("CORPUS", "TOKENIZER", "RUN_DIR"), + required=True, + ) + parser.add_argument("--reference-tokenizer", required=True) + parser.add_argument("--candidate-tokenizer", required=True) + args = parser.parse_args() + entries = [(corpus, tokenizer, Path(path)) for corpus, tokenizer, path in args.cell] + try: + analysis = analyze_sealed_test_matrix( + entries, + reference_tokenizer=args.reference_tokenizer, + candidate_tokenizer=args.candidate_tokenizer, + ) + except (OSError, UnicodeError, TypeError, AttributeError, KeyError, ValueError) as exc: + parser.error(str(exc)) + + print("standard_deviation population_descriptive") + print("contrast candidate_minus_reference_test_bpc") + print() + print("corpus tokenizer seed checkpoint_step test_bpc test_loss test_tokens target_characters") + for (corpus, tokenizer), observations in sorted(analysis.cells.items()): + for item in sorted(observations, key=lambda observation: observation.seed): + print( + f"{corpus} {tokenizer} {item.seed} {item.checkpoint_step} " + f"{item.test_bpc:.6f} {item.test_loss:.6f} {item.test_tokens} " + f"{item.target_characters}" + ) + print() + print("cell corpus tokenizer mean_bpc population_stddev min max") + for (corpus, tokenizer), observations in sorted(analysis.cells.items()): + summary = summarize_values([item.test_bpc for item in observations]) + print( + f"cell {corpus} {tokenizer} {summary.mean:.6f} " + f"{summary.population_stddev:.6f} {summary.minimum:.6f} {summary.maximum:.6f}" + ) + print() + print("paired corpus seed candidate_minus_reference_test_bpc") + for corpus, contrasts in sorted(analysis.contrasts.items()): + for seed, value in contrasts: + print(f"paired {corpus} {seed} {value:.6f}") + summary = summarize_values([value for _, value in contrasts]) + print( + f"contrast_summary {corpus} {summary.mean:.6f} " + f"{summary.population_stddev:.6f} {summary.minimum:.6f} {summary.maximum:.6f}" + ) + print() + for seed, value in analysis.interactions: + print(f"interaction {seed} {value:.6f}") + interaction = summarize_values([value for _, value in analysis.interactions]) + print( + f"interaction_summary {interaction.mean:.6f} {interaction.population_stddev:.6f} " + f"{interaction.minimum:.6f} {interaction.maximum:.6f}" + ) + + +if __name__ == "__main__": + main() diff --git a/src/smallm/evaluation/run_observation.py b/src/smallm/evaluation/run_observation.py index da820fe..8d43273 100644 --- a/src/smallm/evaluation/run_observation.py +++ b/src/smallm/evaluation/run_observation.py @@ -62,7 +62,17 @@ def _required_int(summary: dict[str, Any], field: str) -> int: def _comparison_fingerprint(config: Any, summary: dict[str, Any]) -> str: config_state = asdict(config) train_state = config_state["train"] - for field in ("run_name", "runs_dir", "seed"): + for field in ( + "run_name", + "runs_dir", + "seed", + "sample_prompt", + "sample_max_new_tokens", + "sample_temperature", + "sample_top_k", + "sample_seed", + "sample_greedy", + ): train_state.pop(field) dataset = summary.get("dataset") if not isinstance(dataset, dict) or not isinstance(dataset.get("prepared_sha256"), str): diff --git a/src/smallm/evaluation/sealed_matrix.py b/src/smallm/evaluation/sealed_matrix.py new file mode 100644 index 0000000..0f1c9d8 --- /dev/null +++ b/src/smallm/evaluation/sealed_matrix.py @@ -0,0 +1,190 @@ +from __future__ import annotations + +import json +import re +from collections import defaultdict +from dataclasses import dataclass +from math import isfinite +from pathlib import Path +from typing import Any + +from smallm.evaluation.run_observation import load_run_artifact + +_LABEL = re.compile(r"^[a-z][a-z0-9_-]{0,31}$") +_SHA256 = re.compile(r"^[0-9a-f]{64}$") +_MAX_ARTIFACT_BYTES = 1_000_000 +_MAX_ENTRIES = 64 + + +@dataclass(frozen=True) +class SealedTestObservation: + seed: int + tokenizer: str + prepared_sha256: str + comparison_fingerprint: str + checkpoint_step: int + checkpoint_sha256: str + test_loss: float + test_bpc: float + test_tokens: int + target_tokens: int + target_characters: int + + +@dataclass(frozen=True) +class SealedMatrixAnalysis: + cells: dict[tuple[str, str], list[SealedTestObservation]] + contrasts: dict[str, list[tuple[int, float]]] + interactions: list[tuple[int, float]] + + +def _read_json_object(path: Path) -> dict[str, Any]: + with path.open("rb") as handle: + raw = handle.read(_MAX_ARTIFACT_BYTES + 1) + if len(raw) > _MAX_ARTIFACT_BYTES: + raise ValueError(f"sealed test artifact exceeds {_MAX_ARTIFACT_BYTES} bytes: {path}") + payload = json.loads(raw.decode("utf-8")) + if not isinstance(payload, dict): + raise ValueError(f"sealed test artifact must contain a JSON object: {path}") + return payload + + +def _bounded_int(payload: dict[str, Any], field: str, *, positive: bool = False) -> int: + value = payload.get(field) + if ( + not isinstance(value, int) + or isinstance(value, bool) + or value < int(positive) + or value > 1_000_000_000 + ): + raise ValueError(f"sealed test field {field!r} must be a bounded integer") + return value + + +def _bounded_float(payload: dict[str, Any], field: str, *, positive: bool = False) -> float: + value = payload.get(field) + if not isinstance(value, int | float) or isinstance(value, bool): + raise ValueError(f"sealed test field {field!r} must be numeric") + number = float(value) + if not isfinite(number) or number < 0 or (positive and number == 0) or number > 1e9: + raise ValueError(f"sealed test field {field!r} is outside supported bounds") + return number + + +def load_sealed_test_observation(run_dir: Path) -> SealedTestObservation: + summary, validation = load_run_artifact(run_dir) + path = run_dir / "test_evaluation_best.json" + payload = _read_json_object(path) + if payload.get("schema_version") != 1 or payload.get("status") != "complete": + raise ValueError(f"sealed test artifact is not a complete schema-v1 result: {path}") + if payload.get("checkpoint_kind") != "best": + raise ValueError("sealed matrix requires best-checkpoint test evaluations") + if payload.get("evaluation_mode") != "full" or _bounded_float(payload, "test_coverage") != 1.0: + raise ValueError("sealed matrix requires full test coverage") + + dataset = summary.get("dataset") + prepared_sha256 = dataset.get("prepared_sha256") if isinstance(dataset, dict) else None + if not isinstance(prepared_sha256, str) or not _SHA256.fullmatch(prepared_sha256): + raise ValueError("run summary has an invalid prepared corpus checksum") + if payload.get("prepared_sha256") != prepared_sha256: + raise ValueError("sealed test corpus checksum does not match the run summary") + checkpoint_step = _bounded_int(payload, "checkpoint_step") + if checkpoint_step != validation.best_step: + raise ValueError("sealed test checkpoint step does not match best validation step") + test_characters = _bounded_int(payload, "test_characters", positive=True) + if ( + summary.get("test_status") != "sealed_unread" + or summary.get("test_characters") != test_characters + ): + raise ValueError("sealed test support does not match the run summary") + checkpoint_sha256 = payload.get("checkpoint_sha256") + if not isinstance(checkpoint_sha256, str) or not _SHA256.fullmatch(checkpoint_sha256): + raise ValueError("sealed test checkpoint checksum must be lowercase SHA-256") + test_tokens = _bounded_int(payload, "test_tokens", positive=True) + target_tokens = _bounded_int(payload, "test_target_tokens", positive=True) + total_target_tokens = _bounded_int(payload, "test_total_target_tokens", positive=True) + if target_tokens != total_target_tokens or target_tokens >= test_tokens: + raise ValueError("sealed test target-token support is inconsistent") + target_characters = _bounded_int(payload, "test_target_characters", positive=True) + if target_characters > test_characters: + raise ValueError("sealed test target-character support is inconsistent") + tokenizer = summary.get("tokenizer_type") + if not isinstance(tokenizer, str): + raise ValueError("run summary tokenizer type must be a string") + return SealedTestObservation( + seed=validation.seed, + tokenizer=tokenizer, + prepared_sha256=prepared_sha256, + comparison_fingerprint=validation.comparison_fingerprint, + checkpoint_step=checkpoint_step, + checkpoint_sha256=checkpoint_sha256, + test_loss=_bounded_float(payload, "test_loss", positive=True), + test_bpc=_bounded_float(payload, "test_bits_per_character", positive=True), + test_tokens=test_tokens, + target_tokens=target_tokens, + target_characters=target_characters, + ) + + +def analyze_sealed_test_matrix( + entries: list[tuple[str, str, Path]], + *, + reference_tokenizer: str, + candidate_tokenizer: str, +) -> SealedMatrixAnalysis: + if not 4 <= len(entries) <= _MAX_ENTRIES: + raise ValueError(f"matrix requires between 4 and {_MAX_ENTRIES} run entries") + if reference_tokenizer == candidate_tokenizer: + raise ValueError("reference and candidate tokenizers must differ") + if len({path.resolve() for _, _, path in entries}) != len(entries): + raise ValueError("matrix run directories must be distinct") + + cells: dict[tuple[str, str], list[SealedTestObservation]] = defaultdict(list) + corpus_hashes: dict[str, set[str]] = defaultdict(set) + for corpus, tokenizer, run_dir in entries: + if not _LABEL.fullmatch(corpus) or not _LABEL.fullmatch(tokenizer): + raise ValueError("corpus and tokenizer labels must be lowercase identifiers") + observation = load_sealed_test_observation(run_dir) + if observation.tokenizer != tokenizer: + raise ValueError(f"tokenizer label does not match run summary: {run_dir}") + cells[(corpus, tokenizer)].append(observation) + corpus_hashes[corpus].add(observation.prepared_sha256) + + corpora = sorted(corpus_hashes) + tokenizers = {tokenizer for _, tokenizer in cells} + if len(corpora) != 2 or tokenizers != {reference_tokenizer, candidate_tokenizer}: + raise ValueError("matrix must contain exactly two corpora and the declared two tokenizers") + if any(len(hashes) != 1 for hashes in corpus_hashes.values()): + raise ValueError("each corpus label must identify exactly one prepared corpus checksum") + if len({next(iter(hashes)) for hashes in corpus_hashes.values()}) != len(corpora): + raise ValueError("different corpus labels must identify different prepared checksums") + expected_cells = {(corpus, tokenizer) for corpus in corpora for tokenizer in sorted(tokenizers)} + if set(cells) != expected_cells: + raise ValueError("matrix is missing a corpus-tokenizer cell") + + expected_seeds: set[int] | None = None + for observations in cells.values(): + seeds = {item.seed for item in observations} + if len(seeds) != len(observations) or len(seeds) < 2: + raise ValueError("every matrix cell requires at least two distinct seeds") + if len({item.comparison_fingerprint for item in observations}) != 1: + raise ValueError("runs within a matrix cell do not share an experiment fingerprint") + if expected_seeds is None: + expected_seeds = seeds + elif seeds != expected_seeds: + raise ValueError("every matrix cell must contain the same seed set") + + contrasts: dict[str, list[tuple[int, float]]] = {} + for corpus in corpora: + reference = {item.seed: item.test_bpc for item in cells[(corpus, reference_tokenizer)]} + candidate = {item.seed: item.test_bpc for item in cells[(corpus, candidate_tokenizer)]} + contrasts[corpus] = [ + (seed, candidate[seed] - reference[seed]) for seed in sorted(reference) + ] + first, second = corpora + first_deltas = dict(contrasts[first]) + second_deltas = dict(contrasts[second]) + interactions = [ + (seed, second_deltas[seed] - first_deltas[seed]) for seed in sorted(first_deltas) + ] + return SealedMatrixAnalysis(dict(cells), contrasts, interactions) diff --git a/tests/test_sealed_matrix.py b/tests/test_sealed_matrix.py new file mode 100644 index 0000000..9079136 --- /dev/null +++ b/tests/test_sealed_matrix.py @@ -0,0 +1,184 @@ +import json +from copy import deepcopy +from dataclasses import replace + +import pytest + +from smallm.config import DataConfig, ExperimentConfig, TrainConfig +from smallm.evaluation.sealed_matrix import ( + SealedTestObservation, + analyze_sealed_test_matrix, + load_sealed_test_observation, +) +from smallm.training.artifacts import write_config_snapshot + + +def _observation(seed, tokenizer, corpus, fingerprint, bpc): + return SealedTestObservation( + seed=seed, + tokenizer=tokenizer, + prepared_sha256=corpus, + comparison_fingerprint=fingerprint, + checkpoint_step=10, + checkpoint_sha256="c" * 64, + test_loss=2.0, + test_bpc=bpc, + test_tokens=100, + target_tokens=99, + target_characters=99, + ) + + +def test_sealed_matrix_computes_paired_contrasts(monkeypatch, tmp_path): + observations = {} + entries = [] + for corpus_index, corpus in enumerate(("first", "second")): + for tokenizer in ("char", "byte_bpe"): + for seed in (1, 2): + path = tmp_path / f"{corpus}-{tokenizer}-{seed}" + path.mkdir() + entries.append((corpus, tokenizer, path)) + baseline = 2.0 + corpus_index * 0.1 + seed * 0.01 + bpc = baseline if tokenizer == "char" else baseline - 0.05 + observations[path] = _observation( + seed, + tokenizer, + str(corpus_index) * 64, + f"{corpus}-{tokenizer}", + bpc, + ) + monkeypatch.setattr( + "smallm.evaluation.sealed_matrix.load_sealed_test_observation", + lambda path: observations[path], + ) + + analysis = analyze_sealed_test_matrix( + entries, reference_tokenizer="char", candidate_tokenizer="byte_bpe" + ) + + assert [seed for seed, _ in analysis.contrasts["first"]] == [1, 2] + assert [value for _, value in analysis.contrasts["first"]] == pytest.approx([-0.05, -0.05]) + assert [value for _, value in analysis.contrasts["second"]] == pytest.approx([-0.05, -0.05]) + assert [value for _, value in analysis.interactions] == pytest.approx([0.0, 0.0]) + + +def test_sealed_matrix_rejects_unbalanced_or_inconsistent_cells(monkeypatch, tmp_path): + paths = [tmp_path / str(index) for index in range(8)] + for path in paths: + path.mkdir() + observations = { + path: _observation( + 1 + index % 2, + "char" if (index // 2) % 2 == 0 else "byte_bpe", + "a" * 64 if index < 4 else "b" * 64, + f"fingerprint-{index // 2}", + 2.0, + ) + for index, path in enumerate(paths) + } + monkeypatch.setattr( + "smallm.evaluation.sealed_matrix.load_sealed_test_observation", + lambda path: observations[path], + ) + entries = [ + ("first" if index < 4 else "second", observations[path].tokenizer, path) + for index, path in enumerate(paths) + ] + analyze_sealed_test_matrix(entries, reference_tokenizer="char", candidate_tokenizer="byte_bpe") + + observations[paths[-1]] = replace(observations[paths[-1]], seed=3) + with pytest.raises(ValueError, match="same seed set"): + analyze_sealed_test_matrix( + entries, reference_tokenizer="char", candidate_tokenizer="byte_bpe" + ) + + +def test_load_sealed_test_observation_validates_run_identity(tmp_path): + run_dir = tmp_path / "run" + run_dir.mkdir() + config = ExperimentConfig( + data=DataConfig(train_split=0.8, validation_split=0.1), + train=TrainConfig(seed=2027, run_name="fixture", max_steps=10), + ) + write_config_snapshot(run_dir / "config.yaml", config) + prepared_sha256 = "a" * 64 + summary = { + "schema_version": 2, + "status": "complete", + "actual_steps": 10, + "best_val_step": 8, + "best_val_bits_per_char": 2.0, + "final_val_bits_per_char": 2.1, + "duration_seconds": 1.0, + "tokenizer_type": "char", + "validation_mode": "full", + "validation_coverage": 1.0, + "test_status": "sealed_unread", + "test_characters": 100, + "dataset": { + "prepared_sha256": prepared_sha256, + "train_split": 0.9, + "train_characters": 800, + "validation_characters": 100, + }, + } + (run_dir / "summary.json").write_text(json.dumps(summary), encoding="utf-8") + test_result = { + "schema_version": 1, + "status": "complete", + "checkpoint_kind": "best", + "checkpoint_step": 8, + "checkpoint_sha256": "b" * 64, + "prepared_sha256": prepared_sha256, + "evaluation_mode": "full", + "test_coverage": 1.0, + "test_characters": 100, + "test_tokens": 100, + "test_target_tokens": 99, + "test_total_target_tokens": 99, + "test_target_characters": 99, + "test_loss": 1.5, + "test_bits_per_character": 2.2, + } + path = run_dir / "test_evaluation_best.json" + path.write_text(json.dumps(test_result), encoding="utf-8") + + observation = load_sealed_test_observation(run_dir) + assert observation.seed == 2027 + assert observation.test_bpc == 2.2 + assert observation.checkpoint_step == 8 + + cases = [ + ({"schema_version": 2}, {}, "complete schema-v1"), + ({"checkpoint_kind": "final"}, {}, "best-checkpoint"), + ({"evaluation_mode": "sampled"}, {}, "full test coverage"), + ({"test_coverage": 0.5}, {}, "full test coverage"), + ({"prepared_sha256": "c" * 64}, {}, "corpus checksum"), + ({"checkpoint_step": 7}, {}, "best validation step"), + ({"test_characters": 101}, {}, "support does not match"), + ({"checkpoint_sha256": "B" * 64}, {}, "lowercase SHA-256"), + ({"test_tokens": True}, {}, "bounded integer"), + ({"test_target_tokens": 98}, {}, "target-token support"), + ({"test_target_characters": 101}, {}, "target-character support"), + ({"test_loss": float("inf")}, {}, "outside supported bounds"), + ({}, {"dataset": {"prepared_sha256": "invalid"}}, "prepared corpus checksum"), + ({}, {"tokenizer_type": None}, "tokenizer type"), + ] + for result_updates, summary_updates, message in cases: + changed_result = deepcopy(test_result) + changed_result.update(result_updates) + changed_summary = deepcopy(summary) + changed_summary.update(summary_updates) + path.write_text(json.dumps(changed_result), encoding="utf-8") + (run_dir / "summary.json").write_text(json.dumps(changed_summary), encoding="utf-8") + with pytest.raises(ValueError, match=message): + load_sealed_test_observation(run_dir) + + (run_dir / "summary.json").write_text(json.dumps(summary), encoding="utf-8") + path.write_text(json.dumps([]), encoding="utf-8") + with pytest.raises(ValueError, match="JSON object"): + load_sealed_test_observation(run_dir) + + path.write_bytes(b" " * 1_000_001) + with pytest.raises(ValueError, match="exceeds"): + load_sealed_test_observation(run_dir) diff --git a/tests/test_summarize_runs.py b/tests/test_summarize_runs.py index b889d8a..45305c5 100644 --- a/tests/test_summarize_runs.py +++ b/tests/test_summarize_runs.py @@ -8,7 +8,9 @@ from smallm.training.artifacts import write_config_snapshot -def _write_run(tmp_path, name, *, seed=1, learning_rate=1e-3, summary_updates=None): +def _write_run( + tmp_path, name, *, seed=1, sample_seed=1337, learning_rate=1e-3, summary_updates=None +): run_dir = tmp_path / name run_dir.mkdir() write_config_snapshot( @@ -18,6 +20,7 @@ def _write_run(tmp_path, name, *, seed=1, learning_rate=1e-3, summary_updates=No run_name=name, runs_dir=str(tmp_path), seed=seed, + sample_seed=sample_seed, learning_rate=learning_rate, ) ), @@ -47,8 +50,8 @@ def _write_run(tmp_path, name, *, seed=1, learning_rate=1e-3, summary_updates=No def test_observations_verify_training_comparability(tmp_path): - first = _observation(_write_run(tmp_path, "first", seed=1)) - second = _observation(_write_run(tmp_path, "second", seed=2)) + first = _observation(_write_run(tmp_path, "first", seed=1, sample_seed=101)) + second = _observation(_write_run(tmp_path, "second", seed=2, sample_seed=202)) different = _observation(_write_run(tmp_path, "different", seed=3, learning_rate=2e-3)) assert first.comparison_fingerprint == second.comparison_fingerprint