Add preregistered external corpus panel

AGENTS.md

@@ -90,10 +90,11 @@ If a relevant check cannot be run, state why and what remains unverified.
 
 ## Current Technical Status
 
-Milestone 027 is the latest modeling evidence. It was preregistered before Hamlet corpus access and
-confirms ByteBPE512 over character by `0.0673` sealed-test BPC on a dramatic play. Hamlet test BPC
-is better than validation for both models, reversing milestone 026's harder-tail pattern. Alice,
-Peter Pan, and Hamlet tests are consumed and must not guide further tuning.
+Milestone 028 is the latest modeling evidence. Its preregistered 2-corpus × 2-tokenizer × 3-seed
+panel finds ByteBPE512 wins all six sealed-test pairs, averaging `0.1134` BPC better on Art of War
+and `0.1543` on Lincoln. Both new terminal regions are harder than validation, and effect magnitude
+remains corpus-dependent. All previously reported sealed-test segments are consumed and must not
+drive later selection.
 
 ## Safety And Security
 

README.md

@@ -17,8 +17,8 @@ For a fast technical review, inspect:
 1. [`docs/architecture.md`](docs/architecture.md) for boundaries and the
    end-to-end pipeline.
 2. [`docs/experiments.md`](docs/experiments.md) for the milestone index.
-3. [`experiments/027-hamlet-external-distribution.md`](experiments/027-hamlet-external-distribution.md)
-   for the latest preregistered external-distribution evidence and its limitations.
+3. [`experiments/028-preregistered-external-corpus-panel.md`](experiments/028-preregistered-external-corpus-panel.md)
+   for the latest preregistered multi-corpus, multi-seed evidence and its limitations.
 4. [`src/smallm/model/`](src/smallm/model/) for the from-scratch GPTiny model.
 5. [`src/smallm/training/`](src/smallm/training/) for training and run artifacts.
 6. [`src/smallm/data/`](src/smallm/data/) for corpus and tokenizer contracts.
@@ -52,6 +52,7 @@ chronological 90/10 split unless noted otherwise.
 | [025](experiments/025-corpus-by-seed-matrix.md) | 2 tokenizers × 2 corpora × 3 seeds | Paired ByteBPE512 minus character BPC | `-0.0619` Alice; `-0.0252` Peter Pan | ByteBPE512 wins all six pairs; effect magnitude is corpus-dependent. |
 | [026](experiments/026-sealed-test-evaluation.md) | Frozen decision on terminal 10% test segments | ByteBPE512 minus character test BPC | `-0.0614` Alice; `-0.0258` Peter Pan | The tokenizer decision survives one-shot full-coverage tests on both books. |
 | [027](experiments/027-hamlet-external-distribution.md) | Preregistered Hamlet play replication | ByteBPE512 minus character test BPC | `-0.0673` | The direction replicates on drama, while validation/test difficulty and effect size remain corpus-dependent. |
+| [028](experiments/028-preregistered-external-corpus-panel.md) | 2 new corpora × 2 tokenizers × 3 seeds | Mean paired ByteBPE512 minus character test BPC | `-0.1134` Art of War; `-0.1543` Lincoln | ByteBPE512 wins all six preregistered pairs; magnitude and chronological-tail difficulty remain corpus-dependent. |
 
 Token-level loss and perplexity are not directly comparable between character
 and BPE tokenizers because they predict different units. The tokenizer
@@ -86,7 +87,7 @@ shapes link directly to the implementation and its tests.
 | Corpus preparation | Normalized corpus output, stats, checksums, source metadata, and manifest files. |
 | Tokenization | Character, educational character-BPE, and lossless boundary-aware UTF-8 byte-BPE tokenizers. |
 | Model | Decoder-only GPT-style Transformer with causal self-attention. |
-| Evaluation | Uniform, unigram, and add-one bigram baselines; optional sealed chronological test evaluation. |
+| Evaluation | Uniform, unigram, and add-one bigram baselines; sealed chronological tests and complete factorial-matrix aggregation. |
 | Training | Config-driven training with validation loss, optional early stopping, progress logging, checkpoints, metrics, summaries, and samples. |
 | Run records | Preserved run directories with copied dataset manifests and selected provenance fields in `summary.json`. |
 | Generation | `max_new_tokens`, `temperature`, `top_k`, `seed`, and greedy decoding. |

configs/gptiny_artofwar_bytebpe512_sealed_seed1337.yaml

@@ -0,0 +1,40 @@
+data:
+  input_path: data/raw/art_of_war_body.txt
+  prepared_path: data/processed/art_of_war_corpus.txt
+  manifest_path: data/processed/art_of_war_corpus_sealed_manifest.json
+  tokenizer_path: data/processed/art_of_war_tokenizer_bytebpe512_sealed.json
+  tokenizer_type: byte_bpe
+  bpe_vocab_size: 512
+  bpe_min_frequency: 2
+  block_size: 37
+  train_split: 0.8
+  validation_split: 0.1
+
+model:
+  vocab_size: 512
+  block_size: 37
+  n_layer: 4
+  n_head: 4
+  n_embd: 128
+  dropout: 0.1
+
+train:
+  run_name: gptiny_artofwar_bytebpe512_sealed_seed1337
+  runs_dir: runs
+  batch_size: 27
+  max_steps: 5000
+  learning_rate: 0.001
+  weight_decay: 0.0
+  log_interval: 100
+  eval_interval: 250
+  eval_batches: null
+  early_stopping_patience: 3
+  early_stopping_min_delta: 0.0
+  sample_prompt: "The art"
+  sample_max_new_tokens: 100
+  sample_temperature: 1.0
+  sample_top_k: null
+  sample_seed: 1337
+  sample_greedy: false
+  seed: 1337
+

configs/gptiny_artofwar_bytebpe512_sealed_seed2027.yaml

@@ -0,0 +1,40 @@
+data:
+  input_path: data/raw/art_of_war_body.txt
+  prepared_path: data/processed/art_of_war_corpus.txt
+  manifest_path: data/processed/art_of_war_corpus_sealed_manifest.json
+  tokenizer_path: data/processed/art_of_war_tokenizer_bytebpe512_sealed.json
+  tokenizer_type: byte_bpe
+  bpe_vocab_size: 512
+  bpe_min_frequency: 2
+  block_size: 37
+  train_split: 0.8
+  validation_split: 0.1
+
+model:
+  vocab_size: 512
+  block_size: 37
+  n_layer: 4
+  n_head: 4
+  n_embd: 128
+  dropout: 0.1
+
+train:
+  run_name: gptiny_artofwar_bytebpe512_sealed_seed2027
+  runs_dir: runs
+  batch_size: 27
+  max_steps: 5000
+  learning_rate: 0.001
+  weight_decay: 0.0
+  log_interval: 100
+  eval_interval: 250
+  eval_batches: null
+  early_stopping_patience: 3
+  early_stopping_min_delta: 0.0
+  sample_prompt: "The art"
+  sample_max_new_tokens: 100
+  sample_temperature: 1.0
+  sample_top_k: null
+  sample_seed: 2027
+  sample_greedy: false
+  seed: 2027
+

configs/gptiny_artofwar_bytebpe512_sealed_seed4242.yaml

@@ -0,0 +1,40 @@
+data:
+  input_path: data/raw/art_of_war_body.txt
+  prepared_path: data/processed/art_of_war_corpus.txt
+  manifest_path: data/processed/art_of_war_corpus_sealed_manifest.json
+  tokenizer_path: data/processed/art_of_war_tokenizer_bytebpe512_sealed.json
+  tokenizer_type: byte_bpe
+  bpe_vocab_size: 512
+  bpe_min_frequency: 2
+  block_size: 37
+  train_split: 0.8
+  validation_split: 0.1
+
+model:
+  vocab_size: 512
+  block_size: 37
+  n_layer: 4
+  n_head: 4
+  n_embd: 128
+  dropout: 0.1
+
+train:
+  run_name: gptiny_artofwar_bytebpe512_sealed_seed4242
+  runs_dir: runs
+  batch_size: 27
+  max_steps: 5000
+  learning_rate: 0.001
+  weight_decay: 0.0
+  log_interval: 100
+  eval_interval: 250
+  eval_batches: null
+  early_stopping_patience: 3
+  early_stopping_min_delta: 0.0
+  sample_prompt: "The art"
+  sample_max_new_tokens: 100
+  sample_temperature: 1.0
+  sample_top_k: null
+  sample_seed: 4242
+  sample_greedy: false
+  seed: 4242
+

configs/gptiny_artofwar_char_sealed_seed1337.yaml

@@ -0,0 +1,38 @@
+data:
+  input_path: data/raw/art_of_war_body.txt
+  prepared_path: data/processed/art_of_war_corpus.txt
+  manifest_path: data/processed/art_of_war_corpus_sealed_manifest.json
+  tokenizer_path: data/processed/art_of_war_tokenizer_char_sealed.json
+  tokenizer_type: char
+  block_size: 64
+  train_split: 0.8
+  validation_split: 0.1
+
+model:
+  vocab_size: 256
+  block_size: 64
+  n_layer: 4
+  n_head: 4
+  n_embd: 128
+  dropout: 0.1
+
+train:
+  run_name: gptiny_artofwar_char_sealed_seed1337
+  runs_dir: runs
+  batch_size: 16
+  max_steps: 5000
+  learning_rate: 0.001
+  weight_decay: 0.0
+  log_interval: 100
+  eval_interval: 250
+  eval_batches: null
+  early_stopping_patience: 3
+  early_stopping_min_delta: 0.0
+  sample_prompt: "The art"
+  sample_max_new_tokens: 100
+  sample_temperature: 1.0
+  sample_top_k: null
+  sample_seed: 1337
+  sample_greedy: false
+  seed: 1337
+

configs/gptiny_artofwar_char_sealed_seed2027.yaml

@@ -0,0 +1,38 @@
+data:
+  input_path: data/raw/art_of_war_body.txt
+  prepared_path: data/processed/art_of_war_corpus.txt
+  manifest_path: data/processed/art_of_war_corpus_sealed_manifest.json
+  tokenizer_path: data/processed/art_of_war_tokenizer_char_sealed.json
+  tokenizer_type: char
+  block_size: 64
+  train_split: 0.8
+  validation_split: 0.1
+
+model:
+  vocab_size: 256
+  block_size: 64
+  n_layer: 4
+  n_head: 4
+  n_embd: 128
+  dropout: 0.1
+
+train:
+  run_name: gptiny_artofwar_char_sealed_seed2027
+  runs_dir: runs
+  batch_size: 16
+  max_steps: 5000
+  learning_rate: 0.001
+  weight_decay: 0.0
+  log_interval: 100
+  eval_interval: 250
+  eval_batches: null
+  early_stopping_patience: 3
+  early_stopping_min_delta: 0.0
+  sample_prompt: "The art"
+  sample_max_new_tokens: 100
+  sample_temperature: 1.0
+  sample_top_k: null
+  sample_seed: 2027
+  sample_greedy: false
+  seed: 2027
+

configs/gptiny_artofwar_char_sealed_seed4242.yaml

@@ -0,0 +1,38 @@
+data:
+  input_path: data/raw/art_of_war_body.txt
+  prepared_path: data/processed/art_of_war_corpus.txt
+  manifest_path: data/processed/art_of_war_corpus_sealed_manifest.json
+  tokenizer_path: data/processed/art_of_war_tokenizer_char_sealed.json
+  tokenizer_type: char
+  block_size: 64
+  train_split: 0.8
+  validation_split: 0.1
+
+model:
+  vocab_size: 256
+  block_size: 64
+  n_layer: 4
+  n_head: 4
+  n_embd: 128
+  dropout: 0.1
+
+train:
+  run_name: gptiny_artofwar_char_sealed_seed4242
+  runs_dir: runs
+  batch_size: 16
+  max_steps: 5000
+  learning_rate: 0.001
+  weight_decay: 0.0
+  log_interval: 100
+  eval_interval: 250
+  eval_batches: null
+  early_stopping_patience: 3
+  early_stopping_min_delta: 0.0
+  sample_prompt: "The art"
+  sample_max_new_tokens: 100
+  sample_temperature: 1.0
+  sample_top_k: null
+  sample_seed: 4242
+  sample_greedy: false
+  seed: 4242
+

configs/gptiny_lincoln_bytebpe512_sealed_seed1337.yaml

@@ -0,0 +1,40 @@
+data:
+  input_path: data/raw/lincoln_speeches_body.txt
+  prepared_path: data/processed/lincoln_speeches_corpus.txt
+  manifest_path: data/processed/lincoln_speeches_corpus_sealed_manifest.json
+  tokenizer_path: data/processed/lincoln_speeches_tokenizer_bytebpe512_sealed.json
+  tokenizer_type: byte_bpe
+  bpe_vocab_size: 512
+  bpe_min_frequency: 2
+  block_size: 37
+  train_split: 0.8
+  validation_split: 0.1
+
+model:
+  vocab_size: 512
+  block_size: 37
+  n_layer: 4
+  n_head: 4
+  n_embd: 128
+  dropout: 0.1
+
+train:
+  run_name: gptiny_lincoln_bytebpe512_sealed_seed1337
+  runs_dir: runs
+  batch_size: 27
+  max_steps: 5000
+  learning_rate: 0.001
+  weight_decay: 0.0
+  log_interval: 100
+  eval_interval: 250
+  eval_batches: null
+  early_stopping_patience: 3
+  early_stopping_min_delta: 0.0
+  sample_prompt: "Fellow"
+  sample_max_new_tokens: 100
+  sample_temperature: 1.0
+  sample_top_k: null
+  sample_seed: 1337
+  sample_greedy: false
+  seed: 1337
+

configs/gptiny_lincoln_bytebpe512_sealed_seed2027.yaml

@@ -0,0 +1,40 @@
+data:
+  input_path: data/raw/lincoln_speeches_body.txt
+  prepared_path: data/processed/lincoln_speeches_corpus.txt
+  manifest_path: data/processed/lincoln_speeches_corpus_sealed_manifest.json
+  tokenizer_path: data/processed/lincoln_speeches_tokenizer_bytebpe512_sealed.json
+  tokenizer_type: byte_bpe
+  bpe_vocab_size: 512
+  bpe_min_frequency: 2
+  block_size: 37
+  train_split: 0.8
+  validation_split: 0.1
+
+model:
+  vocab_size: 512
+  block_size: 37
+  n_layer: 4
+  n_head: 4
+  n_embd: 128
+  dropout: 0.1
+
+train:
+  run_name: gptiny_lincoln_bytebpe512_sealed_seed2027
+  runs_dir: runs
+  batch_size: 27
+  max_steps: 5000
+  learning_rate: 0.001
+  weight_decay: 0.0
+  log_interval: 100
+  eval_interval: 250
+  eval_batches: null
+  early_stopping_patience: 3
+  early_stopping_min_delta: 0.0
+  sample_prompt: "Fellow"
+  sample_max_new_tokens: 100
+  sample_temperature: 1.0
+  sample_top_k: null
+  sample_seed: 2027
+  sample_greedy: false
+  seed: 2027
+