All pull requests

Pull requests

• fix: correct BPE merge annotations in test_wikipedia_example

  #111·coding-shalabh opened Mar 15, 2026·1 task done·

  ·
  1

• Fix typos in exercise instructions

  #107·timlt3 opened Jan 19, 2026·

  ·

• add parallel tokenizer

  #99·NinjaKelly opened Sep 20, 2025·

  ·
  1

• Extract best-pair selection from _encode_chunk into find_best_pair() for clarity

  #97·manoogim opened Aug 7, 2025·

  ·

• add lexicographic ordering for breaking ties to make the tokenizer deterministic

  #90·dapopov-st opened Sep 21, 2024·

  ·

• Add pyproject toml

  #89·gianlucagiudice opened Sep 17, 2024·

  ·

• updating stats across merge to reduce computation

  #88·imdaredevil opened Sep 1, 2024·

  ·

• Update README.md

  #86·wenming-ma opened Jul 1, 2024·

  ·

• Optimal algorithm for _encode_chunk(): 20% faster encoding, with 0.5% better COMPRESSION

  #84·Majdoddin opened Jun 17, 2024·

  ·

• Deduplication of text chunks with frequency count, training and encoding 5x speedup

  enhancement
  #82·Majdoddin opened Jun 8, 2024·

  ·
  4

• calling len(ids) in merge() function only once to increase performance

  #76·crpatil1901 opened May 14, 2024·

  ·

• Link to Mojo port added

  #75·dorjeduck opened May 14, 2024·

  ·

• Count only nonoverlapping occurences of a pair

  #72·Majdoddin opened May 6, 2024·

  ·
  1

• Update regex.py to correctly parse scripts with combining marks

  #71·ajaykg opened May 5, 2024·

  ·
  5

• Updated decode() method in GPT4Tokenizer so that it handles special t…

  #63·Vakarva opened Apr 7, 2024·

  ·

• Handle error when running out of pairs to merge

  #54·vinhdq842 opened Mar 6, 2024·

  ·

• updated self.vocab initialization and reuse self._build_vocab()

  #53·muerghq opened Mar 2, 2024·

  ·

• Video2Post Generation Workflow

  #49·xihajun opened Feb 26, 2024·8 of 17 tasks·

  ·
  3

• Update lecture.md based on video tutorial content from 08:15 through 28:23

  #42·astaff opened Feb 23, 2024·

  ·

• Automating testing using github actions

  #41·nobleaustine opened Feb 22, 2024Contributor·

  ·

• Create setup.py

  #39·Biswajit2902 opened Feb 22, 2024·

  ·
  1

• Train BasicTokenizer on GPU with PyTorch, 100x speedup

  #38·kuprel opened Feb 22, 2024·

  ·
  5

• Fix small typos

  #34·ggaughan opened Feb 21, 2024·

  ·
  1

• Simplify generation of printable representation

  #26·SultanOrazbayev opened Feb 21, 2024·

  ·
  2

• Batch encoding decoding

  #22·ezzaldeeen opened Feb 20, 2024·

  ·
  2