commit_msg.txt

feat: implement RFC-013 Tokenizer Framework

Implement the model-independent tokenizer framework providing a unified
abstraction for text-to-token and token-to-text conversion.

Public API (tokenizer.h):
- HB_Tokenizer vtable with encode, decode, incremental encode/decode,
  vocabulary lookup, validation, and lifecycle callbacks
- HB_Vocabulary with FNV-1a hash table, special/unknown/reserved token
  management, duplicate detection, and UTF-8 validation
- HB_TokenSequence growable array with append/append_many/get/clear
- HB_DecodeState for streaming UTF-8 incremental decoding
- HB_TokenizerRegistry name-keyed plugin registry (max 16 slots)
- HB_TokenizerManager binding tokenizer+vocab+allocator with timing
  statistics (encode/decode ns, tokens processed, vocab lookups)
- UTF-8 utilities: encode, decode, validate, sequence length

Mock tokenizer (tokenizer_mock.c):
- Byte-level tokenizer (ID = byte + 5) exercising all vtable paths
- Special tokens at IDs 0-4: UNK, PAD, BOS, EOS, SPACE
- Incremental encode context tracking bytes seen

Tests (tokenizer_test.c, 25 functions):
- UTF-8 encode/decode/validate including multibyte and invalid sequences
- Token sequence growth, vocabulary CRUD, special tokens, duplicates
- Registry register/lookup/duplicate rejection
- Manager encode/decode round-trip, incremental paths, statistics, reset
- Null argument handling, selftest

Build (CMakeLists.txt):
- Links hb_common, hb_memory, hb_platform
- Test links hb_test_support for hbi_test.h include path

Not implemented (out of scope):
- GPT-OSS/GLM/SentencePiece/BPE/WordPiece/Unigram tokenizers
- Model-specific tokenization rules

All 33/33 CTest cases pass under -Werror (GCC 16.1.0 UCRT64).