Preserve existing tokenizer metadata

[NeonBohdan]

Re-adding an existing configured token as a plain string can replace its AddedToken metadata, including special-token, matching, normalization, and whitespace flags.

Skip names already present in the selected output vocabulary. Missing names are still added normally, and configured embedding source and force behavior remains unchanged.

The regression covers explicit-model, base, and union tokenizer construction, plus ordinary missing-token addition on the direct-model path.

---

Note

Low Risk
Small guard in tokenizer construction with targeted tests; no auth, data, or merge-math changes.

Overview
When merge config lists tokens that are already in the output vocabulary, build_tokenizer no longer calls add_tokens for those names. Re-adding them as plain strings could overwrite AddedToken metadata (special-token flags, lstrip/rstrip, normalization, etc.).

Missing configured token names are still added as before; embedding source / force behavior is unchanged.

A new regression test (model_special fixture + parametrized test_configured_tokens_preserve_metadata_and_add_missing) checks that an existing special token keeps id, decoder metadata, and decode behavior across model, base, and union tokenizer sources, and that a genuinely new configured token is still added on the direct-model path.

^{Reviewed by Cursor Bugbot for commit 42a8ecc. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[NeonBohdan]

I have read the CLA Document and I hereby sign the CLA