Skip to content

Bug: Fix token counting error when using Unicode characters #338

Description

@SHAURYASANYAL3

Emojis and multi-byte characters can throw off the token counter. Use the official tokenizer library.

Metadata

Metadata

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions