[dependabot]

Bumps tokenizers from 0.23.0rc0 to 1.0.0rc2.

Release notes

Sourced from tokenizers's releases.

Release candidate v1.0.0.rc.2

Making tokenizers the sota tokenization library, same API, same token IDs, same standards

We are happy to finally bring tokenizers up to speed ⚡ ! tokenizers was released 7 years ago, in a time where running the model was always the bottleneck. The ecosystem has evolved so much that tokenizers became a the bottleneck.

As we saw many different libraries and project pop out, improving the tokenizer, we realized that our library's entire stack was too outdated for external contributors to contribute. This pushed us to refactor the codebase, and make sure the next developers come here to push the performances of tokenization.

The TLDR; is that we're a lot faster than before, crate size 6x smaller, peak memory is reduced. We worked a lot to reduce the UTF-8 tax, and make sure the improvements span across all languages.

For more details on the release, read this.

Big Kudos to @​SBrandeis and @​McPatate for this sprint. Special thanks to @​sebpop for his involvement as well.

v0.23.2

This is the last v0 release, we are moving to v1!!

More details coming soon 👀

What's Changed

• Update daachorse from 1.0.1 to 3.0.0 by @​musicinmybrain in huggingface/tokenizers#2050
• Ci test fixes by @​sebpop in huggingface/tokenizers#2079
• Avoid full vocab clone in get_vocab_size() by @​eunseo9311 in huggingface/tokenizers#2074
• ci: pin benchmark jobs to a larger dedicated runner by @​ArthurZucker in huggingface/tokenizers#2084
• ci: target benchmark runners by group, not nonexistent label by @​ArthurZucker in huggingface/tokenizers#2090
• Bindings: acquire model read-lock once per call instead of per pre-token by @​sebpop in huggingface/tokenizers#2072
• ci: split audit in a separate job by @​SBrandeis in huggingface/tokenizers#2103
• ci: install libcairo on benchmark runners by @​SBrandeis in huggingface/tokenizers#2100
• feat(ci): enable npm's trusted publisher for releases by @​McPatate in huggingface/tokenizers#2111
• chore: bump pyo3 to version 0.29 by @​SBrandeis in huggingface/tokenizers#2115
• perf: add benchmarks for the decode API by @​SBrandeis in huggingface/tokenizers#2099
• ci: cache HF test data and authenticate Hub downloads to avoid rate limits by @​SBrandeis in huggingface/tokenizers#2102
• Implementing Parity-aware BPE by @​cimeister in huggingface/tokenizers#1974
• Make doc builds faster by @​mishig25 in huggingface/tokenizers#2158
• chore(deps-dev): bump websocket-driver from 0.7.4 to 0.7.5 in /tokenizers/examples/unstable_wasm/www by @​dependabot[bot] in huggingface/tokenizers#2208
• chore(deps-dev): bump shell-quote from 1.8.3 to 1.10.0 in /tokenizers/examples/unstable_wasm/www by @​dependabot[bot] in huggingface/tokenizers#2244
• chore: remove stale examples (kills 50% of dependabot traffic) by @​ArthurZucker in huggingface/tokenizers#2285
• chore(node): drop 8 unused devDependencies by @​ArthurZucker in huggingface/tokenizers#2286
• ci: pin every action to one SHA, drop stale audit suppressions by @​ArthurZucker in huggingface/tokenizers#2291
• chore(node): take node security alerts from 33 to 0 by @​ArthurZucker in huggingface/tokenizers#2287
• fix(python): pin the ruff rule set so a ruff release can't redden main by @​ArthurZucker in huggingface/tokenizers#2292
• Add a ParityBpeTrainer example and list it in the trainers API docs by @​cimeister in huggingface/tokenizers#2217

New Contributors

• @​eunseo9311 made their first contribution in huggingface/tokenizers#2074
• @​cimeister made their first contribution in huggingface/tokenizers#1974

Full Changelog: huggingface/tokenizers@v0.23.1...v0.23.2

... (truncated)

Commits

• 7616272 release: v1.0.0-rc.2
• fb11e51 fix(release): max number of keywords is 5 (#2444)
• 8bad46a fix (#2443)
• a9f3f96 fix: crates.io release env name (#2441)
• a1c2cad enrich v1 feature list (#2438)
• 0917be8 Publish to npm via Trusted Publishing (#2439)
• ea787c0 Drop fmt/clippy from bitcannon CI, rust.yml already runs them (#2440)
• 1769c3f manual (#2436)
• 6205240 more cleanup (#2435)
• 50200a3 Delete for_the_blogpost.md
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[dependabot]

Labels

The following labels could not be found: dependencies. Please create it before Dependabot can add it to a pull request.

Please fix the above issues or remove invalid values from dependabot.yml.