Skip to content

chore: update chardet requirement from >=7.6.0 to >=7.7.0 - #10

Merged
laincat merged 1 commit into
mainfrom
dependabot/pip/chardet-gte-7.7.0
Oct 26, 2026
Merged

laincat merged 1 commit into
mainfrom
dependabot/pip/chardet-gte-7.7.0

Conversation

@dependabot

@dependabot dependabot Bot commented on behalf of github Oct 26, 2026

Copy link
Copy Markdown
Contributor

Updates the requirements on chardet to permit the latest version.

Release notes

Sourced from chardet's releases.

7.7.0

Big release: raising max_bytes to cover a huge input no longer makes detect() slow, every answer now decodes the window it was detected from, and markup declarations are read the way browsers read them. A lot of this came from new contributors, so thank you all!

Performance

  • Raising max_bytes on large inputs no longer gets slow. On compiled wheels, detect(data, max_bytes=len(data)) on 272 MiB takes about 0.2s for cp1252 and 0.5s for Shift_JIS, down from several seconds. UTF-8 validation and a final decode of the winner still read the whole window, and the other stages stop at a 256 KiB evidence cap (ADR-0006). The default 200 KB cap is still the fastest path, and default calls are unchanged.
  • The pure-Python wheel on CPython scores bigram models through operator.itemgetter instead of a Python loop, making detect() about 1.25x faster and detect_all() about 1.35x faster. (#404, thanks @​deadstrobe5)
  • Fixed three scans that went quadratic on hostile input: the UTF-7 + scan on line-wrapped base64 (128 KiB went from 5.4s to 1.2ms), unclosed <meta and <?xml tags in the markup scan, and the EBCDIC charset-declaration regex. (#403, thanks @​Samin061)

Bug Fixes

  • Every answer now decodes the examined window under the name you get back. BOMs, BOM-less UTF-16/32 null patterns, ISO-2022/HZ escapes, and <meta charset> declarations used to be trusted without decoding the rest of the window, so a confident answer could still raise further in. The 5.x/6.x display names (EUC-JP, SHIFT_JIS, Big5, ISO-2022-JP) are only used when those narrower codecs decode it too. (#394 and #396, thanks @​Samin061)
  • Markup declarations follow the WHATWG prescan: declarations inside comments are ignored, a charset= inside another attribute (like a refresh URL) no longer beats a real <meta charset>, quoted or spaced content charsets work, and an ASCII page declaring utf-16le no longer comes back as UTF-16-LE. (#389 and #398, thanks @​SashaMIT and @​xiehuanyi)
  • PEP 263 cookies resolve the way tokenize.detect_encoding resolves them, including only counting a line-2 cookie when line 1 is blank or a comment. (#391, thanks @​xiehuanyi)
  • Fixed BOM-bearing UTF-32 coming back as UTF-16 when max_bytes cuts a code unit. (#405, thanks @​AryanRG)
  • Fixed language labels on Unicode text tilting toward accent-heavy languages, so a French paragraph with a few accents no longer comes back as Hungarian. Mislabels on the test corpus's Unicode files dropped from 140 to 104.
  • Fixed a mostly-ASCII Windows-1252 file with a single Ö detecting as HP-Roman8, and English with a lone É detecting as MacRoman. (#383, thanks @​aadsm)
  • Fixed null-separated records like find -print0 output coming back as UTF-16-LE with Chinese as the language.
  • prefer_superset=True no longer swaps in a code page that can't decode the data, and now respects include_encodings/exclude_encodings. (#393, thanks @​00200200)
  • chardetect and UniversalDetector no longer mistake input cut exactly at the byte cap for the whole stream. (#392, thanks @​00200200)
  • lookup_encoding's cache is capped at 1024 entries, since charset names come from untrusted markup. (#390, thanks @​Samin061)
  • Plus smaller fixes to detect_all() ranking, EBCDIC XML MIME types, and the deprecated chardet.equivalences shim. The changelog has the full list.

Improvements

  • New -m/--mime-type flag for chardetect shows the MIME type next to the encoding, like file -i. (#384, thanks @​smi11 for the request)
  • Visual-order Hebrew is now detected, and cp864 Arabic is trained on contextually shaped text in visual order, the way real cp864 systems stored it.
  • EncodingInfo.is_multibyte is now is_multibyte_cjk. The old name still works with a DeprecationWarning and goes away in 8.0.
  • pipeline/markup.py is now compiled by mypyc (15 modules total), and CI enforces 100% test coverage.

Full Changelog: chardet/chardet@7.6.0...7.7.0

Changelog

Sourced from chardet's changelog.

7.7.0 (2026-10-17)

Features:

  • Added a -m/--mime-type flag to the chardetect CLI, which shows the detected MIME type next to the encoding, like file -i. (Dan Blanchard <https://github.com/dan-blanchard>_ via Claude, [#384](https://github.com/chardet/chardet/issues/384) <https://github.com/chardet/chardet/issues/384>_)

Accuracy:

  • Visual-order Hebrew is now detected. The ISO-8859-8 model trains on both bidi conventions, so 1990s visual-order pages come back as ISO-8859-8 while logical-order text still prefers Windows-1255. (Dan Blanchard <https://github.com/dan-blanchard>_ via Claude)
  • cp864 Arabic now trains on contextually shaped text in visual order, the way real cp864 systems stored it, instead of isolated letter forms in logical order. (Dan Blanchard <https://github.com/dan-blanchard>_ via Claude)

Performance:

  • Raising max_bytes on large inputs no longer gets slow. On the compiled build, detect(data, max_bytes=len(data)) on 272 MiB takes about 0.23s for cp1252 and 0.56s for Shift_JIS, down from several seconds. UTF-8 validation and a final decode of the winner still read the whole window, so the answer still decodes it. The other stages stop at a 256 KB evidence cap, which sits above the default max_bytes, so default calls are unchanged, and the default 200 KB cap is still the fastest path. ADR-0006 has the design. (Dan Blanchard <https://github.com/dan-blanchard>_ via Claude)
  • The UTF-7 + scan is no longer quadratic on line-wrapped base64 like PEM certificates, MIME attachments, and JWTs. 128 KiB of it went from 5.4 seconds to 1.2 milliseconds. (Dan Blanchard <https://github.com/dan-blanchard>_ via Claude)
  • A head full of unclosed <meta or <?xml tags no longer makes the markup scan quadratic, taking it from about 5 ms to about 20 µs. A literal < inside an attribute value now hides that one tag. (Dan Blanchard <https://github.com/dan-blanchard>_ via Claude)
  • An EBCDIC charset= or encoding= followed by whitespace and no value no longer makes the declaration scan backtrack quadratically. A crafted 4 KB document used to cost detect() about 60 ms. (Samina <https://github.com/Samin061>, [#403](https://github.com/chardet/chardet/issues/403) <https://github.com/chardet/chardet/pull/403>)
  • The pure-Python install on CPython scores bigram models with operator.itemgetter instead of a Python loop, making detect() about 1.25x faster and detect_all() about 1.35x faster across the test corpus. Compiled builds and PyPy keep their existing paths. (Afonso Ribeiro <https://github.com/deadstrobe5>_)

... (truncated)

Commits
  • 7a87294 Release 7.7.0
  • 1785c56 docs: label the base64 large-input rows as ascii with a footnote
  • 4ec13a0 test: take the best of three cold starts in the cold-vs-warm check
  • d7be8c7 docs: re-measure the large-input table for 7.7.0
  • d6dcc41 docs: update benchmark numbers for 7.7.0
  • cc5924a Rename EncodingInfo.is_multibyte to is_multibyte_cjk
  • e0a7ff9 docs: derive the multi-byte column from the codec
  • a3699c9 Fix UTF-32 BOM detection for truncated max_bytes windows (#405)
  • 89cab01 Score bigram models through itemgetter on interpreted CPython (#404)
  • ea3ef5f Fix quadratic backtracking in the EBCDIC charset-declaration scan (#403)
  • Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

Updates the requirements on [chardet](https://github.com/chardet/chardet) to permit the latest version.
- [Release notes](https://github.com/chardet/chardet/releases)
- [Changelog](https://github.com/chardet/chardet/blob/main/docs/changelog.rst)
- [Commits](chardet/chardet@7.6.0...7.7.0)

---
updated-dependencies:
- dependency-name: chardet
  dependency-version: 7.7.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
@dependabot @github

dependabot Bot commented on behalf of github Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Labels

The following labels could not be found: dependencies. Please create it before Dependabot can add it to a pull request.

Please fix the above issues or remove invalid values from dependabot.yml.

@laincat
laincat merged commit 22b89a9 into main Oct 26, 2026
7 checks passed
@laincat
laincat deleted the dependabot/pip/chardet-gte-7.7.0 branch October 26, 2026 02:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant