Repository navigation
fix(findrefs): reach non-BMP strings and class names in the locators - #47
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two more places assumed UTF-8 where the dex stores MUTF-8, and Python string order where
type_idsis sorted by UTF-16 code unit. Both fail silently — no error, just no results.1. String search never matched a non-BMP pattern
StringLocator._match_string_offsetencoded its pattern with.encode('utf-8')and regex-matched it against the string-data region, which holds MUTF-8: a code point above the BMP is stored as a surrogate pair (CESU-8), so that pattern can never match.Measured on a fixture whose table contains
Ll/😀;:set(){0}Ll/(ASCII control){0}{0}End to end, with a
const-stringpointing at that string,findrefs string 😀returned 0 hits and now returns 2.2. Precise class lookup missed entries in mixed-BMP tables
MethodLocator/FieldLocator._find_type_idx_preciselybinary-searched the type table comparing Python strings.type_idsis sorted by UTF-16 code unit, where a surrogate pair occupies 0xD800–0xDFFF and so sorts before U+E000 — the opposite of Python's code point order. The search prunes the wrong half and walks past the entry it wants. The trigger is the table's contents, not the query: a plain BMP name misses when an astral neighbour breaks the order.On
['Ll/\U00010000;', 'Ll/\ue000;'], which is that table's correct order, looking up the second entry returned -1 whileDEX.get_classfound it.Comparing in the table's own order fixes it without a fallback, since
utf-16bebyte order is UTF-16 code-unit order.surrogatepassbecause a malformed dex can decode to lone surrogates, which plain utf-16 encoding rejects — without it a broken APK would crash the search rather than simply not match.I did not copy
tinydex.get_type's linear fallback: that runs once per class lookup, where a full scan is affordable, but this runs once per dex, and a miss is the common case in a multi-dex search — a fallback would mean a full type-table decode scan for every dex that does not contain the class.Scope
Below the BMP both encodings and both orders agree, so ASCII and BMP lookups are byte-for-byte unchanged; I checked each against its old form before switching. These live in
asc_core, so the CLI, GUI and MCP all inherit them.Tests
test_references.py→StringTests.test_non_bmp_string_is_searchabletest_source_edit.py→test_precise_class_lookup_survives_utf16_code_unit_orderBoth were verified to fail with the fixes reverted, and pass with them. Full suite: 109 tests (was 107), OK.
One note for anyone extending these: both fixtures are built in dex order (sorted by
utf-16be). An unsorted type table makes the binary search return arbitrary entries, so a test built carelessly can pass or fail for the wrong reason.