Skip to content

fix(findrefs): reach non-BMP strings and class names in the locators - #47

Merged
MG1937 merged 1 commit into
MG1937:dev-0.1.0from
dhtzs:fix/mutf8-locators
Oct 12, 2026
Merged

MG1937 merged 1 commit into
MG1937:dev-0.1.0from
dhtzs:fix/mutf8-locators

Conversation

@dhtzs

@dhtzs dhtzs commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

Two more places assumed UTF-8 where the dex stores MUTF-8, and Python string order where type_ids is sorted by UTF-16 code unit. Both fail silently — no error, just no results.

1. String search never matched a non-BMP pattern

StringLocator._match_string_offset encoded its pattern with .encode('utf-8') and regex-matched it against the string-data region, which holds MUTF-8: a code point above the BMP is stored as a surrogate pair (CESU-8), so that pattern can never match.

Measured on a fixture whose table contains Ll/😀;:

query before after
the emoji set() {0}
Ll/ (ASCII control) {0} {0}

End to end, with a const-string pointing at that string, findrefs string 😀 returned 0 hits and now returns 2.

2. Precise class lookup missed entries in mixed-BMP tables

MethodLocator / FieldLocator._find_type_idx_precisely binary-searched the type table comparing Python strings. type_ids is sorted by UTF-16 code unit, where a surrogate pair occupies 0xD800–0xDFFF and so sorts before U+E000 — the opposite of Python's code point order. The search prunes the wrong half and walks past the entry it wants. The trigger is the table's contents, not the query: a plain BMP name misses when an astral neighbour breaks the order.

On ['Ll/\U00010000;', 'Ll/\ue000;'], which is that table's correct order, looking up the second entry returned -1 while DEX.get_class found it.

Comparing in the table's own order fixes it without a fallback, since utf-16be byte order is UTF-16 code-unit order. surrogatepass because a malformed dex can decode to lone surrogates, which plain utf-16 encoding rejects — without it a broken APK would crash the search rather than simply not match.

I did not copy tinydex.get_type's linear fallback: that runs once per class lookup, where a full scan is affordable, but this runs once per dex, and a miss is the common case in a multi-dex search — a fallback would mean a full type-table decode scan for every dex that does not contain the class.

Scope

Below the BMP both encodings and both orders agree, so ASCII and BMP lookups are byte-for-byte unchanged; I checked each against its old form before switching. These live in asc_core, so the CLI, GUI and MCP all inherit them.

Tests

  • test_references.py → StringTests.test_non_bmp_string_is_searchable
  • test_source_edit.py → test_precise_class_lookup_survives_utf16_code_unit_order

Both were verified to fail with the fixes reverted, and pass with them. Full suite: 109 tests (was 107), OK.

One note for anyone extending these: both fixtures are built in dex order (sorted by utf-16be). An unsorted type table makes the binary search return arbitrary entries, so a test built carelessly can pass or fail for the wrong reason.

Copilot AI balanced review requested due to automatic review settings October 11, 2026 16:08

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@MG1937
MG1937 merged commit 3092cd1 into MG1937:dev-0.1.0 Oct 12, 2026
2 checks passed
@dhtzs
dhtzs deleted the fix/mutf8-locators branch October 12, 2026 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants