fix(windows): read search_code temp files as UTF-8#956
Merged
DeusData merged 2 commits intoJul 9, 2026
Conversation
1e7f518 to
8695f3c
Compare
Owner
|
Thanks for covering the Windows UTF-8 temp-file path. Triage: Windows query/read bug, high priority because it affects non-ASCII projects after indexing. Review focus is PowerShell 5.1 encoding behavior, CJK path coverage, and no regression to POSIX temp-file reads. |
ac9ed81 to
c5cbbcf
Compare
Signed-off-by: Cosm1cAC <cosm1cac@github.com>
c5cbbcf to
9971fab
Compare
Signed-off-by: Cosm1cAC <87748355+Cosm1cAC@users.noreply.github.com>
e4d4198 to
8e0c29a
Compare
Owner
|
Merged — thank you! Surgical fix with the right PS 5.1 root cause (BOM-less UTF-8 decoded as ANSI), and the test locks it in. The Windows non-ASCII path class keeps biting; this closes another corner of it. Much appreciated. |
27 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
search_codecan miss matches on Windows when the indexed project is under a non-ASCII path, such as a CJK directory name.Scoped search writes two temporary files: one for the pattern and one for the indexed file list. Both files are UTF-8. The Windows command builder was reading them back with plain
Get-Contentbefore handing each path toSelect-String -LiteralPath.That is fragile on Windows PowerShell 5.1. UTF-8 without a BOM is decoded with the local ANSI code page unless the encoding is explicit. A path like
C:/Users/.../移动版agent终端/src/file.jsis read back as mojibake, soSelect-Stringreceives a literal path that does not exist andsearch_codereturns a clean zero-match result.This patch makes the Windows
search_codecommand builder read the pattern file and scoped file list withGet-Content -Encoding UTF8 -LiteralPath.Related to #903.
Reproduction
I reproduced this on Windows PowerShell 5.1 with a project rooted under:
C:/Users/<user>/Documents/移动版agent终端and a source file containing
agentReachCommandForAction.Before the fix:
Select-String -LiteralPath <actual file> -Pattern agentReachCommandForActionfinds the expected lines.search_codepipeline reads the UTF-8 file list through plainGet-Content.Test-Path -LiteralPath <decoded path>is false.search_codereturns zero grep matches.After the fix:
Get-Content -Encoding UTF8 -LiteralPath.Select-Stringfinds the expected matches.Verification
Local command-level check on Windows PowerShell 5.1:
current_matches=0patched_matches=2FalseTrueAlso ran:
git diff --checkI added a Windows-only regression test that creates a scoped
search_codefixture under a UTF-8 CJK project root.I could not run the full native Windows C test suite locally because this machine does not have the MSYS2/CLANG64 GNU Make toolchain used by the Windows CI job. Visual Studio
clis installed, but that is not this project's Windows CI build path.Checklist
git commit -s)make -f Makefile.cbm test)make -f Makefile.cbm lint-ci)