Skip to content

fix(windows): read search_code temp files as UTF-8#956

Merged
DeusData merged 2 commits into
DeusData:mainfrom
Cosm1cAC:fix/windows-search-code-utf8-paths
Jul 9, 2026
Merged

fix(windows): read search_code temp files as UTF-8#956
DeusData merged 2 commits into
DeusData:mainfrom
Cosm1cAC:fix/windows-search-code-utf8-paths

Conversation

@Cosm1cAC

@Cosm1cAC Cosm1cAC commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Summary

search_code can miss matches on Windows when the indexed project is under a non-ASCII path, such as a CJK directory name.

Scoped search writes two temporary files: one for the pattern and one for the indexed file list. Both files are UTF-8. The Windows command builder was reading them back with plain Get-Content before handing each path to Select-String -LiteralPath.

That is fragile on Windows PowerShell 5.1. UTF-8 without a BOM is decoded with the local ANSI code page unless the encoding is explicit. A path like C:/Users/.../移动版agent终端/src/file.js is read back as mojibake, so Select-String receives a literal path that does not exist and search_code returns a clean zero-match result.

This patch makes the Windows search_code command builder read the pattern file and scoped file list with Get-Content -Encoding UTF8 -LiteralPath.

Related to #903.

Reproduction

I reproduced this on Windows PowerShell 5.1 with a project rooted under:

C:/Users/<user>/Documents/移动版agent终端

and a source file containing agentReachCommandForAction.

Before the fix:

  • Select-String -LiteralPath <actual file> -Pattern agentReachCommandForAction finds the expected lines.
  • The scoped search_code pipeline reads the UTF-8 file list through plain Get-Content.
  • The CJK path is decoded as mojibake, so Test-Path -LiteralPath <decoded path> is false.
  • search_code returns zero grep matches.

After the fix:

  • The same file list is read with Get-Content -Encoding UTF8 -LiteralPath.
  • The decoded path exists.
  • Select-String finds the expected matches.

Verification

Local command-level check on Windows PowerShell 5.1:

  • current pipeline: current_matches=0
  • patched pipeline: patched_matches=2
  • default-decoded filelist path exists: False
  • UTF-8-decoded filelist path exists: True

Also ran:

  • git diff --check

I added a Windows-only regression test that creates a scoped search_code fixture under a UTF-8 CJK project root.

I could not run the full native Windows C test suite locally because this machine does not have the MSYS2/CLANG64 GNU Make toolchain used by the Windows CI job. Visual Studio cl is installed, but that is not this project's Windows CI build path.

Checklist

  • Every commit is signed off (git commit -s)
  • Tests pass locally (make -f Makefile.cbm test)
  • Lint passes (make -f Makefile.cbm lint-ci)
  • New behavior is covered by a test (reproduce-first for bug fixes)

@Cosm1cAC Cosm1cAC requested a review from DeusData as a code owner July 8, 2026 11:41
@Cosm1cAC Cosm1cAC force-pushed the fix/windows-search-code-utf8-paths branch from 1e7f518 to 8695f3c Compare July 8, 2026 11:58
@DeusData DeusData added bug Something isn't working windows Windows-specific issues ux/behavior Display bugs, docs, adoption UX priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker. labels Jul 8, 2026
@DeusData

DeusData commented Jul 8, 2026

Copy link
Copy Markdown
Owner

Thanks for covering the Windows UTF-8 temp-file path. Triage: Windows query/read bug, high priority because it affects non-ASCII projects after indexing. Review focus is PowerShell 5.1 encoding behavior, CJK path coverage, and no regression to POSIX temp-file reads.

@DeusData DeusData added this to the 0.9.1-rc milestone Jul 8, 2026
@Cosm1cAC Cosm1cAC force-pushed the fix/windows-search-code-utf8-paths branch from ac9ed81 to c5cbbcf Compare July 8, 2026 14:36
Signed-off-by: Cosm1cAC <cosm1cac@github.com>
@Cosm1cAC Cosm1cAC force-pushed the fix/windows-search-code-utf8-paths branch from c5cbbcf to 9971fab Compare July 8, 2026 14:43
Signed-off-by: Cosm1cAC <87748355+Cosm1cAC@users.noreply.github.com>
@Cosm1cAC Cosm1cAC force-pushed the fix/windows-search-code-utf8-paths branch from e4d4198 to 8e0c29a Compare July 8, 2026 15:10
@DeusData DeusData merged commit a70d39d into DeusData:main Jul 9, 2026
19 checks passed
@DeusData

DeusData commented Jul 9, 2026

Copy link
Copy Markdown
Owner

Merged — thank you! Surgical fix with the right PS 5.1 root cause (BOM-less UTF-8 decoded as ANSI), and the test locks it in. The Windows non-ASCII path class keeps biting; this closes another corner of it. Much appreciated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker. ux/behavior Display bugs, docs, adoption UX windows Windows-specific issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants