Skip to content

fix(nodejs): prevent cursor-file corruption and survive it at startup - #108

Closed
tangquanwei wants to merge 1 commit into
corespeed-io:mainfrom
tangquanwei:fix/robust-state-and-transport
Closed

tangquanwei wants to merge 1 commit into
corespeed-io:mainfrom
tangquanwei:fix/robust-state-and-transport

Conversation

@tangquanwei

Copy link
Copy Markdown

Problem

A single corrupt state file could permanently kill the bot.

Symptom

✗ Poll error: Unexpected token '', ""... is not valid JSON

After this, the bridge stops receiving and replying, sets connected = false, and never recovers.

Root cause

~/.wechatbot/cursor.json can end up zero-filled (or otherwise truncated). FileStorage.get() did a bare JSON.parse(raw) and rethrew on failure, so client.start() rejected inside poller.loadCursor() before the poll loop ever began. Three independent defects combined:

  1. Corruption is produced — FileStorage.set() used a plain writeFile: non-atomic, no fsync, and the poller fires it without awaiting, so overlapping writes to the same path interleave.
  2. The failure is amplified — FileStorage.get() rethrew parse errors instead of tolerating a damaged file.
  3. No recovery — the pi-agent extension did bot.start().catch(...) and went permanently silent.

Reproduction (old vs new)

Stress test: same path, 8 concurrent set() calls, 3000 rounds, then JSON.parse the resulting file:

OLD (non-atomic, unawaited): 1906/3000 rounds wrote an invalid-JSON file  { INVALID_JSON: 1906 }
NEW (atomic + serialized)  :    0/3000 rounds wrote an invalid-JSON file  {}

A zero-filled file (e.g. 107 NUL bytes) specifically matches an unflushed write: the filesystem records the file size in metadata but the data is lost, so reads come back as zeros.

Changes

nodejs/src/storage/file.ts — stop producing corruption, and survive it anyway

  • Atomic writes: temp file → fsync → rename.
  • Serialize writes per key; delete/clear wait for in-flight writes.
  • Reads tolerate a UTF-8 BOM; corrupt files are quarantined (*.corrupt-<ts>) and treated as missing instead of throwing.

nodejs/src/transport/http.ts, core/errors.ts — transport robustness

  • Parse BOM / empty / null / non-JSON bodies safely. Non-JSON responses become a typed, retryable TransportError (5xx/429 retried, 4xx not).
  • Network failures are now retryable as well.

nodejs/src/messaging/ — never crash on bad state

  • loadCursor() and ContextStore.load() never throw.
  • Log cursor persistence failures; skip redundant cursor writes.

nodejs/src/core/client.ts — expose resetCursor() for recovery.

pi-agent — auto-recover from a crashed poll loop (reset cursor + capped backoff) instead of going permanently silent.

nodejs/package.json — npm test was node node_modules/.bin/vitest run, which fails on Windows (the .bin entry is a shell script). Now vitest run.

Tests

  • tests/file-storage.test.ts — atomicity, concurrent writes, BOM, NUL/corrupt quarantine, delete-vs-write race.
  • tests/http.test.ts — BOM/empty/null/non-JSON bodies, retry classification.
  • tests/poller.test.ts — cursor load/persist resilience.

77 → 102 tests, all passing. tsc --noEmit clean.

Notes

  • No public API removed; resetCursor() is additive.
  • The npm test change is exactly what the CI npm test step runs, so CI stays green.

A crash or overlapping write to the persisted cursor file could leave it
zero-filled. FileStorage.get then rethrew the JSON.parse error from
client.start(), which aborted the poll loop before it began — surfaced as
"Poll error: Unexpected token ... is not valid JSON".

- storage/file: atomic writes (temp + fsync + rename), serialize writes per
  key, tolerate a UTF-8 BOM, and quarantine corrupt files instead of throwing
- transport/http: parse BOM/empty/non-JSON bodies safely; non-JSON responses
  become typed TransportErrors that are retried for 5xx/429 but not 4xx;
  network failures are now retryable
- messaging: loadCursor/contextStore.load never throw; log cursor persist
  failures; skip redundant cursor writes
- core/client: expose resetCursor() for recovery
- pi-agent: auto-recover from a crashed poll loop (reset cursor + capped
  backoff) instead of going permanently silent
- add tests for file storage, HTTP parsing, and poller cursor handling
Copilot AI lite review requested due to automatic review settings September 16, 2026 14:17

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@tangquanwei

Copy link
Copy Markdown
Author

Superseded by a smaller, focused PR that fixes the same root cause: #109 (storage atomicity + corruption tolerance). Closing this to keep review scope tight; the transport/poller/pi-agent hardening can be split out separately if useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants