feat(read): extract PDF text via pdftotext instead of base64 dump (#236) - #283
Open
innocarpe wants to merge 1 commit into
Open
feat(read): extract PDF text via pdftotext instead of base64 dump (#236)#283innocarpe wants to merge 1 commit into
innocarpe wants to merge 1 commit into
Conversation
…ssweb#236) When pdftotext is available, the read tool returns extracted text (capped at 30K chars) so the model can actually read PDF contents. Falls back to the binary warning when unavailable. (cherry picked from commit e94ededbd465931747e2b4db40d0eab43c270782)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
readtool now extracts PDF text via the externalpdftotextbinary when available, instead of returning onlyWARNING: File is binary.with the raw base64.Why
Issue #236: reading a PDF dumps the whole binary as base64 into the conversation and returns
WARNING: File is binary.— useless for the model and wasteful of context. With text extraction, a PDF becomes a normal readable document.Changes
packages/core/src/tools/read-handler.ts: runpdftotext -layout <file> -viaspawnSync(10s timeout, 10MB buffer); return extracted text (capped atMAX_PDF_TEXT_CHARS = 30000) withencoding: "text"; fall back to the existing binary warning whenpdftotextis unavailable or returns nothing.packages/core/src/tests/read-handler.test.ts: tests for the extraction path (fakepdftotextshim on PATH) and the fallback path (empty PATH).Validation
npm run typecheck✅npm test— new read-handler tests pass (2/2) ✅Closes #236