Skip to content

fix: Read files in raw mode so flobs preserve every byte - #25

Open
joethorley wants to merge 2 commits into
mainfrom
fix-raw-flob
Open

joethorley wants to merge 2 commits into
mainfrom
fix-raw-flob

Conversation

@joethorley

Copy link
Copy Markdown
Member

Closes #24

Problem

flob() read files with readBin(what = "integer", n = size), which:

  • dropped the trailing 1-3 bytes of any file whose size is not a multiple of 4, corrupting formats such as .tar.gz and .xlsx
  • returned NA_integer_ for any 4-byte-aligned chunk equal to 00 00 00 80, so vld_flob() returned FALSE and unflob() errored depending on file content

Changes

  • flob() reads files with readBin(what = "raw").
  • vld_exint() / chk_exint() accept raw or integer elements, so legacy integer flobs (e.g. those already stored in databases) remain valid and unflob.
  • The not-NA check is dropped, since NA is a legitimate bit pattern in legacy integer flobs. This also removes the call to the deprecated chk::chk_no_missing().
  • flob_obj is regenerated as a raw flob. The internal flob_old, flob_noname and slob_obj objects stay integer-based to keep legacy coverage.
  • The DESCRIPTION, README and ?flob descriptions now say raw-mode instead of integer-mode.
  • Tests added for byte-for-byte round trips of 0-9 byte files, the NA bit pattern, legacy integer flobs containing NA, and the new chk_flob() error message.

Notes

  • Bytes already lost from existing integer flobs cannot be recovered.
  • dbflobr has its own copy of vld_exint() / vld_slob() (R/vld.R) that requires integer elements. It will need the same change to accept raw flobs through its slob path.
  • The remaining test failures in test-chk.R, test-ext.R, test-flobr.R, test-path.R and test-tools.R also occur on main. They come from newer chk error messages and tools::file_ext() behaviour (CI: file-extension test failures on main (pre-existing) #19).
  • The issue's tarball example now round-trips with an identical md5sum.

🤖 Generated with Claude Code

`flob()` read files with `readBin(what = "integer")`, which dropped the
trailing 1-3 bytes of files whose size is not a multiple of 4 and
produced NA elements for the bit pattern 0x80000000, causing
`vld_flob()` and `unflob()` to reject the flob.

Flobs are now read as raw vectors. Legacy integer flobs remain valid,
including those containing NA values.

Closes #24

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

flob() drops last 1-3 bytes of files when size isnt multiple of 4

1 participant