Skip to content

Speed up reading packages and fix some bugs - #37

Open
w1tcherrr wants to merge 10 commits into
ValveResourceFormat:masterfrom
w1tcherrr:perf
Open

w1tcherrr wants to merge 10 commits into
ValveResourceFormat:masterfrom
w1tcherrr:perf

Conversation

@w1tcherrr

Copy link
Copy Markdown

Changes

  • Archive files are opened and memory mapped once per package. GetMemoryMappedStreamIfPossible returns a lightweight stream over that mapping. ReadEntry reads with RandomAccess from the cached handles.
  • Both are now thread-safe.
  • The directory tree is parsed from a single buffer. Sorting is unchanged.
  • Verification reads into reused buffers.

Benchmark

Benchmark Unit Before After
Open CS2 pak01_dir.vpk ms 50.6 27.9
Open Dota 2 pak01_dir.vpk ms 210.6 101.7
Mapped stream + read, entries up to 4 KiB us 36.70 1.13
Mapped stream + read, entries of 16-256 KiB us 44.10 3.33
Mapped stream + read, entries of 4 MiB or more us 3137 321
ReadEntry, entries up to 4 KiB us 37.4 3.7
Mapped reads on 16 threads, entries up to 4 KiB reads/s 25,300 7,890,000
VerifyChunkHashes, CS2 archives 0-5 ms 1358 626
VRF loading 7,397 CS2 resources, 1 thread ms 544 320
VRF loading 7,397 CS2 resources, 16 threads ms 283 203

Fixes

  • Concurrent ReadEntry calls could return wrong data.
  • Entries cut off by the end of an archive were returned zero-filled. They now throw EndOfStreamException.
  • Calling Read again kept the previous package's signature and checksums, and IsDirVPK never reset in SetFileName.
  • AddFile on a optimized package appended unsorted, so FindEntry missed the file.
  • Read(Stream) and Write(Stream) broke when the package did not start at position 0, and writing into a longer stream hashed the leftover bytes.
  • Malformed signature and archive MD5 section sizes now throw InvalidDataException.

Behaviour changes

  • GetMemoryMappedStreamIfPossible returns an internal stream type instead of MemoryMappedViewStream. Returned streams must not be disposed while another thread reads them.
  • Archive files stay open and mapped until the package is disposed, and chunk files use FileShare.ReadWrite.
  • ReadEntry, GetMemoryMappedStreamIfPossible and the Verify* methods throw ObjectDisposedException after Dispose.

I ran this PR over hundreds of different game paks and they read identical to before.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant