[#1071] Wrap the sources of the large values PDBStorageTest writes, so that four copies of a 63 MB value are not live at once - #1072
Open
vharseko wants to merge 1 commit into
Conversation
…torageTest writes, so that four copies of a 63 MB value are not live at once testCanAddLargeValues built each source with ByteString.valueOfBytes(), which copies, and a value on its way into Persistit is copied twice more - toByteArray() in bytesToValue() and the value buffer of the exchange, which doubles up to 64 MB. With the 63 MB value that is four copies live at once on top of the buffer pool and the server: about 310 MB after a collection in the 512 MB of the test JVM, and build-maven (ubuntu-latest, 17) of OpenIdentityPlatform#1069 ran out of heap on the 32 MB put. Wrapping the sources leaves about 165 MB. Measured under JDK 17 with -Xlog:gc*: the worst pause goes from 436M->311M with 300->177 humongous regions to 471M->165M with 328->31. Splitting the three puts over three writes does not help - the value buffer grows again for each - so they stay in one transaction.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1071.
PDBStorageTest.testCanAddLargeValueswrites values of 4, 32 and 63 MB, andbuild-maven (ubuntu-latest, 17)of #1069 (run 35304975160) ran the 512 MB of the test JVM out on the 32 MB put, inValue.ensureFit()- a PR which touches nothing near the test, in a fork of its own (reuseForks=false), so the margin spent was the test's.Why
Each value was copied four times on its way into Persistit: the source array,
ByteString.valueOfBytes()(a copy),toByteArray()inPDBStorage.bytesToValue()(another), and the value buffer of the exchange, which doubles up to 64 MB. With the 63 MB value that is about 250 MB of humongous arrays live at once, on top of the buffer pool of the test storage (76 MB, 20% of the quota) and the test server.The change
Test only: the three sources are
ByteString.wrap()ped rather than copied, which takes one copy per value out. The three puts stay in one transaction on purpose - the value buffer the 32 MB one grows to fits the 63 MB one without growing again - and the javadoc says so.Measured
JDK 17.0.20,
-Xmx512m -Xlog:gc*- the settings of the leg - worst pause and humongous regions, before -> after the collection:The split over three writes was measured first and rejected: a fresh exchange grows its value buffer again for each put, and the live set is no smaller.
PDBStorageTestis 10/10 on this head, under JDK 17 and JDK 26.Left out
The two copies in main -
valueOfBytes()callers elsewhere andtoByteArray()inbytesToValue()- are as they were; aByteSequencegives no zero-copy access to its backing array, and this is the test's margin to fix. Raising the-Xmx512mof the failsafe fork in the rootpom.xmlwould widen the margin for every class, and is a change of its own.