Skip to content

[#1071] Wrap the sources of the large values PDBStorageTest writes, so that four copies of a 63 MB value are not live at once - #1072

Open
vharseko wants to merge 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:tests/pdb-large-values-peak
Open

vharseko wants to merge 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:tests/pdb-large-values-peak

Conversation

@vharseko

Copy link
Copy Markdown
Member

Fixes #1071.

PDBStorageTest.testCanAddLargeValues writes values of 4, 32 and 63 MB, and build-maven (ubuntu-latest, 17) of #1069 (run 35304975160) ran the 512 MB of the test JVM out on the 32 MB put, in Value.ensureFit() - a PR which touches nothing near the test, in a fork of its own (reuseForks=false), so the margin spent was the test's.

Why

Each value was copied four times on its way into Persistit: the source array, ByteString.valueOfBytes() (a copy), toByteArray() in PDBStorage.bytesToValue() (another), and the value buffer of the exchange, which doubles up to 64 MB. With the 63 MB value that is about 250 MB of humongous arrays live at once, on top of the buffer pool of the test storage (76 MB, 20% of the quota) and the test server.

The change

Test only: the three sources are ByteString.wrap()ped rather than copied, which takes one copy per value out. The three puts stay in one transaction on purpose - the value buffer the 32 MB one grows to fits the 63 MB one without growing again - and the javadoc says so.

Measured

JDK 17.0.20, -Xmx512m -Xlog:gc* - the settings of the leg - worst pause and humongous regions, before -> after the collection:

variant worst pause humongous regions
master 436M -> 311M 300 -> 177
the three puts in three writes 452M -> 324M 302 -> 112
this PR 471M -> 165M 328 -> 31

The split over three writes was measured first and rejected: a fresh exchange grows its value buffer again for each put, and the live set is no smaller. PDBStorageTest is 10/10 on this head, under JDK 17 and JDK 26.

Left out

The two copies in main - valueOfBytes() callers elsewhere and toByteArray() in bytesToValue() - are as they were; a ByteSequence gives no zero-copy access to its backing array, and this is the test's margin to fix. Raising the -Xmx512m of the failsafe fork in the root pom.xml would widen the margin for every class, and is a change of its own.

…torageTest writes, so that four copies of a 63 MB value are not live at once

testCanAddLargeValues built each source with ByteString.valueOfBytes(), which copies, and a
value on its way into Persistit is copied twice more - toByteArray() in bytesToValue() and
the value buffer of the exchange, which doubles up to 64 MB. With the 63 MB value that is
four copies live at once on top of the buffer pool and the server: about 310 MB after a
collection in the 512 MB of the test JVM, and build-maven (ubuntu-latest, 17) of OpenIdentityPlatform#1069 ran
out of heap on the 32 MB put. Wrapping the sources leaves about 165 MB.

Measured under JDK 17 with -Xlog:gc*: the worst pause goes from 436M->311M with 300->177
humongous regions to 471M->165M with 328->31. Splitting the three puts over three writes
does not help - the value buffer grows again for each - so they stay in one transaction.
@vharseko vharseko added bug tests Test suites: fixing, enabling, un-disabling java Changes to Java sources labels Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug java Changes to Java sources tests Test suites: fixing, enabling, un-disabling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PDBStorageTest.testCanAddLargeValues runs the 512 MB test JVM out of heap: four copies of a 63 MB value live at once

1 participant