[#1068] Apply what a running backend takes of a configuration change, and ask for a restart for what it does not

[vharseko]

Fixes #1068. Stacked on #1066 (64f3cffd00, on master since #994 merged): the same lines of both applyConfigurationChange, both XMLs and the ordinal after its 630; JEStorageTest exists only from #999 on. Merge after #1066.

What was wrong

Nine properties of the JE and PDB backends were neither applied to a running backend nor marked as requiring a restart, so dsconfig reported a change of them applied and the backend ran on unchanged until it was next opened. JEStorage.applyConfigurationChange handled db-directory, its permissions and the disk thresholds and left the environment - configured once, at the open - as it was; nothing in the server called Environment.setMutableConfig. The same went for a native property changed through je-property, mutable or not (not in the issue's table, same cause).

Two things the issue had wrong, found on the way:

• a property marked requires-admin-action in the XML reaches the reference documentation and property help alone - dsconfig prints nothing about it at set time, and a change result without adminActionRequired carries nothing to the error log either. So "as db-log-file-max already does" was not true of the change result: the marking is all those properties had. The old RootContainer reported every changed immutable parameter in the change result, which the server logs as a warning (WARN 647); that shape is restored for all of them.
• db-txn-write-no-sync is on by default, so dsconfig set-backend-prop --set db-txn-no-sync:true on a JE backend yields a durability which sets both flags. Nothing checked that at change time: the change was admitted and written, and the backend failed to open on ERR_CONFIG_JEB_DURABILITY_CONFLICT at its next restart. (aChangeWhichLeavesTheCacheSizeAloneAsksForNothing of [#1063] Give back what the open reserved rather than what the configuration says by then, and ask for a restart when the cache size changes #1066 used exactly that as its "unrelated change"; it now sets the write flag off as well.)

What this does

JE - applyToEnvironment builds the environment configuration the changed configuration describes (ConfigurableEnvironment.toEnvironmentConfig) and hands it to Environment.setMutableConfig, which takes of it what JE accepts while it runs: db-cleaner-min-utilization, db-run-cleaner, db-evictor-core-threads, db-evictor-max-threads, db-evictor-keep-alive, db-num-cleaner-threads, the durability (db-txn-no-sync / db-txn-write-no-sync, every way - a configuration which sets neither now sets COMMIT_SYNC explicitly, since JE leaves the durability an environment has in place when handed none) and a mutable je-property. All but the cache: je.maxMemory / je.maxMemoryPercent are mutable too, but the cache stays with the memory reserved for it until the restart #1063 asks for, so the change hands the environment its current values back. Every immutable JE parameter whose value differs from the running environment's is reported with the new NOTE_CONFIG_DB_PROPERTY_REQUIRES_RESTART (631), naming the property as dsconfig knows it (or the JE property name for je-property), the value the environment runs with and the one configured. An import's environment is left alone - it runs on a configuration of its own and the backend opens again on the changed one once the import is over; held to the import's configuration, every property the import sets differently would ask for a restart.

toEnvironmentConfig is the build alone - no cache size probe against the memory quota (#1067), no level set on the JE loggers (that moves to parseConfigEntry, the open's road) - so that a change can be checked against it: isConfigurationChangeAcceptable and isConfigurationAcceptable now refuse a conflicting durability and a native property JE does not know before the change is written.

PDB - db-checkpointer-wakeup-interval is set on the PersistIt configuration at the open alone and Persistit.setConfiguration refuses once one is set, so a change of it reports 631 against the interval the database opened with (db.getConfiguration().getCheckpointInterval(), no new field), and the property is marked component-restart in PDBBackendConfiguration.xml. je-property's definition says which of its changes wait for a restart.

Left as they are: db-logging-level and db-logging-file-handler-on keep their component-restart marking. The JUL level is set by the open alone, as before; the file handler's level (je.env.fileLoggingLevel) is mutable in JE and so follows a change from now on - the marking is conservative about it, not wrong.

Tests

JEStorageTest: the six mapped properties reach env.getMutableConfig(); the durability follows the change every way; a mutable je-property is applied and an immutable one asks for a restart with 631; db-log-file-max asks for a restart with 631; a change while open leaves the cache where the open reserved it (the #1063 interplay); a change during an import leaves the import's environment alone; a change while closed touches nothing; a durability which sets both flags and an unknown native property are refused by both acceptability checks. PDBStorageTest: a changed interval asks for a restart with 631 and the database keeps its own; a change while closed asks for nothing.

Verified locally: the issue reproduced on master, on the #999 head and on the #1066 head with a direct TestNG repro of every row (JE 7/7 red, PDB 1/1 red, db-txn-no-sync on PDB green as a control); the new tests are red without the fix (5 JE + 1 PDB) and green with it; mutants (no setMutableConfig, cache not pinned, immutables unreported, import's environment held to the configuration, PDB silent) each red on their own case; regression set of 18 classes / 240 tests (FailedBackendOpenTest, PDB/JE TestCase + Encrypted, ReplayedConfigChangeTest, OnDiskMergeImporterTest, the pluggable tree tests, ImportLDIF/RebuildIndex/VerifyIndex, BackendConfigManagerTestCase) green.

[vharseko]

Rebased onto the restacked #1066 (725ff9af7d, itself on master now that #999 is merged); the commit is unchanged - git range-diff =. Head is 4bc401fec2; the first line of the description names the new base.

Re-run green on the rebased head: PDBStorageTest (22), JEStorageTest (17), FailedBackendOpenTest (8), JETestCase / PDBTestCase (35 each).

[vharseko]

Restacked on the new head of #1066 (64f3cffd00, itself rebased onto master now that #994 is merged): 4bc401fec2 → 4ea1d18893. No conflict of its own — the same commit over the moved base; the conflict #994 caused was in #1066's tail of backend.properties, and 630-631 now follow #994's 624-627. JEStorageTest and PDBStorageTest on the new head: 39/39 green. Description updated (base SHA).