[Performance Regression]: TPCC throughput drops ~77% after 43d361ff (#27713)

[XuPeng-SH]

Description

Commit 43d361ff7710f1d5d9af7c86638a9307f1f3f67e (fix(frontend): specialize direct prepared result metadata, #27713) introduces a severe TPCC throughput regression compared with its direct parent c1fe7b4e596c81a023d5cebd1953f388b5bd8500.

Both runs use the same nightly TPCC configuration:

• 100 warehouses
• 100 terminals
• 5-minute measurement window
• fixed terminal warehouse assignment
• successful job completion

Direct comparison

Revision	tpmC	tpmTOTAL	Job
c1fe7b4e596c81a023d5cebd1953f388b5bd8500	30,098.46	66,735.13	https://github.com/matrixorigin/mo-nightly-regression/actions/runs/33166455405/job/98834333792
43d361ff7710f1d5d9af7c86638a9307f1f3f67e	7,007.01	15,522.74	https://github.com/matrixorigin/mo-nightly-regression/actions/runs/33167546643/job/98837888099

Regression introduced by 43d361ff:

• tpmC: -76.72%
• tpmTOTAL: -76.74%

The workload completes without functional test failure, so this presents as a throughput/CPU-allocation regression rather than a correctness failure.

Suspected root cause

High-confidence code inspection points to execute-time prepared-plan specialization work added to the TPCC hot path:

1. Before 43d361ff, the PREPARE-generation decision needsRuntimeSpecialization was cached and passed to specializePreparedExecutionPlan.
2. 43d361ff repurposes that argument as directResultSpecialization and unconditionally calls PreparedPlanNeedsRuntimeSpecialization(executionPlan) for each execution.
3. PreparedPlanNeedsRuntimeSpecialization performs DeepCopyPlan plus a full plan/expression walk.
4. The subsequent parameter-fill path still performs another plan copy/rebind, and replaceParamVals now also calls PreparedPlanDirectResultParamPositions(plan0) during execution.
5. TPCC repeatedly executes arithmetic prepared DML statements, amplifying these plan copies, walks, allocations, rebinding, compile work, and GC pressure.

The direct-result metadata feature is orthogonal to ordinary TPCC DML. Its admission/tracing should not add per-execution work to that path.

A CPU/alloc pprof from the same nightly environment is still needed to quantify the exact contribution of each new scan/copy, but the direct parent/child benchmark and code-path change establish a strong regression boundary.

Expected behavior

• Direct prepared-result metadata specialization must not change the ordinary prepared DML fast path.
• Static runtime-specialization eligibility must be computed at most once per prepared-plan generation.
• Direct-result parameter positions must be discovered during PREPARE/rebuild and reused, not rediscovered for every execution.
• Stable-type repeated prepared DML must reuse its cached plan/compile whenever semantics permit.

Proposed fix direction

• Split the cached needsRuntimeSpecialization decision and execute-specific directResultSpecialization into separate explicit inputs; do not overload a boolean with two meanings.
• Avoid calling PreparedPlanNeedsRuntimeSpecialization during every execute when the generation-cached result is available.
• Keep direct-result propagation/tracing out of generic DML parameter replacement when the prepared plan has no direct result parameters.
• Add allocation/CPU benchmarks for repeated TPCC-shaped prepared arithmetic UPDATE statements.
• Preserve fix(frontend): specialize direct prepared result metadata #27713 direct SELECT ? metadata correctness, including DECIMAL and binary-protocol unhappy paths.

Acceptance criteria

• On the same TPCC 100/100/5-minute environment, throughput returns to within 5% of the direct-parent result: tpmC >= 28,593 and tpmTOTAL >= 63,398.
• Repeated stable-domain prepared arithmetic DML does not perform a full plan deep-copy/static-specialization scan on every execution.
• Direct prepared-result metadata tests from fix(frontend): specialize direct prepared result metadata #27713 continue to pass.
• Type changes, NULL, invalid/oversized DECIMAL, compile retry, and prepared-plan rebuild paths remain correct.

Note: the parent revision itself is already below the earlier healthy TPCC baseline. This issue isolates the additional ~77% regression introduced specifically by 43d361ff; removing it does not by itself resolve the earlier regression.

[ck89119]

Follow-up after investigating #27816:

The original local benchmark evidence in that PR was not valid against its actual base. The reported 76,489 B/op -> 48,901 B/op improvement compared a revision from before #27556 (8b6a44e83ca), while the PR base already contained #27556.

On the current production path, runtimeSpecializationNeeded is computed once for the prepared-plan generation and passed as true for the TPCC-shaped arithmetic DML case. The base helper then short-circuits before calling PreparedPlanNeedsRuntimeSpecialization, so the PR's forced-false control was not reachable in that state.

I reran the same reachable benchmark on the exact PR base and head:

Revision	ns/op median	B/op	allocs/op
base 52336df805	113,316	48,901	412
head dab9d1a1eb	112,962	48,901	412

The allocation results are identical and the timing difference is noise-level. Therefore, #27816 did not demonstrate an improvement over its base and has been withdrawn and closed.

#27556 already restored the cached-specialization hot path implicated by this issue. However, the issue acceptance criterion has not yet been verified: we still need to run the same TPCC 100 warehouses / 100 terminals / 5-minute workload against a revision containing #27556 and confirm:

• tpmC >= 28,593
• tpmTOTAL >= 63,398

Until that nightly evidence is available, this issue should remain open.