[VL] Fix broadcast hash table reuse for reused exchanges

[wecharyu]

What changes are proposed in this pull request?

Create unique hash table ID with plan_id, build_keys, join_type and null_aware, which determine the hash table contents.

How was this patch tested?

Add UT.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex GPT-5.5

[github-actions]

Run Gluten Clickhouse CI on x86

[liujiayi771]

Is this part still needed when buildHashTableOncePerExecutor is disabled?

[wecharyu]

doCanonicalizeForBroadcastMode() is not invoked anywhere, broadcast canonicalization goes through ColumnarBroadcastExchangeExec.doCanonicalize().

[liujiayi771]

This code is actually useful — when buildHashTableOncePerExecutor is disabled, it provides more opportunities to reuse broadcast exchanges. Moreover, I now think that even when buildHashTableOncePerExecutor is enabled, the comment in doCanonicalizeForBroadcastMode still holds true: we still broadcast byte arrays and build HashRelation at the executor side.
@JkSelf Can you explain why this was removed? This allows us to reuse broadcast exchanges for different build keys with the same data.
We should either restore the code before ColumnarBroadcastExchangeExec.doCanonicalize, or at least follow the original logic when buildHashTableOncePerExecutor is disabled.

[JkSelf]

Replied in https://github.com/apache/gluten/pull/8931/changes#r3385848526.

[liujiayi771]

@JkSelf Thank you for the explanation, I understand now. @wecharyu According to the instructions here, you can restore the original behavior of doCanonicalizeForBroadcastMode when enableBroadcastBuildOncePerExecutor=false.

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[JkSelf]

@wecharyu Thanks for your work.

For ColumnarBroadcastExchangeExec, reuse is already decided by doCanonicalize(). It now directly uses mode.canonicalized, and that canonicalized BroadcastMode already captures the key BHJ semantics such as bound build keys and null-aware behavior. So we no longer need this handling for the unique broadcastId.

[wecharyu]

@JkSelf Thanks for the prompt response. HashedRelationBroadcastMode does not contains joinType, which could also change the build hash table data. For example in this PR's test the LEFT SEMI JOIN and INNER JOIN would use the same hash table, which cause the test failed before this PR. https://github.com/apache/spark/blob/62ae4db28f3be8a0ca2c3016d27ca5a62f02915d/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala#L1152-L1153

And since current Gluten is still broadcast the raw build plan data bytes instead of hash table,I agree with @liujiayi771 that BroadcastExchangeExec can be reused for the same build plan even when the build keys differ. This reuse avoids broadcasting the same data multiple times.

I think now we can make following improvements:

1. Move doCanonicalizeForBroadcastMode() back to reuse the broadcast exchange as much as possible.
2. Associate the constructed hash table with the plan_id, build_keys, drop_duplicates, and null_aware flag, as these attributes uniquely identify the hash table.

[JkSelf]

HashedRelationBroadcastMode does not contains joinType, which could also change the build hash table data. For example in this PR's test the LEFT SEMI JOIN and INNER JOIN would use the same hash table, which cause the test failed before this PR.

Gluten's implementation is the same as vanilla Spark's. Just out of curiosity, can Spark pass this failed test?