fix: restrict array_filter array_compact fast path to the lambda variable

[andygrove]

Which issue does this PR close?

Closes #4830.

Rationale for this change

CometArrayFilter.convert has a fast path that lowers filter(arr, x -> x IS NOT NULL) to the native array_compact serde, to avoid the per-batch JNI cost of the codegen dispatcher. The guard only checked that the lambda body is an IsNotNull, not that the IsNotNull operand is the lambda's element variable.

As a result, a lambda whose body is IsNotNull of something other than the element (for example a captured column, x -> c IS NOT NULL) was incorrectly treated as array_compact. Spark keeps every element of arr when c is non-null and returns an empty array when c is null; the fast path instead dropped the null elements of arr regardless of c. This produced silently wrong results whenever arr contained null elements or the captured column was null.

What changes are included in this PR?

Tighten the guard in CometArrayFilter.convert to fire the array_compact fast path only when the IsNotNull operand is the lambda's own NamedLambdaVariable (matched by exprId, which also handles nested-lambda shadowing). Any other shape falls through to the codegen dispatcher, which runs Spark's own evaluation and matches Spark exactly.

How are these changes tested?

Added regression queries to array_filter.sql: a genuine array_compact case (x -> x IS NOT NULL) that must still fast-path, and the bug case (x -> c IS NOT NULL with a captured column and null array elements) that must match Spark. Verified the captured-column case fails against the pre-fix code and passes with the fix, and that the genuine fast path still runs natively.

[parthchandra]

Can we add tests for the following cases -

• IsNotNull on an expression of the lambda var (x -> (x + 1) IS NOT NULL) — exercises the NamedLambdaVariable type check
  • Compound predicate (x -> x IS NOT NULL AND x > 0) — exercises the structural match
  • Null array input — verifies null propagation through the fast path
  • Empty array — boundary condition

[parthchandra]

Also, Spark4xCometExprShim.scala has a shim for this case. Does that need to be updated?