[Bug]: mo-tester sorts rows by ignored columns, causing false BVT failures

[gouhongshen]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Branch Name

main

Commit ID

MatrixOne PR head 34cfec2c408bef6a25e4d2f02bd06e20960727f4; mo-tester b95f5c09930c42bdbe77828b6ae72d39f2be48ed

Other Environment Information

- Hardware parameters: GitHub-hosted x86 runner
- OS type: Ubuntu 22.04
- Others: Matrixone Compose CI, multi-CN BVT with PROXY

Actual Behavior

The PROXY BVT job in MatrixOne PR #27154 falsely fails two logically matching queries in test/distributed/cases/snapshot/sys_restore_system_table_to_sys_account.sql:

ERROR RSRow - [row:0,column:1] one is [test_if_hit_second_elseif],
but the other is [test_if_hit_else]

The SQL marks columns 0,9,10 as ignored. Expected rows use procedure IDs 1920000 and 1920001; actual rows use IDs 9 and 10. All checked (non-ignored) values match.

Root cause is in mo-tester: RSSet.equals sorts rows using RSRow.rowText, and rowText contains ignored columns. Lexical sorting keeps expected IDs in 1920000,1920001 order but sorts actual IDs as 10,9. The subsequent cell comparison skips ignored columns, but by then it compares the two procedure names crosswise.

Evidence:

• failed job 94708833688
• RSSet sorts by full row text
• RSRow skips ignored cells only during equality

The same mismatch occurs before and after snapshot restore, while the actual rows contain the expected procedures. This proves the failure is in result matching rather than snapshot restore or PR #27154.

Expected Behavior

Ignored columns must not influence unordered row matching or sort order. These two result sets should compare equal, and the BVT should pass.

Steps to Reproduce

1. Use mo-tester b95f5c09930c42bdbe77828b6ae72d39f2be48ed.
2. Compare a two-row expected result with ignored column 0 values 1920000 and 1920001.
3. Return the same checked values with ignored column 0 values 9 and 10.
4. Observe that RSSet.equals sorts by the full row text, pairs the rows in different orders, and reports a false difference in checked column 1.
5. The concrete CI reproducer is test/distributed/cases/snapshot/sys_restore_system_table_to_sys_account.sql at the two "select * from mo_catalog.mo_stored_procedure" statements around SQL rows 167 and 183.

Additional information

Searches in both matrixorigin/matrixone and matrixorigin/mo-tester found no issue or PR for ignored columns affecting row sorting. MatrixOne #22191 / mo-tester #178 concern a different metadata-column-count crash.

[gouhongshen]

Automated AI triage

This issue requires human involvement and cannot currently be completed independently by the
automated fix loop.

Reason

The stated root cause and required fix are in the external matrixorigin/mo-tester repository, so a matrixone-scoped agent cannot deliver the necessary PR or release integration. A human/mo-tester maintainer must approve and merge the ignored-column row-ordering fix, then rerun the MatrixOne PROXY BVT.

Evidence

The issue explicitly identifies mo-tester RSSet.equals/RSRow as the fault location and links files outside this matrixone checkout; no mo-tester implementation exists in the repository.

Please use this analysis to provide the missing decision, environment, evidence, or ownership
needed before the issue is returned to the AI candidate pool.

[gouhongshen]

The single permitted rerun reproduced the same false negative, so this is not an isolated attempt-1 result.

• Workflow: 31795529089
• Attempt: 2
• PROXY BVT job: 94763041266
• Tested PR head: 9cf70871568445211d1d5e094f92f9c85eef193a
• mo-tester result: sys_restore_system_table_to_sys_account.sql completed 313 statements with 311 successes and the same two failures.

Both failures are again select * from mo_catalog.mo_stored_procedure. The expected rows use ignored procedure IDs 1920000/1920001; the actual rows use 9/10. The checked procedure data is present, but sorting by the ignored IDs pairs test_if_hit_second_elseif with test_if_hit_else, producing the same row:0,column:1 mismatch described in this issue.

The downstream Coverage job 94772054154 is derivative: it receives prerequisites_ready: false because the Compose BVT producer failed and exits before processing coverage profiles.

Per the PR CI policy, workflow attempt 2 will not be rerun again. PR #27154 still has no changes in mo-tester or this snapshot case.

[gouhongshen]

The false negative recurred unchanged on PR #27154 after its rebase onto current main.

• Workflow: 32885250997 attempt 1
• PROXY BVT job: 97926799349
• Tested head: 6ad3ad6b7fe4ab12b3e6929d1eb1696980a6b580
• Failing case: sys_restore_system_table_to_sys_account.sql, 311/313 statements successful

The only two failed statements were again the queries at SQL rows 167 and 183:

select * from mo_catalog.mo_stored_procedure;

Expected ignored proc_id values were 1920000/1920001; actual ignored values were 9/10. The expected procedures and checked data were present, but mo-tester again paired test_if_hit_second_elseif with test_if_hit_else and reported the same row:0,column:1 mismatch. This is the exact ignored-column row-sorting root cause documented above.

The downstream Coverage job 97938717369 is derivative: it received prerequisites_ready: false and exited before coverage processing. The PR diff remains limited to clone allocator reconstruction and clone UT/BVT files; it does not touch this snapshot case or mo-tester. The full Ubuntu UT, coverage UT, shared build, SCA, and PESSIMISTIC BVT jobs passed on the same head.

This records the unrelated attempt-1 evidence before the single permitted failed-job rerun.

[gouhongshen]

Fresh recurrence on PR #27154 workflow 32885250997, attempt 2, exact head 6ad3ad6b7fe4ab12b3e6929d1eb1696980a6b580:

• The PROXY BVT job again completed sys_restore_system_table_to_sys_account.sql with 313 statements, 311 successful and exactly two failures.
• Both failures are the same select * from mo_catalog.mo_stored_procedure; comparisons at case rows 167 and 183. Expected ignored proc_id values are 1920000/1920001; actual ignored values are 9/10. The checked procedure rows are present, but mo-tester pairs test_if_hit_second_elseif with test_if_hit_else after sorting on the ignored IDs.
• The Coverage job is derivative: it reports prerequisites_ready: false and skips merging because a required producer failed.
• On the same head, the full Ubuntu UT, coverage UT, shared build, SCA, and PESSIMISTIC BVT jobs passed. The PR diff changes only clone allocator code/tests and does not touch this snapshot case or mo-tester.

This confirms the tracked false negative repeated unchanged after the one permitted rerun. The workflow is already on attempt 2, so it will not be rerun again.

[gouhongshen]

Fresh recurrence on PR #27154 workflow 32955538828 attempt 1, exact head 2b30e359edb447e9efca4739f91adc01b4270219:

• The PESSIMISTIC BVT job completed snapshot/sys_restore_system_table_to_sys_account.sql with 313 statements, 311 successful and exactly two failures.
• Both failures are again select * from mo_catalog.mo_stored_procedure; at SQL rows 167 and 183. Expected ignored proc_id values are 1920000/1920001; actual ignored values are 9/10. The checked procedure rows are present, but mo-tester reports the same test_if_hit_second_elseif versus test_if_hit_else cross-pairing after sorting on ignored values.
• The downstream Coverage job is derivative: prerequisites_ready: false makes it exit before coverage processing because the required BVT producer failed.
• The same-head shared build, SCA, PROXY BVT, and coverage UT passed. The PR diff remains limited to clone allocator code/tests and clone BVT files; it does not touch this snapshot case or mo-tester.

This records the unrelated attempt-1 evidence before the single permitted failed-job rerun.

[gouhongshen]

Fresh recurrence on PR #27154 workflow 32965794843 attempt 1, exact head b92cccb80de28238ff878a2b1dc4a750aba6552d:

• The PROXY BVT job completed snapshot/sys_restore_system_table_to_sys_account.sql with exactly two failed statements.
• Both failures are again select * from mo_catalog.mo_stored_procedure; at SQL rows 167 and 183. Expected ignored proc_id values are 1920000/1920001; actual ignored values are 9/10. The checked rows are present, but mo-tester reports the same test_if_hit_second_elseif versus test_if_hit_else cross-pairing after sorting on ignored values.
• The downstream Coverage job is derivative: it receives prerequisites_ready: false and exits before coverage processing because the required BVT producer failed.
• The same-head shared build, SCA, coverage UT, and PESSIMISTIC BVT passed. The PR diff remains limited to six clone allocator/test files and does not touch this snapshot case or mo-tester.

This records the unrelated attempt-1 evidence before the single permitted failed-job rerun.

[gouhongshen]

Fresh recurrence on PR #27154 workflow 32980530753 attempt 1, exact head 1e6e18ab0b35c8aa5f76ce285c39bd2050d20e60:

• The PROXY BVT job execution artifact bvt-execution-compose-attempt-1 shows snapshot/sys_restore_system_table_to_sys_account.sql completed 313 statements with 311 successful and exactly two failures.
• Both failures are again select * from mo_catalog.mo_stored_procedure; at SQL rows 167 and 183. Expected ignored proc_id values are 1920000/1920001; actual ignored values are 9/10. The checked rows are present, but mo-tester reports the same test_if_hit_second_elseif versus test_if_hit_else cross-pairing after sorting on ignored values.
• The downstream Coverage job is derivative: it receives prerequisites_ready: false for expected BVT generation 32980530753-1 and exits before coverage processing.
• The same-head shared build, SCA, coverage UT, and PESSIMISTIC BVT passed. The PR diff remains limited to six clone allocator/test files and does not touch this snapshot case or mo-tester.

This records the unrelated attempt-1 evidence before the single permitted failed-job rerun.

[gouhongshen]

The unrelated false negative repeated unchanged on PR #27154 workflow 32980530753 attempt 2, exact head 1e6e18ab0b35c8aa5f76ce285c39bd2050d20e60.

• The rerun PROXY BVT job and artifact bvt-execution-compose-attempt-2 show snapshot/sys_restore_system_table_to_sys_account.sql completed 313 statements with 311 successful and exactly two failures.
• Both failures remain select * from mo_catalog.mo_stored_procedure; at SQL rows 167 and 183. Expected ignored proc_id values are 1920000/1920001; actual ignored values are 9/10. The rows are present, but mo-tester again cross-pairs test_if_hit_second_elseif with test_if_hit_else.
• The attempt-2 Coverage job is derivative: it receives prerequisites_ready: false for expected BVT generation 32980530753-1 and exits before coverage processing.
• The same-head Ubuntu UT, shared build, SCA, coverage UT, and PESSIMISTIC BVT passed. The PR diff remains limited to six clone allocator/test files and does not touch this snapshot case or mo-tester.

This is the single permitted rerun and confirms the tracked root cause is stable. Workflow attempt 2 will not be rerun again; the required rebase has superseded this head.