Skip to content

Fix SCE mask shape on early exits - #700

Open
SarveshaKumarKS wants to merge 1 commit into
arcee-ai:mainfrom
SarveshaKumarKS:agent/fix-sce-mask-shape
Open

Fix SCE mask shape on early exits#700
SarveshaKumarKS wants to merge 1 commit into
arcee-ai:mainfrom
SarveshaKumarKS:agent/fix-sce-mask-shape

Conversation

@SarveshaKumarKS

@SarveshaKumarKS SarveshaKumarKS commented Jul 30, 2026

Copy link
Copy Markdown

Summary

  • return SCE masks with the parameter tensor shape on every early-exit path
  • cover zero and full density, small densities where k == 0, and the merge-level output shape

Root cause

The normal sce_mask path builds its mask from the variance tensor, whose shape excludes the leading model dimension. The early exits instead used zeros_like(tvs) or ones_like(tvs), retaining that dimension. sce_merge then unsqueezed the mask, and PyTorch broadcasting silently introduced an extra leading dimension into the merged tensor.

Impact

Degenerate select_topk values such as 0, as well as small positive values that round down to k == 0, now return the unchanged base tensor with its original rank instead of a rank-corrupted tensor.

Validation

  • python -m pytest tests/ -q — 94 passed
  • python -m pytest tests/test_sce.py -q — 6 passed
  • python -m pytest tests/test_basic_merges.py::TestBasicMerges::test_sce_merge -q — 1 passed
  • pre-commit run --files mergekit/merge_methods/sce.py tests/test_sce.py — all hooks passed

@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@SarveshaKumarKS

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

@SarveshaKumarKS
SarveshaKumarKS marked this pull request as ready for review July 30, 2026 01:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant