Skip to content

[Fix] Apply weight decay in SignSGD - #518

Merged
kozistr merged 1 commit into
kozistr:mainfrom
11NOel11:fix-signsgd-weight-decay
Aug 23, 2026
Merged

[Fix] Apply weight decay in SignSGD#518
kozistr merged 1 commit into
kozistr:mainfrom
11NOel11:fix-signsgd-weight-decay

Conversation

@11NOel11

Copy link
Copy Markdown
Contributor

Problem (Why?)

close #517

SignSGD validates weight_decay and stores it with weight_decouple in defaults, but neither is
used in either step path, so both are silently ignored.

Solution (What/How?)

  • apply weight decay in _step_foreach and _step_per_param

Both calls go after gradient maximisation and before the momentum buffer update, following Tiger.
SGDW applies it after the buffer update instead, which is fine there because grad = buf aliases the
buffer; in SignSGD the buffer and the gradient stay separate, so decay applied after the lerp would be
dropped for the coupled case.

fixed_decay=False is hardcoded as in SGDW, since SignSGD has no fixed_decay argument. Happy to
add one if you would rather have it configurable.

Other changes (bug fixes, small refactors)

N/A

Notes

This changes results for anyone currently passing a non-zero weight_decay, since it did nothing
before. The default path is unaffected: at weight_decay=0.0 the parameters are bit-identical to main
across momentum, weight_decouple and foreach settings.

One thing I noticed while testing, not caused by this change: the two step paths can disagree when the
momentum buffer lands near zero, since _step_per_param uses buf.mul_().add_() and _step_foreach uses
_foreach_lerp_ - algebraically equal, different rounding, and sign() turns 1e-17 into a full lr
step. It reproduces on main at weight_decay=0.0. Happy to file it separately.

Three tests added, each parametrised over both foreach paths. With p=2.0, zero gradient, lr=0.1,
momentum=0.9, weight_decay=0.2: decoupled gives 2.0 * (1 - 0.2 * 0.1) = 1.96, coupled gives
grad = 0.2 * 2.0 = 0.4 so the update is -lr * sign(0.04) = -0.1 and p = 1.9. The four decay tests
fail on main and pass with this change; the weight_decay=0.0 test passes on both.

Checklist

  • Make sure to run just format before commit
  • My code adheres to the style guidelines of this project (just check shows no errors)
  • Both new and existing unit tests pass successfully on my local environment by running just test
  • I have made the necessary changes to the documentation

ruff check is clean on both files; I did not run pyright. Test suite: 2488 passed before, 2494 after,
with the same two pre-existing failures (test_complex_optimizers[FTRL_...] and test_parse_version).

`weight_decay` and `weight_decouple` were validated and stored in `defaults`
but never used, so both were silently ignored.

Apply them in both step paths, after gradient maximisation and before the
momentum buffer update, following `Tiger`. `SGDW` decays after the buffer
update, which works there because `grad` aliases the buffer; in `SignSGD` the
two stay separate, so coupled decay would be dropped.
@11NOel11
11NOel11 requested a review from kozistr as a code owner August 23, 2026 08:01
@github-actions github-actions Bot added the optimizer about optimizer label Aug 23, 2026
@codecov

codecov Bot commented Aug 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (cabe9bf) to head (d3b5349).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #518   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          128       128           
  Lines        12442     12444    +2     
=========================================
+ Hits         12442     12444    +2     
Flag Coverage Δ
3.10 100.00% <100.00%> (ø)
3.11 100.00% <100.00%> (ø)
3.12 100.00% <100.00%> (ø)
3.13 100.00% <100.00%> (ø)
3.14 100.00% <100.00%> (ø)
3.8 99.98% <100.00%> (ø)
3.9 99.98% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@kozistr kozistr left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me! thanks for your contribution :)

@kozistr
kozistr merged commit 833817c into kozistr:main Aug 23, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer about optimizer size/S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SignSGD accepts weight_decay but never applies it

2 participants