Fix GroupNormalization with num_groups=1 to include channel axis in reduction - #1052
Open
ssam18 wants to merge 1 commit into
Open
Fix GroupNormalization with num_groups=1 to include channel axis in reduction#1052ssam18 wants to merge 1 commit into
ssam18 wants to merge 1 commit into
Conversation
…eduction When num_groups equals 1 the ONNX importer was building axesMask from only the spatial dimensions, so INormalizationLayer ended up reducing per channel and the output silently matched InstanceNorm instead of the LayerNorm over CHW behavior that ONNX and PyTorch group_norm specify. This change threads an includeChannelAxis flag through normalizationHelper and adds axis 1 to axesMask only when num_groups is 1, leaving InstanceNormalization and the proper multi group case untouched. Fixes NVIDIA/TensorRT#4756 Signed-off-by: Samaresh Kumar Singh <ssam3003@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ONNX GroupNormalization with num_groups equal to 1 was silently producing per channel instance normalization instead of the LayerNorm over CHW behavior that the ONNX spec and PyTorch group_norm define. The cause was that the importer built axesMask from spatial dimensions only and excluded the channel axis, so when INormalizationLayer ran with nbGroups equal to 1 there was no implicit reduction over channels and the result collapsed to per channel statistics. This patch adds the channel axis to axesMask in the nbGroups equal to 1 case while keeping the existing spatial only mask for nbGroups greater than 1 (where the runtime explicitly forbids axis 1 in axesMask) and for InstanceNormalization.
Test plan
Fixes NVIDIA/TensorRT#4756
Signed-off-by: Samaresh Kumar Singh ssam3003@gmail.com