[InstCombine] Narrow signed min/max DAGs before truncation - #218688
Open
mangsgi wants to merge 2 commits into
Open
[InstCombine] Narrow signed min/max DAGs before truncation#218688mangsgi wants to merge 2 commits into
mangsgi wants to merge 2 commits into
Conversation
Recognize fixed-vector signed min/max DAGs whose leaves are sign extensions from the truncation destination type, and rebuild the DAG in that narrow type. Require all wide nodes to have no users outside the matched DAG so the fold does not leave the wide computation alive. The DAG structure proves that its result fits in the narrow signed type, so the fold applies to trunc with or without nsw. Proof: https://alive2.llvm.org/ce/z/ngfMYZ Fixes llvm#214224
|
Hello @mangsgi 👋 Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.
Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description. Frequently asked questionsHow do I add reviewers? This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically. You can also add reviewers manually using the Reviewers section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using What if there are no comments? If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers. Are any special GitHub settings required to contribute to LLVM? We only require contributors to have a public email address associated with their GitHub commits, see this section of LLVM Developer Policy for details. If you have questions, feel free to leave a comment on this PR, or ask on LLVM Discord or LLVM Discourse. Thank you, |
llvmorg-github-actions
Bot
added
llvm:instcombine
|
@llvm/pr-subscribers-llvm-transforms Author: KIM MYEONG SEOK (mangsgi) ChangesRecognize fixed-vector signed For example, this changes the following expression shape: into: Sign extension preserves signed ordering, so signed minimum and maximum To avoid increasing code size, require every wide instruction to have no users For the motivating pattern, this removes three sign extensions and the final Tests cover:
Testing:
Proof:
Fixes #214224 Full diff: https://github.com/llvm/llvm-project/pull/218688.diff 2 Files Affected:
diff --git a/llvm/lib/Transforms/InstCombine/InstCombineCasts.cpp b/llvm/lib/Transforms/InstCombine/InstCombineCasts.cpp
index 2db6396b9d661..4d9e020b27c14 100644
--- a/llvm/lib/Transforms/InstCombine/InstCombineCasts.cpp
+++ b/llvm/lib/Transforms/InstCombine/InstCombineCasts.cpp
@@ -1042,6 +1042,76 @@ static Instruction *shrinkInsertElt(CastInst &Trunc,
return nullptr;
}
+/// Return true if Trunc has a same-block fixed-vector signed min/max DAG
+/// that can be rebuilt in the destination type without leaving wide uses.
+static bool
+collectNarrowableSignedMinMaxDAG(Value *V, FixedVectorType *WideTy,
+ FixedVectorType *NarrowTy, BasicBlock *BB,
+ SmallSetVector<Instruction *, 16> &Nodes,
+ unsigned &NumMinMax) {
+ auto *I = dyn_cast<Instruction>(V);
+ if (!I || I->getParent() != BB || I->getType() != WideTy)
+ return false;
+
+ // A shared node in the DAG may be reached by multiple paths.
+ if (!Nodes.insert(I))
+ return true;
+
+ // Every leaf must be a matching sext from the destination type.
+ if (auto *Ext = dyn_cast<SExtInst>(I))
+ return Ext->getSrcTy() == NarrowTy;
+
+ // Every non-leaf node must be signed min/max.
+ auto *MM = dyn_cast<MinMaxIntrinsic>(I);
+ if (!MM || !MM->isSigned())
+ return false;
+
+ ++NumMinMax;
+ return collectNarrowableSignedMinMaxDAG(MM->getLHS(), WideTy, NarrowTy, BB,
+ Nodes, NumMinMax) &&
+ collectNarrowableSignedMinMaxDAG(MM->getRHS(), WideTy, NarrowTy, BB,
+ Nodes, NumMinMax);
+}
+
+/// Return true if Trunc can be replaced by rebuilding its signed min/max
+/// operand DAG in the destination type.
+static bool canNarrowSignedMinMaxDAG(TruncInst &Trunc) {
+ auto *WideTy = dyn_cast<FixedVectorType>(Trunc.getSrcTy());
+ auto *NarrowTy = dyn_cast<FixedVectorType>(Trunc.getDestTy());
+ if (!WideTy || !NarrowTy)
+ return false;
+
+ if (WideTy->getNumElements() != NarrowTy->getNumElements() ||
+ !WideTy->getElementType()->isIntegerTy() ||
+ !NarrowTy->getElementType()->isIntegerTy())
+ return false;
+
+ SmallSetVector<Instruction *, 16> Nodes;
+ unsigned NumMinMax = 0;
+
+ if (!collectNarrowableSignedMinMaxDAG(Trunc.getOperand(0), WideTy, NarrowTy,
+ Trunc.getParent(), Nodes, NumMinMax))
+ return false;
+
+ // Avoid overlap with existing one-node folds.
+ if (NumMinMax < 2)
+ return false;
+
+ // Every wide node and extension must become dead after replacing Trunc.
+ for (Instruction *I : Nodes) {
+ for (User *U : I->users()) {
+ if (U == &Trunc)
+ continue;
+
+ auto *UserI = dyn_cast<Instruction>(U);
+ if (!UserI || !Nodes.contains(UserI))
+ return false;
+ }
+ }
+
+ return true;
+}
+
Instruction *InstCombinerImpl::visitTrunc(TruncInst &Trunc) {
if (Instruction *Result = commonCastTransforms(Trunc))
return Result;
@@ -1056,7 +1126,8 @@ Instruction *InstCombinerImpl::visitTrunc(TruncInst &Trunc) {
// expression tree to something weird like i93 unless the source is also
// strange.
if ((DestTy->isVectorTy() || shouldChangeType(SrcTy, DestTy)) &&
- TypeEvaluationHelper::canEvaluateTruncated(Src, DestTy, *this, &Trunc)) {
+ (TypeEvaluationHelper::canEvaluateTruncated(Src, DestTy, *this, &Trunc) ||
+ canNarrowSignedMinMaxDAG(Trunc))) {
// If this cast is a truncate, evaluting in a different type always
// eliminates the cast, so it is always a win.
diff --git a/llvm/test/Transforms/InstCombine/trunc-minmax-intrinsics.ll b/llvm/test/Transforms/InstCombine/trunc-minmax-intrinsics.ll
index 8a2d38fb85bc3..4dd6b40fbd536 100644
--- a/llvm/test/Transforms/InstCombine/trunc-minmax-intrinsics.ll
+++ b/llvm/test/Transforms/InstCombine/trunc-minmax-intrinsics.ll
@@ -2,6 +2,7 @@
; RUN: opt -passes=instcombine %s -S | FileCheck %s
declare void @use_i32(i32)
+declare void @use_v8i32(<8 x i32>)
define i8 @umin_mul3_clamp(i8 %x) {
; CHECK-LABEL: define i8 @umin_mul3_clamp(
@@ -621,3 +622,120 @@ define <vscale x 4 x i8> @smax_sext_scalable_vec(<vscale x 4 x i8> %x, <vscale x
%t = trunc <vscale x 4 x i32> %u to <vscale x 4 x i8>
ret <vscale x 4 x i8> %t
}
+
+define <8 x i16> @smin_smax_dag_shared_sext_vec(
+; CHECK-LABEL: define <8 x i16> @smin_smax_dag_shared_sext_vec(
+; CHECK-SAME: <8 x i16> [[A:%.*]], <8 x i16> [[B:%.*]], <8 x i16> [[C:%.*]]) {
+; CHECK-NEXT: [[LO:%.*]] = call <8 x i16> @llvm.smin.v8i16(<8 x i16> [[A]], <8 x i16> [[B]])
+; CHECK-NEXT: [[HI:%.*]] = call <8 x i16> @llvm.smax.v8i16(<8 x i16> [[A]], <8 x i16> [[B]])
+; CHECK-NEXT: [[MID:%.*]] = call <8 x i16> @llvm.smin.v8i16(<8 x i16> [[C]], <8 x i16> [[HI]])
+; CHECK-NEXT: [[WIDE:%.*]] = call <8 x i16> @llvm.smax.v8i16(<8 x i16> [[MID]], <8 x i16> [[LO]])
+; CHECK-NEXT: ret <8 x i16> [[WIDE]]
+;
+ <8 x i16> %a, <8 x i16> %b, <8 x i16> %c) {
+ %aw = sext <8 x i16> %a to <8 x i32>
+ %bw = sext <8 x i16> %b to <8 x i32>
+ %cw = sext <8 x i16> %c to <8 x i32>
+ %lo = call <8 x i32> @llvm.smin.v8i32(<8 x i32> %aw, <8 x i32> %bw)
+ %hi = call <8 x i32> @llvm.smax.v8i32(<8 x i32> %aw, <8 x i32> %bw)
+ %mid = call <8 x i32> @llvm.smin.v8i32(<8 x i32> %cw, <8 x i32> %hi)
+ %wide = call <8 x i32> @llvm.smax.v8i32(<8 x i32> %mid, <8 x i32> %lo)
+ %result = trunc nsw <8 x i32> %wide to <8 x i16>
+ ret <8 x i16> %result
+}
+
+; Check that the fold is independent of the element widths and also handles a
+; non-adjacent narrowing from i64 to i8.
+define <4 x i8> @smin_smax_dag_sext_i8_i64_vec(
+; CHECK-LABEL: define <4 x i8> @smin_smax_dag_sext_i8_i64_vec(
+; CHECK-SAME: <4 x i8> [[A:%.*]], <4 x i8> [[B:%.*]], <4 x i8> [[C:%.*]]) {
+; CHECK-NEXT: [[LO:%.*]] = call <4 x i8> @llvm.smin.v4i8(<4 x i8> [[A]], <4 x i8> [[B]])
+; CHECK-NEXT: [[HI:%.*]] = call <4 x i8> @llvm.smax.v4i8(<4 x i8> [[A]], <4 x i8> [[B]])
+; CHECK-NEXT: [[MID:%.*]] = call <4 x i8> @llvm.smin.v4i8(<4 x i8> [[C]], <4 x i8> [[HI]])
+; CHECK-NEXT: [[WIDE:%.*]] = call <4 x i8> @llvm.smax.v4i8(<4 x i8> [[MID]], <4 x i8> [[LO]])
+; CHECK-NEXT: ret <4 x i8> [[WIDE]]
+;
+ <4 x i8> %a, <4 x i8> %b, <4 x i8> %c) {
+ %aw = sext <4 x i8> %a to <4 x i64>
+ %bw = sext <4 x i8> %b to <4 x i64>
+ %cw = sext <4 x i8> %c to <4 x i64>
+ %lo = call <4 x i64> @llvm.smin.v4i64(<4 x i64> %aw, <4 x i64> %bw)
+ %hi = call <4 x i64> @llvm.smax.v4i64(<4 x i64> %aw, <4 x i64> %bw)
+ %mid = call <4 x i64> @llvm.smin.v4i64(<4 x i64> %cw, <4 x i64> %hi)
+ %wide = call <4 x i64> @llvm.smax.v4i64(<4 x i64> %mid, <4 x i64> %lo)
+ %result = trunc nsw <4 x i64> %wide to <4 x i8>
+ ret <4 x i8> %result
+}
+
+; Do not build a narrow DAG when that would leave an externally-used copy of
+; the wide DAG alive.
+define <8 x i16> @smin_smax_dag_external_wide_use_vec(
+; CHECK-LABEL: define <8 x i16> @smin_smax_dag_external_wide_use_vec(
+; CHECK-SAME: <8 x i16> [[A:%.*]], <8 x i16> [[B:%.*]], <8 x i16> [[C:%.*]]) {
+; CHECK-NEXT: [[AW:%.*]] = sext <8 x i16> [[A]] to <8 x i32>
+; CHECK-NEXT: [[BW:%.*]] = sext <8 x i16> [[B]] to <8 x i32>
+; CHECK-NEXT: [[CW:%.*]] = sext <8 x i16> [[C]] to <8 x i32>
+; CHECK-NEXT: [[LO:%.*]] = call <8 x i32> @llvm.smin.v8i32(<8 x i32> [[AW]], <8 x i32> [[BW]])
+; CHECK-NEXT: call void @use_v8i32(<8 x i32> [[LO]])
+; CHECK-NEXT: [[HI:%.*]] = call <8 x i32> @llvm.smax.v8i32(<8 x i32> [[AW]], <8 x i32> [[BW]])
+; CHECK-NEXT: [[MID:%.*]] = call <8 x i32> @llvm.smin.v8i32(<8 x i32> [[CW]], <8 x i32> [[HI]])
+; CHECK-NEXT: [[WIDE:%.*]] = call <8 x i32> @llvm.smax.v8i32(<8 x i32> [[MID]], <8 x i32> [[LO]])
+; CHECK-NEXT: [[RESULT:%.*]] = trunc nsw <8 x i32> [[WIDE]] to <8 x i16>
+; CHECK-NEXT: ret <8 x i16> [[RESULT]]
+;
+ <8 x i16> %a, <8 x i16> %b, <8 x i16> %c) {
+ %aw = sext <8 x i16> %a to <8 x i32>
+ %bw = sext <8 x i16> %b to <8 x i32>
+ %cw = sext <8 x i16> %c to <8 x i32>
+ %lo = call <8 x i32> @llvm.smin.v8i32(<8 x i32> %aw, <8 x i32> %bw)
+ call void @use_v8i32(<8 x i32> %lo)
+ %hi = call <8 x i32> @llvm.smax.v8i32(<8 x i32> %aw, <8 x i32> %bw)
+ %mid = call <8 x i32> @llvm.smin.v8i32(<8 x i32> %cw, <8 x i32> %hi)
+ %wide = call <8 x i32> @llvm.smax.v8i32(<8 x i32> %mid, <8 x i32> %lo)
+ %result = trunc nsw <8 x i32> %wide to <8 x i16>
+ ret <8 x i16> %result
+}
+
+; Do not narrow a DAG with a leaf that is not a sign extension from the
+; destination type.
+define <8 x i16> @smin_smax_dag_non_sext_leaf_vec(
+; CHECK-LABEL: define <8 x i16> @smin_smax_dag_non_sext_leaf_vec(
+; CHECK-SAME: <8 x i16> [[A:%.*]], <8 x i16> [[B:%.*]], <8 x i32> [[X:%.*]]) {
+; CHECK-NEXT: [[AW:%.*]] = sext <8 x i16> [[A]] to <8 x i32>
+; CHECK-NEXT: [[BW:%.*]] = sext <8 x i16> [[B]] to <8 x i32>
+; CHECK-NEXT: [[LO:%.*]] = call <8 x i32> @llvm.smin.v8i32(<8 x i32> [[AW]], <8 x i32> [[X]])
+; CHECK-NEXT: [[WIDE:%.*]] = call <8 x i32> @llvm.smax.v8i32(<8 x i32> [[LO]], <8 x i32> [[BW]])
+; CHECK-NEXT: [[RESULT:%.*]] = trunc nsw <8 x i32> [[WIDE]] to <8 x i16>
+; CHECK-NEXT: ret <8 x i16> [[RESULT]]
+;
+ <8 x i16> %a, <8 x i16> %b, <8 x i32> %x) {
+ %aw = sext <8 x i16> %a to <8 x i32>
+ %bw = sext <8 x i16> %b to <8 x i32>
+ %lo = call <8 x i32> @llvm.smin.v8i32(<8 x i32> %aw, <8 x i32> %x)
+ %wide = call <8 x i32> @llvm.smax.v8i32(<8 x i32> %lo, <8 x i32> %bw)
+ %result = trunc nsw <8 x i32> %wide to <8 x i16>
+ ret <8 x i16> %result
+}
+
+; The min/max DAG itself proves that the wide result fits in the narrow type,
+; so the final trunc does not need an nsw flag.
+define <4 x i16> @smin_smax_dag_plain_trunc_vec(
+; CHECK-LABEL: define <4 x i16> @smin_smax_dag_plain_trunc_vec(
+; CHECK-SAME: <4 x i16> [[A:%.*]], <4 x i16> [[B:%.*]], <4 x i16> [[C:%.*]]) {
+; CHECK-NEXT: [[LO:%.*]] = call <4 x i16> @llvm.smin.v4i16(<4 x i16> [[A]], <4 x i16> [[B]])
+; CHECK-NEXT: [[HI:%.*]] = call <4 x i16> @llvm.smax.v4i16(<4 x i16> [[A]], <4 x i16> [[B]])
+; CHECK-NEXT: [[MID:%.*]] = call <4 x i16> @llvm.smin.v4i16(<4 x i16> [[C]], <4 x i16> [[HI]])
+; CHECK-NEXT: [[WIDE:%.*]] = call <4 x i16> @llvm.smax.v4i16(<4 x i16> [[MID]], <4 x i16> [[LO]])
+; CHECK-NEXT: ret <4 x i16> [[WIDE]]
+;
+ <4 x i16> %a, <4 x i16> %b, <4 x i16> %c) {
+ %aw = sext <4 x i16> %a to <4 x i32>
+ %bw = sext <4 x i16> %b to <4 x i32>
+ %cw = sext <4 x i16> %c to <4 x i32>
+ %lo = call <4 x i32> @llvm.smin.v4i32(<4 x i32> %aw, <4 x i32> %bw)
+ %hi = call <4 x i32> @llvm.smax.v4i32(<4 x i32> %aw, <4 x i32> %bw)
+ %mid = call <4 x i32> @llvm.smin.v4i32(<4 x i32> %cw, <4 x i32> %hi)
+ %wide = call <4 x i32> @llvm.smax.v4i32(<4 x i32> %mid, <4 x i32> %lo)
+ %result = trunc <4 x i32> %wide to <4 x i16>
+ ret <4 x i16> %result
+}
|
nikic
reviewed
Aug 25, 2026
nikic
left a comment
nikic
left a comment
Contributor
There was a problem hiding this comment.
Why does this need separate handling from canEvaluateTruncated?
Author
|
The existing canEvaluateTruncated path rejects this DAG before reaching the sext leaves, because ComputeMaxSignificantBits() reports the intermediate min/max operands at the wide width. I handled it separately so that the structural check and the no-external-users restriction remain local to this fold, without broadening the general canEvaluateTruncated logic. However, I don't think separate handling is fundamentally required. I'll check whether the existing path can be extended cleanly instead. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Recognize fixed-vector signed
llvm.smin/llvm.smaxDAGs whose leaves aresign extensions from the truncation destination type, and rebuild the same DAG
directly in that narrow type.
For example, this changes the following expression shape:
into:
Sign extension preserves signed ordering, so signed minimum and maximum
commute with sign extension. Since each min/max result lane is selected from
the corresponding operand lanes, a DAG whose leaves are sign extensions from
the narrow type also remains within that narrow signed range. The fold therefore
applies whether or not the final
trunccarriesnsw.To avoid increasing code size, require every wide instruction to have no users
outside the matched same-block DAG and the final truncation. Existing one-node
min/max folds remain handled by the general
canEvaluateTruncatedpath.For the motivating pattern, this removes three sign extensions and the final
truncation, and performs all four min/max operations in the narrow vector type.
Tests cover:
<8 x i16>to<8 x i32>signed min/max DAG<4 x i64>to<4 x i8>nswTesting:
llvm-lit -v llvm/test/Transforms/InstCombine/trunc-minmax-intrinsics.llllvm-lit -sv -j2 llvm/test/Transforms/InstCombineopt -passes=instcombine -verify-each -disable-output llvm/test/Transforms/InstCombine/trunc-minmax-intrinsics.llProof:
--disable-undef-input):https://alive2.llvm.org/ce/z/ngfMYZ
https://alive2.llvm.org/ce/z/7VtWe8
Fixes #214224