Add deterministic duplicate-issue detection and remove the broken similar-issues bot

[Jaylyn Barbee (Jaylyn-Barbee)]

Fixes

Implements Phase 2 of #11673 (Proposal: issue triage and duplicate detection automation).

PR Type

• Other (please describe): repository issue-triage automation. No product, sample, or test-app code is touched — the diff is confined to .github/.

Description

Replaces the repo's dead similar-issues bot with a deterministic duplicate detector that runs on every issue opened, edited, or reopened.

When confident duplicates are found, the workflow posts one canonical comment listing them, and — only when the top match is High confidence — submits GitHub's native duplicate suggestion, which renders the accept/decline agent suggestion chip at the top of the issue. When nothing clears the threshold, nothing is posted. Issues are never closed and no labels are ever applied or removed.

Current Behavior

.github/workflows/similar-issues-bot.yml is still active and still triggers on every new issue, but it has not worked in roughly 16 months:

Evidence	Value
Retained workflow runs	647
Successful runs	0
Failed runs	647
Oldest retained run (already failing)	2025-07-11
Last bot comment actually posted	issue #10489, 2025-04-04
Issues that ever received the comment	362

Every failing log ends with the same error:

##[error]getaddrinfo ENOTFOUND gitgudissues.azurewebsites.net

craigloewen-msft/GitGudSimilarIssues POSTs the issue title and body to an external Azure App Service that no longer resolves. Because add-comment is gated on needs.getsimilarissues.outputs.message != '', the failure is silent — the job goes red, and no comment is ever posted.

Two secondary concerns with the old bot: it shipped public issue text to a third-party endpoint, and it was pinned to a mutable @main ref.

New Behavior

Duplicates are detected in-repo, with no external service and no model in the loop. This PR deletes similar-issues-bot.yml — keeping both would post two bot comments per issue.

Top match	Comment	Agent suggestion chip
Nothing at or above threshold	—	—
Low / Medium	✅	—
High	✅	✅

The chip is submitted with suggest: true, so GitHub holds the close for maintainer review rather than applying it. The comment is intentionally the wider net: it is cheap and reversible, whereas the chip is a one-click close and should appear only when the evidence is strong.

How Duplicates Are Detected

File	Responsibility
.github/workflows/duplicate-issue-detection.yml	Read-only detect job, then a separate issues: write publish job
.github/scripts/issue-triage/find_duplicates.py	Retrieval and deterministic scoring. No writes.
.github/scripts/issue-triage/tests/	40 unit tests
.github/workflows/issue-triage-tests.yml	Runs the tests on PRs that touch these paths
.github/scripts/issue-triage/README.md	Behavior, scoring table, chip contract, safety properties, local run instructions

Scoring is deterministic and fully auditable:

Component	Weight	Source
Title agreement	0.60	Token overlap and character ratio of the titles
Body agreement	0.25	Token overlap of the defect-describing issue-form sections
Shared identifiers	0.15	Control/API names, exception names, and 0x error codes

Version and metadata sections (NuGet package version, Windows version, Screenshots, Additional context) are excluded from scoring, so two unrelated reports on the same Windows build do not score as similar. Retrieval is capped at 30 candidates and 6 search queries per issue; at most 5 suggestions are returned. Default threshold is 0.62, configurable via SIMILARITY_THRESHOLD.

A label gate on the detect job restricts the workflow to issues carrying needs-triage, bug, or feature proposal. Both issue templates apply these at creation and blank_issues_enabled: false, so every user-filed issue qualifies; internal [WinUI OSS] tracking work does not. The gate is at job level, so excluded issues cost zero API calls.

Safety

Aligned with the requirements in #11673:

• Issue text is treated as untrusted. HTML comments, fenced code, images, links, and control characters are stripped before scoring; mentions and Markdown control characters are escaped before any text is republished.
• The detection job has no write permission. The publishing job re-reads issue state before writing and refuses to publish a body missing the canonical marker.
• Retrieval failures exit non-zero so publish is skipped, rather than producing a misleading "no duplicates found" result. This is the exact failure mode that made the old bot's breakage invisible.
• The chip step verifies the suggestion was actually recorded, via the GraphQL pendingSuggestions union, instead of trusting the HTTP status — the PATCH returns 200 either way. It defers entirely if a human has already proposed a duplicate, and if GitHub ever applied the close rather than suggesting it, the step reopens the issue and fails.
• Per-issue concurrency with cancel-in-progress: true.
• All third-party actions pinned to full commit SHAs. No external services; only the GitHub API is used.

Customer Impact

Not user-facing in the product sense — no runtime, API, or XAML behavior changes. The impact is on issue reporters and maintainers: reporters get duplicates surfaced within a minute of filing instead of never, and maintainers get a one-click close on high-confidence matches.

The visible risk is a false positive: an incorrect comment or an incorrect chip. That risk is quantified below, and the chip never closes anything on its own.

Regression Potential

• Low risk — isolated change, limited scope

No product code is touched. The one behavioral change to the repo is that issues begin receiving duplicate comments again, which is the intended outcome — and which has been broken since April 2025, so there is no working behavior to regress.

How Has This Been Tested?

• I have performed a self-review of my own code
• I have added tests to cover my changes
• Existing tests pass locally

Unit tests. 40 tests in .github/scripts/issue-triage/tests/, run in CI by issue-triage-tests.yml on any PR touching these paths.

Offline evaluation against the last 500 real issues. Every issue was scored against every older issue — 124,750 ordered pairs. This surfaced two significant false-positive sources, both fixed in this PR:

Stage	Pairs flagged at 0.62	From the [WinUI OSS] epic
Initial	228	218 (95.6%)
+ strip bracketed title prefixes	79	69
+ label gate	10	0

Templated epic sub-tasks share a title prefix, a body skeleton, and an identifier, and scored as near-identical to each other. After both fixes: 10 pairs across 7 issues (2.2% of the gated corpus), with zero false positives at any threshold at or above 0.55. All five clusters are genuine — French keyboard (#11646–#11649), PipsPager (#11467/#11469), ScrollView (#11039/#11040), AnimatedIcon (#10866/#10887), AOT custom control (#11065/#11248).

Two ground-truth checks: #11646 was closed by a maintainer with "Since this is a duplicate ticket, we are closing it." And #11467/#11469 is a real duplicate pair that is still open and unlinked today — the kind this would have caught at filing time.

End-to-end in a public fork (Jaylyn-Barbee/microsoft-ui-xaml), seeded with copies of real upstream issues:

Pilot issue	Source	Expected	Observed
#1, #2, #3, #4	Unrelated bugs	No comment	No comment
#5	First copy of the duplicate cluster	No comment	No comment
#6, #7, #8	Duplicate cluster (upstream #11646–#11649)	Comment	High-confidence comment
#9	Duplicate cluster	Comment + chip	Comment + chip; issue stayed open

Additional checks:

• Idempotency. Editing NavView and Alt + Left KeyBoardAccelerators #8 re-ran the workflow and updated the existing comment; the count stayed at 1. Re-running the chip step logged A duplicate suggestion pointing at #7 is already pending. Nothing to do. and the suggestion count stayed at 1.
• Confidence gate. An edit that dropped Create issue templates #9's match to Low correctly skipped the chip step while still posting the comment.
• Accepting the chip. The pending suggestion on Create issue templates #9 was accepted from the UI; the issue closed with state_reason: duplicate, linked to the canonical issue.

The pilot surfaced one real defect, fixed in the second commit: opening several issues in quick succession exhausted the search API's 30-requests-per-minute limit and returned HTTP 403. Requests are now paced, and retries honor Retry-After and x-ratelimit-reset with capped exponential backoff.

Scoped Out

This is Phase 2 of #11673, narrowed to duplicate detection. Deliberately not included: model-based judgment, area-* classification, reproduction-completeness analysis, and the maintainer digest.

One known gap, left out as a separate decision: the workflow triggers on opened, edited, and reopened, but not labeled. An issue transferred in from another repo, or one a maintainer labels after the fact, is not re-scanned. Adding labeled is a one-line change and safe given the idempotency above — happy to include it here if reviewers prefer.

[Jaylyn Barbee (Jaylyn-Barbee)]

Evaluation against 500 real issues, and two significant changes

Following review feedback (thanks Niels Laute (@niels9001)) I evaluated the detector offline against the 500 most recent issues in this repository, scoring 124,750 ordered pairs. Each issue was compared only against issues that already existed when it was filed, mirroring what the bot sees when a new issue is opened. This used the REST API only, with no writes and no search API.

The first run was bad, and it is worth showing why.

Finding: 95.6% of matches were one tracking epic

Run	Pairs at 0.62	From the [WinUI OSS] epic	Signal
Initial	228	218 (95.6%)	10
After stripping title prefixes	79	69 (87%)	10
After the label gate	10	0	10

The [WinUI OSS] Phase 4: ... sub-tasks are generated from a template. They share a title prefix, a body skeleton, and the XamlCompiler identifier, so they scored as duplicates of each other. In production this bot would have been roughly 96% noise.

Two fixes:

1. Strip structural title prefixes. [WinUI OSS] Phase 4: Update metadata factory layer is compared as Update metadata factory layer. The shared prefix is boilerplate, and it was inflating the title character ratio.
2. Only run on template-created issues. Both bug_report.yaml and feature_proposal.yaml apply needs-triage. Tracking sub-tasks carry no labels at all, so gating on needs-triage / bug / feature proposal excludes them. This is a workflow-level if:, so excluded issues cost no API calls.

Results after the fixes

313 of the 500 issues pass the label gate. At the 0.62 default, 10 pairs across 7 issues (2.2% of the corpus) are flagged:

Cluster	Pairs	Verdict
#11646 / #11647 / #11648 / #11649 - French keyboard special characters	6	True duplicate
#11467 / #11469 - PipsPager crash on resize inside Viewbox	1	True duplicate
#11039 / #11040 - ScrollView layout cycle	1	True duplicate
#10866 / #10887 - Rotation on AnimatedIcon causes WMC0612	1	True duplicate
#11065 / #11248 - AOT exception with a custom control	1	True duplicate

No false positives at any threshold from 0.55 up.

These are corroborated by what maintainers already did: in four of the five clusters the redundant copies are closed while one remains open, and #11646 was closed with the comment "Since this is a duplicate ticket, we are closing it." The PipsPager pair (#11467 / #11469) is still open and undetected today, which is the case the bot is meant to catch.

Threshold sweep on the gated corpus:

Threshold	Pairs	Issues flagged
0.55	11	8 (2.6%)
0.62 (default)	10	7 (2.2%)
0.70	9	6 (1.9%)
0.80	8	5 (1.6%)

The only pair between 0.55 and 0.62 is #11197 "Test Bug" against #11032 "test", which is correctly left below the cutoff.

Duplicate suggestion chip

Adopted from the PowerToys issue-triage workflow. For High-confidence matches only, the workflow now submits GitHub's native duplicate suggestion so a maintainer can accept and close from the chip:

PATCH /repos/{owner}/{repo}/issues/{issue_number}
X-GitHub-Api-Version: 2026-03-10

{
  "state": { "value": "closed", "rationale": "...", "confidence": "HIGH", "suggest": true },
  "state_reason": "duplicate",
  "duplicate_issue_id": <id of the canonical issue>
}

suggest: true holds the close for human review rather than applying it. Since this is a preview API, the step checks the response and, if a close was actually applied and is attributable to this request, reopens the issue and fails the run. ENABLE_DUPLICATE_SUGGESTION turns the chip off for comment-only operation.

Verification status: in a fork test the PATCH was accepted and the issue correctly stayed open. However, neither the REST timeline nor GraphQL exposes a pending suggestion, so I could not confirm from the CLI that the chip actually renders. That likely needs the duplicate-detection preview enabled on the repository. Worth a visual check by someone with the preview enabled before relying on it; the comment path is unaffected either way.

Also changed

• Removed the label write. The workflow no longer creates or applies possible-duplicate, or any label. It only comments, plus the optional High-confidence chip.
• Test suite is now 40 tests, covering prefix stripping, the epic false-positive case, and the new outputs.

End-to-end

Verified in Jaylyn-Barbee/microsoft-ui-xaml: issue #9 received one comment ranking #7, #8, and #5 as High confidence, the suggestion was submitted against #7, and the issue remained open. Unlabeled issues are skipped by the gate as intended.