ci: add a Buildkite flaky-test issue bot by zhuangzhewei09 · Pull Request #5381 · ray-project/kuberay

zhuangzhewei09 · 2026-10-06T12:11:00Z

Why are these changes needed?

Add a daily/manual GitHub Actions bot that directly queries Buildkite's Tests
API with labels=flaky. Buildkite identifies flaky tests; this bot only opens
issues for tests without an existing open issue.

Add only .github/workflows/flaky-tracker.yaml, using inline Bash with
curl, jq, and gh. No Python scripts, committed tests, documentation,
changes under ci/, or changes to existing workflows.
Reuse secrets.BUILDKITE_API_TOKEN in GitHub Actions, not Buildkite jobs.
Target organization ray-project and suite kuberay-e2e, confirmed
by the successful Test Engine upload response in the RayCluster/GCS job of
Buildkite #17419.
Paginate both flaky tests and open GitHub issues. Deduplicate using
buildkite-test-id: <ID> in the issue body, without relying on search indexing.
Create issues with the existing flaky label and a Test Engine link.
Existing issues are not updated or closed.
Default manual runs to dry-run and serialize runs. Scheduled writes remain
opt-in through the repository variable KUBERAY_FLAKY_TRACKER_ENABLED=true.

Real token/suite access still requires a trusted GitHub Actions dry-run.

Related issue number

Refs #5064. Separate bot PR following #5169; live rollout is still pending.

Labels

If this PR has user-facing changes that require documentation updates at release time, I have added the doc-updates-required label.
If this PR contains breaking changes, I have added the breaking-change label.

Neither applies: internal CI automation with no breaking changes.

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

Local validation: YAML parsing, Bash syntax, actionlint, and whitespace checks
passed. Credential-free local mocks covered multiple Buildkite pages, duplicate
test IDs, existing open issues, dry-run, creation, and HTTP failures. No live
API writes were performed, and no verification scripts were added to the PR.

Manual test instructions

Once the workflow is available on the upstream default branch, manually run
it with apply=false to validate real API access and review planned creations.
Leave scheduled runs disabled during validation.
After reviewing the dry-run, manually use apply=true for issue creation.
Enable daily runs with KUBERAY_FLAKY_TRACKER_ENABLED=true when ready.

cursor

Stale Bugbot comment from a previous run.

machichima · 2026-10-06T16:23:01Z

We can directly get the flaky test through flaky test API right? https://buildkite.com/docs/apis/rest-api/test-engine/tests, So we only need a new yaml for the bot of creating issue if there's a new flaky test.

BUILDKITE_API_TOKEN should be available in github workflow and not in buildkite jobs

Run the bot through GitHub Actions with the existing BUILDKITE_API_TOKEN. Query tests labeled flaky in the ray-project/kuberay-e2e Test Engine suite, whose slugs are confirmed by a successful CI upload response. Use actions/github-script for GitHub pagination, issue creation, and updates. Deduplicate open issues by test identity and preserve human notes outside the managed report. Keep manual runs read-only by default and scheduled writes opt-in. Cover the workflow script with credential-free tests. Validate real API access through a trusted GitHub Actions dry-run, not Buildkite jobs or local permission probes. Refs ray-project#5064 Signed-off-by: zhuangzhewei09 <zhuangzhewei09@dingtalk.com>

Run the bot through GitHub Actions with the existing BUILDKITE_API_TOKEN. Query tests labeled flaky in the ray-project/kuberay-e2e Test Engine suite, whose organization and suite slugs are confirmed by a CI upload response. Keep the implementation in scripts/flaky_tracker.py with Python standard library HTTP handling and gh api for GitHub pagination and issue writes. Deduplicate open issues by test identity and preserve human notes outside the managed report. Default manual runs to dry-run and keep scheduled writes opt-in. Cover the bot and its transports with credential-free Python tests. Real token and suite access still require a trusted GitHub Actions dry-run; Buildkite jobs and local permission probes are not used for validation. Refs ray-project#5064 Signed-off-by: zhuangzhewei09 <zhuangzhewei09@dingtalk.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 668372d. Configure here.}

Run the bot through GitHub Actions with the existing BUILDKITE_API_TOKEN. Query tests labeled flaky in the ray-project/kuberay-e2e Test Engine suite, whose organization and suite slugs are confirmed by a CI upload response. Keep the implementation in scripts/flaky_tracker.py with Python standard library HTTP handling and gh api for GitHub pagination and issue writes. Deduplicate open issues by test identity and preserve human notes outside the managed report. Default manual runs to dry-run and keep scheduled writes opt-in. Use only existing repository labels when creating issues and log missing labels without provisioning them. Keep dry-runs read-only and avoid label discovery for update-only runs. Cover the bot and its transports with credential-free Python tests. Real token and suite access still require a trusted GitHub Actions dry-run; Buildkite jobs and local permission probes are not used for validation. Refs ray-project#5064 Signed-off-by: zhuangzhewei09 <zhuangzhewei09@dingtalk.com>

zhuangzhewei09 · 2026-10-12T09:10:56Z

Thanks @machichima. Updated to the single-YAML approach you suggested: inline Bash queries the Tests API with labels=flaky for ray-project/kuberay-e2e, using BUILDKITE_API_TOKEN only in GitHub Actions.

It paginates tests and open issues, skips test IDs already tracked by an open issue, and creates new issues with the flaky label and test link. The Python helper and tests are removed, with no changes to existing workflows, ci/, or Buildkite jobs. Manual runs default to dry-run; scheduled writes require KUBERAY_FLAKY_TRACKER_ENABLED=true.

Run a daily/manual GitHub Actions bot in a single YAML workflow. Query the Buildkite Tests API for tests labeled flaky on the default branch in the ray-project/kuberay-e2e suite using BUILDKITE_API_TOKEN. Use inline Bash with curl, jq, and gh to paginate tests and open issues. Deduplicate by Buildkite test ID and create issues with the flaky label and a Test Engine link. Skip existing open issues without modifying them. Default manual runs to dry-run and serialize workflow runs. Keep scheduled writes opt-in through KUBERAY_FLAKY_TRACKER_ENABLED. Remove the Python helper, its unit tests, and changes to the existing test workflow. Validate syntax, actionlint, and credential-free local mocks for pagination, deduplication, dry-run, issue creation, and HTTP failures. Real token access still requires a trusted GitHub Actions dry-run. Refs ray-project#5064 Signed-off-by: zhuangzhewei09 <zhuangzhewei09@dingtalk.com>

machichima

I just found that the Tests API only returns the test’s URL on Buildkite Test Engine, which currently is not public for contributors to view.

https://buildkite.com/docs/apis/rest-api/test-engine/tests

So contributors would need to copy the test name and search Buildkite manually, which makes me wonder whether this is beneficial at all. cc @win5923 WDYT?

machichima · 2026-10-14T04:38:29Z

+    if: >-
+      github.repository == 'ray-project/kuberay' &&
+      github.ref == format('refs/heads/{0}', github.event.repository.default_branch) &&
+      (github.event_name == 'workflow_dispatch' || vars.KUBERAY_FLAKY_TRACKER_ENABLED == 'true')


I don't think we need this? For CI YAML setup I think you can reference .github/workflows/stale.yaml

machichima · 2026-10-14T04:44:17Z

+          BUILDKITE_API_TOKEN: ${{ secrets.BUILDKITE_API_TOKEN }}
+          GH_TOKEN: ${{ github.token }}
+          GH_REPO: ${{ github.repository }}
+          FLAKY_TRACKER_BRANCH: ${{ github.event.repository.default_branch }}


We can hard code master here

machichima · 2026-10-14T04:45:33Z

+          GH_TOKEN: ${{ github.token }}
+          GH_REPO: ${{ github.repository }}
+          FLAKY_TRACKER_BRANCH: ${{ github.event.repository.default_branch }}
+          FLAKY_TRACKER_APPLY: ${{ github.event_name == 'schedule' || inputs.apply }}


Let's remove the dry-run switch, we do not need it

machichima · 2026-10-14T04:54:35Z

+          while true; do
+            batch=$(curl --fail --silent --show-error --max-time 60 --get \
+              -H "Authorization: Bearer $BUILDKITE_API_TOKEN" \
+              --data-urlencode 'labels=flaky' \
+              --data-urlencode "branch=$FLAKY_TRACKER_BRANCH" \
+              --data-urlencode 'per_page=100' --data-urlencode "page=$page" "$suite/tests")
+            jq -e 'type == "array" and all(.[];
+              (.id | type == "string" and test("^[A-Za-z0-9_-]+$")) and
+              (.name | type == "string" and length > 0))' <<< "$batch" > /dev/null
+            tests=$(jq -cn --argjson tests "$tests" --argjson batch "$batch" '$tests + $batch')
+            if [ "$(jq 'length' <<< "$batch")" -lt 100 ]; then
+              break
+            fi
+            page=$((page + 1))
+          done


Based on the docs, I think the curl command can be much shorter: https://buildkite.com/docs/apis/rest-api/test-engine/tests

curl -H "Authorization: Bearer $TOKEN" \ -H "Buildkite-Version: 2026-08-01" \ -X GET "https://api.buildkite.com/v2/analytics/organizations/{org.slug}/suites/{suite.slug}/tests?labels=flaky&sort_by=reliability&order=asc"

Could you please explain why we need a long command here?

Also we will need to add Buildkite-Version: 2026-08-01 header

machichima · 2026-10-14T07:21:59Z

+            page=$((page + 1))
+          done
+
+          # Fetch all open issues once, avoiding search indexing delays and result limits.


I think we can list the issue with "flaky-tracker" label. Like this

gh issue list --state open --label flaky-tracker --limit 1000 --json body --jq '.[].body' > issues.txt jq -c '.[]' tests.json | while read -r t; do url=$(jq -r .web_url <<< "$t") grep -qF "$url" issues.txt && continue gh issue create --label flaky-tracker,ci,flaky,P0,triage \ --title "[Flaky test] $(jq -r .name <<< "$t")" --body "$url" done

machichima · 2026-10-15T02:25:47Z

I think a workaround would be to take the test UUID from the Tests API, then list the failed runs and filter them by the test UUID.

@zhuangzhewei09 Do you mind if I take over this work? I’ll list you as a co-author in my PR.

zhuangzhewei09 mentioned this pull request Oct 6, 2026

[Feature] Bot to open issue for flaky tests #5064

Open

zhuangzhewei09 force-pushed the feat/flaky-bot-api branch from 2aba236 to 221db11 Compare October 6, 2026 13:30

zhuangzhewei09 changed the title ~~ci: add Buildkite Test Engine query CLI~~ ci: add a Buildkite flaky-test issue bot Oct 6, 2026

cursor Bot reviewed Oct 6, 2026

View reviewed changes

Comment thread ci/flaky_tracker/github.py Outdated

Comment thread ci/flaky_tracker/cli.py Outdated

zhuangzhewei09 force-pushed the feat/flaky-bot-api branch from 221db11 to 16c1c5a Compare October 6, 2026 15:19

zhuangzhewei09 force-pushed the feat/flaky-bot-api branch from 16c1c5a to c240abf Compare October 7, 2026 05:04

zhuangzhewei09 force-pushed the feat/flaky-bot-api branch from c240abf to 6c5215d Compare October 7, 2026 05:21

machichima reviewed Oct 7, 2026

View reviewed changes

Comment thread ci/flaky-tracker.js Outdated

zhuangzhewei09 force-pushed the feat/flaky-bot-api branch from 6c5215d to 668372d Compare October 7, 2026 05:44

cursor Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread scripts/flaky_tracker.py Outdated

zhuangzhewei09 force-pushed the feat/flaky-bot-api branch from 668372d to 92bc283 Compare October 12, 2026 09:06

zhuangzhewei09 force-pushed the feat/flaky-bot-api branch from 92bc283 to 5304c42 Compare October 13, 2026 10:18

machichima reviewed Oct 14, 2026

View reviewed changes

Conversation

zhuangzhewei09 commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Labels

Checks

Manual test instructions

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

machichima commented Oct 6, 2026

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zhuangzhewei09 commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

machichima left a comment

Choose a reason for hiding this comment

Uh oh!

machichima Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

machichima Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

machichima Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

machichima Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

machichima Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

machichima commented Oct 15, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zhuangzhewei09 commented Oct 6, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

zhuangzhewei09 commented Oct 12, 2026 •

edited

Loading