Skip to content

feat: add test-pr skill for testing PRs across downstream repos - #201

Draft
lengau wants to merge 7 commits into
mainfrom
work/testing-skill
Draft

lengau wants to merge 7 commits into
mainfrom
work/testing-skill

Conversation

@lengau

@lengau lengau commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

This PR introduces the test-pr agent skill in .github/skills/test-pr/ to automate downstream CI integration testing across all team repositories.

Downstream Test PRs

Repository Draft PR Test Workflow Runs
canonical/charmcraft #2954 Check policy, QA, Check policy, QA, Running Copilot Code Review, Schema, Check policy, QA
canonical/craft-application #1226 Copilot Setup Steps, Check policy, QA, QA, Check policy, Running Copilot Code Review, QA, Spread, Check policy
canonical/craft-archives #266 Check policy, QA, Check policy, QA, Running Copilot Code Review, QA, Check policy
canonical/craft-cli #503 QA, Check policy, Check policy, QA, QA, Check policy, Running Copilot Code Review, QA, Check policy
canonical/craft-grammar #224 Check policy, QA, Check policy, QA, Running Copilot Code Review, QA, Check policy
canonical/craft-parts #1771 Check policy, QA, QA, Check policy, Running Copilot Code Review, Check policy, QA
canonical/craft-platforms #281 QA, Check policy, Security scan, Running Copilot Code Review, QA, Check policy, Security scan
canonical/craft-providers #1025 Check policy, QA, weekly-tests, Running Copilot Code Review, weekly-tests, QA, Check policy
canonical/craft-store #443 Check policy, QA, Running Copilot Code Review, QA, Check policy
canonical/debcraft #207 Check policy, QA, Running Copilot Code Review, Schema, QA, Check policy
canonical/fetch-service #667 Sync to Launchpad, Security scan, Running Copilot Code Review, Tests, cla-check, Security scan, Sync to Launchpad
canonical/imagecraft #500 Check policy, QA, Running Copilot Code Review, Check policy, Run a spread test of the generated snap, Snap, QA
canonical/pydantic-kitbash #181 QA, Check policy, Running Copilot Code Review, Zizmor, QA, Check policy
canonical/rockcraft #1390 Check policy, QA, Pull Request Labeler, Running Copilot Code Review, Check policy, QA, Spread tests for source, Spread (manual - large/flaky), Schema, Pull Request Labeler
canonical/snapcraft #6496 Running Copilot Code Review, Check policy, QA
canonical/starbase #634 Copilot Setup Steps, Running Copilot Code Review, Check policy, QA

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings October 15, 2026 00:50
…m repos

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Moderate issues remain in JSON output, fork/PR ref handling, discovery parsing, cleanup, and workflow-run selection.

4 open findings
What changed in this PR

Adds the test-pr skill and helper script for downstream repository integration testing.

Changes:

  • Discovers downstream repositories and rewrites workflow references.
  • Creates draft test PRs and reports workflow runs.
  • Supports cleanup, dry-run modes, and documented usage.
File Description
.github/​skills/​test-pr/​test-downstream.py Implements downstream testing automation.
.github/​skills/​test-pr/​SKILL.md Documents the skill, prerequisites, and commands.

🧠 Review effort: Lite


💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/skills/test-pr/test-downstream.py Outdated
Comment thread .github/skills/test-pr/test-downstream.py Outdated
Comment thread .github/skills/test-pr/test-downstream.py Outdated
Comment thread .github/skills/test-pr/test-downstream.py
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
lengau and others added 2 commits October 14, 2026 21:03
… test-pr skill

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
lengau and others added 2 commits October 14, 2026 21:13
…iles

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Use jq line emission for paginated repository discovery
- Default --owner to None to allow detected fork owner fallback
- Resolve headRefName and owner when --pr is specified from another branch
- Route progress and log messages to stderr when --json is enabled

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@mr-cal mr-cal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about retrieving the results of the jobs and putting them in the table?

For example, if I update the policy workflow, I'd like to see that all policy workflows pass in the table, without having to open a dozen or more workflows. If it only fails on 1 or 2 repos, it's quicker to know exactly which workflow logs to load.

@lengau

lengau commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

@mr-cal I agree - I need to break it into separate steps though, as it was originally waiting for CI to complete before even making the table.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants