The gap
We have two kinds of GitHub issue test today and nothing in between:
-m unit (tests/github_issues/test_system.py) — builds MagicMock issues and never
touches the API. Fast, runs in CI, but exercises none of our assumptions about what the
GitHub API actually returns.
pytest tests/github_issues — the real thing: a search-backed crawl of every repo in
targets.ini's Repositories, currently ~96 issues across five repos, and then NodeNorm
and NameRes calls for every assertion found. Takes ~95s and needs both a token and two
live services.
So the GitHub integration itself is only ever covered by the expensive path. Worse, without
a token the issue tests skip at module level rather than failing, so a run — including a
CI run — can go green having tested none of it.
What a middle tier would cover
Tests that need a token but not a crawl, and no NodeNorm/NameRes at all:
issue_id() against a real html_url, and the assumption behind it — that a search result
carries body and html_url so we spend no core requests re-hydrating (see the comment on
issue_id() and the rate-limit table in README.md).
get_issues_by_ids() resolving all three ID formats (org/repo#N, repo#N, N) against
the real API.
- The configured-repositories allowlist added in b0f5e43's series — that an unconfigured
org/repo#N is refused, and a configured one still resolves.
- The 401/expired-token path in
tests/github_issues/conftest.py, which currently only ever
runs when someone's token happens to have expired.
Fetching one or two issues by ID is a couple of core requests against a 5,000/hour budget,
versus the search endpoint's 30/minute.
Things to decide
- How to select them. A new marker (
github_api?) seems natural, so -m unit stays
offline and -m github_api is the token-requiring tier. Worth checking this composes with
the existing deselected_by_markexpr() collection-time short-circuit.
- How to avoid the silent-skip trap. These tests exist precisely because skipping looks
like passing. They should probably fail when selected without a token, rather than skip —
the opposite of what the current issue tests do.
- What to pin them against. Asserting on live issues in
NCATSTranslator/Babel means
edits elsewhere break our tests. A dedicated fixture issue in this repo, with known
BabelTest syntax in its body and a note in it saying it is a test fixture, would be stable —
but its body then has to stay in sync with whatever the tests assert.
- Whether NodeNorm/NameRes stay out. Keeping them out is what makes this tier cheap and
keeps it honest about what it covers, but it does mean the tier can't catch an
assertion-evaluation regression.
CI
.github/workflows/tests.yaml already has the recipe in a comment: add issues: read to the
permissions block and pass GITHUB_TOKEN: ${{ github.token }}. That token covers this
repository only — the other repos in Repositories are readable because they are public, not
because the token is scoped to them — which is another argument for pinning against a fixture
issue here rather than in NCATSTranslator/Babel.
The gap
We have two kinds of GitHub issue test today and nothing in between:
-m unit(tests/github_issues/test_system.py) — buildsMagicMockissues and nevertouches the API. Fast, runs in CI, but exercises none of our assumptions about what the
GitHub API actually returns.
pytest tests/github_issues— the real thing: a search-backed crawl of every repo intargets.ini'sRepositories, currently ~96 issues across five repos, and then NodeNormand NameRes calls for every assertion found. Takes ~95s and needs both a token and two
live services.
So the GitHub integration itself is only ever covered by the expensive path. Worse, without
a token the issue tests skip at module level rather than failing, so a run — including a
CI run — can go green having tested none of it.
What a middle tier would cover
Tests that need a token but not a crawl, and no NodeNorm/NameRes at all:
issue_id()against a realhtml_url, and the assumption behind it — that a search resultcarries
bodyandhtml_urlso we spend no core requests re-hydrating (see the comment onissue_id()and the rate-limit table inREADME.md).get_issues_by_ids()resolving all three ID formats (org/repo#N,repo#N,N) againstthe real API.
org/repo#Nis refused, and a configured one still resolves.tests/github_issues/conftest.py, which currently only everruns when someone's token happens to have expired.
Fetching one or two issues by ID is a couple of core requests against a 5,000/hour budget,
versus the search endpoint's 30/minute.
Things to decide
github_api?) seems natural, so-m unitstaysoffline and
-m github_apiis the token-requiring tier. Worth checking this composes withthe existing
deselected_by_markexpr()collection-time short-circuit.like passing. They should probably fail when selected without a token, rather than skip —
the opposite of what the current issue tests do.
NCATSTranslator/Babelmeansedits elsewhere break our tests. A dedicated fixture issue in this repo, with known
BabelTest syntax in its body and a note in it saying it is a test fixture, would be stable —
but its body then has to stay in sync with whatever the tests assert.
keeps it honest about what it covers, but it does mean the tier can't catch an
assertion-evaluation regression.
CI
.github/workflows/tests.yamlalready has the recipe in a comment: addissues: readto thepermissionsblock and passGITHUB_TOKEN: ${{ github.token }}. That token covers thisrepository only — the other repos in
Repositoriesare readable because they are public, notbecause the token is scoped to them — which is another argument for pinning against a fixture
issue here rather than in
NCATSTranslator/Babel.