Skip to content

Add a tier of GitHub API tests that need a token but not a full issue crawl #114

Description

@gaurav

The gap

We have two kinds of GitHub issue test today and nothing in between:

  • -m unit (tests/github_issues/test_system.py) — builds MagicMock issues and never
    touches the API. Fast, runs in CI, but exercises none of our assumptions about what the
    GitHub API actually returns.
  • pytest tests/github_issues — the real thing: a search-backed crawl of every repo in
    targets.ini's Repositories, currently ~96 issues across five repos, and then NodeNorm
    and NameRes calls for every assertion found. Takes ~95s and needs both a token and two
    live services.

So the GitHub integration itself is only ever covered by the expensive path. Worse, without
a token the issue tests skip at module level rather than failing, so a run — including a
CI run — can go green having tested none of it.

What a middle tier would cover

Tests that need a token but not a crawl, and no NodeNorm/NameRes at all:

  • issue_id() against a real html_url, and the assumption behind it — that a search result
    carries body and html_url so we spend no core requests re-hydrating (see the comment on
    issue_id() and the rate-limit table in README.md).
  • get_issues_by_ids() resolving all three ID formats (org/repo#N, repo#N, N) against
    the real API.
  • The configured-repositories allowlist added in b0f5e43's series — that an unconfigured
    org/repo#N is refused, and a configured one still resolves.
  • The 401/expired-token path in tests/github_issues/conftest.py, which currently only ever
    runs when someone's token happens to have expired.

Fetching one or two issues by ID is a couple of core requests against a 5,000/hour budget,
versus the search endpoint's 30/minute.

Things to decide

  • How to select them. A new marker (github_api?) seems natural, so -m unit stays
    offline and -m github_api is the token-requiring tier. Worth checking this composes with
    the existing deselected_by_markexpr() collection-time short-circuit.
  • How to avoid the silent-skip trap. These tests exist precisely because skipping looks
    like passing. They should probably fail when selected without a token, rather than skip —
    the opposite of what the current issue tests do.
  • What to pin them against. Asserting on live issues in NCATSTranslator/Babel means
    edits elsewhere break our tests. A dedicated fixture issue in this repo, with known
    BabelTest syntax in its body and a note in it saying it is a test fixture, would be stable —
    but its body then has to stay in sync with whatever the tests assert.
  • Whether NodeNorm/NameRes stay out. Keeping them out is what makes this tier cheap and
    keeps it honest about what it covers, but it does mean the tier can't catch an
    assertion-evaluation regression.

CI

.github/workflows/tests.yaml already has the recipe in a comment: add issues: read to the
permissions block and pass GITHUB_TOKEN: ${{ github.token }}. That token covers this
repository only — the other repos in Repositories are readable because they are public, not
because the token is scoped to them — which is another argument for pinning against a fixture
issue here rather than in NCATSTranslator/Babel.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions