Typed Python stubs: mypy-protobuf .pyi + hash-locked generators (#158)

[craigmcchesney]

Implements #158 (step 2 of the plan's cross-repo sequencing). Step 1, the dp-python-lib preparation PR osprey-dcs/dp-python-lib#60, has already merged.

Changes

• generate-python-stubs.yml
  • Installs from a hash-locked requirements file (pip install --require-hashes) instead of unversioned grpcio grpcio-tools protobuf. The unhashed pip install --upgrade pip is dropped too.
  • Adds --mypy_out / --mypy_grpc_out, and widens the import fixup find to *.pyi.
  • New guard step: fails if any file still has an absolute import of a generated module in either form (import x_pb2 / from x_pb2 import), or if any .py has no matching .pyi, or if nothing was generated.
  • Uploads out/python as the python-stubs artifact on every run, using release.yml's pinned upload-artifact SHA. The dry-run summary counts .py and .pyi separately.
• tools/python-stubs-requirements.in / .txt: pins grpcio-tools==1.84.0 and mypy-protobuf==5.1.0. The .txt is pip-compile --generate-hashes --allow-unsafe output. --allow-unsafe is needed because grpcio-tools depends on setuptools. The .in header records the regeneration command, the dp-python-lib coordination rule, and why the file is not Dependabot-managed.
• doc/cookbook/python-stubs.md: updated publishing steps, generation commands, and fixup, plus a new Type checking section.
• doc/release-notes/NEXT.md: a Generate typed Python stubs (.pyi) in generate-python-stubs.yml #158 section, plus a new first item in "Cutting the release". It confirms dp-python-lib#60 is still merged, then checks dp-python-lib (mypy src/ and its snippet checker) against the dry-run artifact before tagging.
• CLAUDE.md: the tools/ layout line.

Verified locally

• A fresh venv installs the lock under --require-hashes, and generates with the lock plus the workflow's flags. Against that output:

  • it is identical to the stubs dp-python-lib#60 was verified against (mypy src/ and the snippet checker);
  • all 10 .py files are byte-identical to dp-python-lib main.

• Every locked package supports Python 3.11 and has a linux x86_64 wheel. The recorded hashes match PyPI. The lock was compiled on 3.12 and the workflow runs 3.11, and no environment markers are involved.

• The guard step, extracted from the workflow and run under bash, gave:

  Case	Exit
  Clean tree	0
  Unfixed import common_pb2 as in a .pyi	1
  Injected from common_pb2 import …	1
  Injected import common_pb2_grpc	1
  Deleted .pyi	1
  Empty tree	1

• mvn package and tools/check-cookbook-snippets.py pass. actionlint's only findings are three SC2086 infos already present on main (the "Determine sync mode" step).

Dry run on this branch (plan work-breakdown step 7)

Run 36068796341 was dispatched from this branch with dry_run: true. It succeeded, and every dp-python-lib step was skipped. Checks on its output:

• The python-stubs artifact downloads, with 10 .py and 10 .pyi files. It is byte-identical to the local locked output above.

• All 10 .py files are byte-identical to dp-python-lib main.

• Every .pyi import between generated modules is relative (from . import …), with zero absolute ones.

• dp-python-lib origin/main (f856b94, which includes rename bidirectional streaming ingestion method #60), with the artifact's .pyi files dropped in:

  Config	mypy src/	Cookbook snippet checker
  As configured	no issues, 24 files	OK, 107 snippets
  Suppression removed (exclude + follow_imports = "skip")	no issues, 34 files	OK, 107 snippets

  The plan expected only zero name-defined / attr-defined here. The hand-written errors are gone as well, because rename bidirectional streaming ingestion method #60 fixed them. The unsuppressed result is clean partly because _stub is still Any in dp-python-lib. Typing it is step 5.

Nothing reaches dp-python-lib until the next rel-* tag (step 4). The new first item in NEXT.md's "Cutting the release" repeats this check before that tag is pushed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01We3Lmjj8XNLK1zpHRr1AwQ

[Copilot]

Copilot review overview

🔵 Needs a closer look

Workflow-level verification remains pending, and regeneration instructions are inconsistent.

Review effort: Lite
Findings: 1

Open (1)

• Regeneration command conflicts with the lockfile header · New

What changed in this PR

Adds reproducible, hash-locked mypy-protobuf generation and publishes typed Python stubs.

Changes:

• Pins and hash-locks generator dependencies.
• Generates and validates .pyi stubs, then uploads them as an artifact.
• Updates documentation, release notes, and repository guidance.

File	Summary
tools/​python-stubs-requirements.txt	Hash-locked dependencies. Nit (2 votes): reconcile the --no-index regeneration command with the .in instructions.
tools/​python-stubs-requirements.in	Defines generator pins and regeneration guidance.
doc/​release-notes/​NEXT.md	Adds release and cross-repository sequencing guidance.
doc/​cookbook/​python-stubs.md	Documents stub generation and type checking.
CLAUDE.md	Updates the tools directory documentation.
.github/​workflows/​generate-python-stubs.yml	Generates, validates, and uploads typed stubs. Moderate (1 vote): complete the pending dry_run: true workflow and downstream verification.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.