dp-python-lib/README.env at main · osprey-dcs/dp-python-lib · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
dp-python-lib — Release Artifacts and Verification
==================================================

This repository publishes versioned dp-python-lib Python distributions via
GitHub Releases and, from the first release after rel-1.16.0, PyPI.  Each
release corresponds to a Git tag of the form:

  rel-<version>

Example:
  rel-1.16.0

------------------------------------------------------------
Release Contents
------------------------------------------------------------

Each release contains:

  dp_python_lib-<version>-py3-none-any.whl
    The wheel.  This is what you install.

  dp_python_lib-<version>.tar.gz
    The source distribution.

  SHA256SUMS
    SHA-256 checksums for the two files above, in sha256sum format.

  dp_python_lib-<version>-py3-none-any.whl.sigstore.json
  dp_python_lib-<version>.tar.gz.sigstore.json
  SHA256SUMS.sigstore.json
    Keyless Sigstore signature bundles, one per file above.

Releases before rel-1.16.0 may lack SHA256SUMS or the Sigstore bundles; the
verification steps below do not apply to them.

------------------------------------------------------------
Download and Installation
------------------------------------------------------------

1. Download the artifacts from the GitHub Release page into a single directory,
   with no subdirectories.  Both commands below expect to find the files side
   by side.  With the GitHub CLI:

     gh release download rel-<version> -R osprey-dcs/dp-python-lib

2. Verify the checksums:

     sha256sum -c SHA256SUMS

   (On macOS without GNU coreutils: shasum -a 256 -c SHA256SUMS)

   The output should indicate:

     dp_python_lib-<version>-py3-none-any.whl: OK
     dp_python_lib-<version>.tar.gz: OK

   SHA256SUMS lists both distributions, so this fails if you downloaded only
   the wheel.  To check the files you actually have and ignore the rest:

     sha256sum --ignore-missing -c SHA256SUMS

   Note that --ignore-missing succeeds if it checked nothing at all, so confirm
   the file you care about is listed as OK rather than relying on the exit code.

3. Verify the signatures.

   The checksums establish integrity but not origin: they are published to the
   same release page as the artifacts, so anyone able to replace a file could
   replace its checksum alongside it.  The Sigstore signatures close that gap by
   binding each file to the repository, workflow file, and tag that produced
   it.  There is no public key to fetch or trust: the signing identity is a
   short-lived certificate issued to the GitHub Actions run and recorded in the
   public Rekor transparency log.

   Install the Sigstore client:

     pip install sigstore

   Then, substituting the release's tag in both places:

     sigstore verify identity \
       --cert-identity "https://github.com/osprey-dcs/dp-python-lib/.github/workflows/release.yml@refs/tags/rel-<version>" \
       --cert-oidc-issuer "https://token.actions.githubusercontent.com" \
       dp_python_lib-*.whl dp_python_lib-*.tar.gz SHA256SUMS

   Each file's bundle is found automatically as <file>.sigstore.json.  The
   output should be one line per file:

     OK: dp_python_lib-<version>-py3-none-any.whl
     OK: dp_python_lib-<version>.tar.gz
     OK: SHA256SUMS

   The --cert-identity is an exact match, pinned to this repository, this
   workflow file, and one tag.  A signature made by any other workflow, any
   other repository, or the run for a different tag fails with
   "Certificate's SANs do not match".  Do not loosen it to make a failure go
   away; a mismatch is the check doing its job.

4. Install the wheel:

     pip install dp_python_lib-<version>-py3-none-any.whl

   Optional extras are installed the usual way, e.g.:

     pip install "dp_python_lib-<version>-py3-none-any.whl[analysis]"

------------------------------------------------------------
Installing from PyPI
------------------------------------------------------------

The usual install is from PyPI:

  pip install dp-python-lib
  pip install "dp-python-lib[analysis]"

(Quote the extra: zsh, the macOS default shell, otherwise treats the brackets
as a glob.)

The wheel and sdist on PyPI are the same files, byte for byte, as those on the
GitHub Release.  The release workflow uploads the files it signed, and only
after the GitHub Release is published; it then fetches the digests PyPI
reports and fails unless they match SHA256SUMS exactly.  So a file from PyPI
can be verified with the release's SHA256SUMS and Sigstore bundles:

1. Download the wheel from PyPI, without its dependencies:

     pip download --no-deps "dp-python-lib==<version>"

2. Into the same directory, download SHA256SUMS and the bundles from the
   GitHub Release:

     gh release download rel-<version> -R osprey-dcs/dp-python-lib \
       -p SHA256SUMS -p '*.sigstore.json'

3. Check the wheel's checksum.  SHA256SUMS also lists the sdist, which is not
   here, so --ignore-missing is needed; confirm the wheel is reported OK:

     sha256sum --ignore-missing -c SHA256SUMS

   (On macOS without GNU coreutils: shasum -a 256 --ignore-missing -c SHA256SUMS)

4. Verify the signatures of the wheel and SHA256SUMS as in step 3 of
   "Download and Installation", naming only those two files:

     sigstore verify identity \
       --cert-identity "https://github.com/osprey-dcs/dp-python-lib/.github/workflows/release.yml@refs/tags/rel-<version>" \
       --cert-oidc-issuer "https://token.actions.githubusercontent.com" \
       dp_python_lib-*.whl SHA256SUMS

Without --no-deps, pip download also fetches every dependency, none of which
SHA256SUMS lists.  Without --ignore-missing, sha256sum reports the sdist
missing and exits non-zero.

PyPI also holds PEP 740 attestations for each file, made by the same workflow
through PyPI's Trusted Publishing.  They are a second, independent record of
where the files came from, shown on each file's page on pypi.org; the steps
above do not depend on them.

------------------------------------------------------------
Release Notes
------------------------------------------------------------

The body of each GitHub Release is the version-controlled file
doc/release-notes/rel-<version>.md, published verbatim.  Each carries a short
"Verifying these artifacts" section with that release's exact identity; this
file is the fuller reference it points to.

------------------------------------------------------------
Notes
------------------------------------------------------------

- dp-python-lib is a client library; it does not run as a standalone service.
- It talks to the MLDP services implemented in dp-service, over the gRPC API
  defined in dp-grpc.
- PyPI and GitHub Releases carry the same files; see "Installing from PyPI"
  above for how that is checked.