Repository navigation
feat: add sample status client, conversions, and query filtering - #37
Conversation
The 1.16.0 stub sync (PR #36) regenerated the gRPC stubs with a newer toolchain. The generated code now hard-requires newer runtimes at import time, but the dependency floors were left at the previous values: - *_pb2_grpc.py declare GRPC_GENERATED_VERSION = '1.83.0' and raise a RuntimeError on import under an older grpcio. - *_pb2.py call ValidateProtobufRuntimeVersion(PUBLIC, 7, 35, 1). The pins still allowed grpcio 1.82.1 / protobuf 7.35.0, so a resolver picking the floor produced an install that fails on the first import: RuntimeError: The grpc package installed is at version 1.82.1, but the generated code in annotation_pb2_grpc.py depends on grpcio>=1.83.0. Bump both runtime floors and the [codegen] grpcio-tools floor to match, as the comment above the dependency block already directs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CvckaDwdLbD29hd23ur2a
Adds the client interface to the MLDP Sample Status API (dp-grpc #139), which assigns int32 status codes to individual PV samples keyed by (pvName, timestamp, domain, layer). Wraps the four live RPCs, following the established client patterns: three-tier error handling, ApiResultBase result objects whose typed accessors return None/[] on error, the query_*/iter_* paging pair raising RuntimeError on a page error, and the streaming internal-yield/public-raise contract from QueryClient. Mounted on the annotation facade as client.annotation.sample_status. The deferred domain-registry methods are deliberately not wrapped; they are reserved placeholders returning "not implemented". Four places needed new patterns rather than the existing ones: - Frame construction is the first genuinely structured client input: a time axis plus parallel arrays. sampling_clock() and timestamp_list() build the two axis forms, and the per-timestamp length rules for statusCodes/confidence/reasons are validated client-side so a mistake surfaces as a ValueError naming the offending PV rather than as a server rejection of the whole batch. timestamp_list() enforces the strictly-increasing rule, comparing nanoseconds and not just seconds. - No criterion-builder class, unlike PvMetadataQuery/ConfigurationQuery. The query filters are plain repeated strings ANDed together with no criterion oneof, so plain lists are the honest representation. - The empty-pvNames wildcard is explicit on delete. The same field is benign discovery on query but removes every PV's statuses on delete, so delete_sample_statuses() requires either pv_names or all_pvs=True and raises when given neither or both -- the destructive wildcard is never reached by omitting an argument. - An all-empty reasons list is dropped rather than sent as empty strings, as the API directs, while a partially-populated one is kept whole. Request building guards limit on `is not None` so a meaningful limit=0 survives (the issue #13 bug shape). Its regression test spies on the assignment, since a proto3 scalar set to 0 is indistinguishable from an unset one by both reflection and wire bytes. 67 unit tests; verified against four targeted mutations (limit truthiness, delete wildcard opt-in, timestamp ordering, empty-reasons handling), each of which fails the suite. Refs #8 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CvckaDwdLbD29hd23ur2a
Expands the compact SampleStatusBucket form (a time axis plus parallel arrays) into per-sample SampleStatusRow objects. SamplingClock expansion is computed in integer nanoseconds (startTime + i * periodNanos). Status-to-sample matching is by exact timestamp at nanosecond precision, and present-day epoch nanoseconds need ~61 bits, so a float-seconds round-trip would produce timestamps that no longer match the samples they label. Absent confidence and reasons surface as None rather than 0.0 or "". Absence means "no assertion" throughout this API, and 0.0 is a legitimate confidence value, so a fabricated default would be indistinguishable from a real one. Refs #8 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CvckaDwdLbD29hd23ur2a
Adds a SampleStatusFilter helper and an optional sample_status_filter parameter on QueryParams, populated into QuerySpec field 4, so a time-series query can be restricted to (or away from) samples carrying particular sample statuses. The selector is built only through include()/exclude(), which makes the MODE_UNSPECIFIED zero value unreachable rather than something a caller falls into by leaving the mode unset. _build_query_spec() is shared with the future bucket-oriented client (#16), where the server rejects sampleStatusSelector -- a bucket cannot be partially filtered without rewriting it. A note at the seam records that a bucket request builder must refuse the parameter rather than copying it through into a request the server would reject. Refs #8 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CvckaDwdLbD29hd23ur2a
Adds doc/cookbook/sample-status.md (all 11 snippets pass the cookbook checker), a cookbook README row, and a CLAUDE.md "Sample Status API" section with Key Files entries. Corrects the two places describing valueStatus as merely unpopulated to name sample status as its replacement. Integration tests probe for the API before running: reachability alone is not enough, because a pre-1.16.0 Annotation Service accepts the connection and then answers every sample status call with UNIMPLEMENTED -- which surfaced as seven assertion failures rather than a skip against the server currently on localhost:50053. Also softens the two places hardcoding "docker compose up -d" as the way to start the ecosystem, since the services can equally be run directly. Refs #8 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CvckaDwdLbD29hd23ur2a
The integration tests now run rather than skip: the exact-timestamp contract, absent-stays-absent, full-replace upsert, and layer independence are all confirmed against a real Annotation Service. Narrows the remaining caveat to what is actually still unverified -- status filtering of query results, which needs ingested sample data to attach to (#17). Refs #8 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CvckaDwdLbD29hd23ur2a
There was a problem hiding this comment.
Pull request overview
This PR adds first-class support for MLDP’s Sample Status API to dp-python-lib, including a new SampleStatusClient, per-sample “bucket → rows” conversions, and optional sample-status-based filtering for v2 sample queries.
Changes:
- Introduces
SampleStatusClientunder the annotation facade (client.annotation.sample_status) with params/types and three-tier error handling. - Adds
sample_status_conversionsutilities to expandSampleStatusBucketresults into flat per-sampleSampleStatusRowobjects with nanosecond-exact timestamp expansion. - Extends
QueryClient/QueryParamswithSampleStatusFilter→QuerySpec.sampleStatusSelectorsupport, and documents/tests the new behavior.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
src/dp_python_lib/client/sample_status_client.py |
New Sample Status feature client + axis builders + request/response wrappers. |
src/dp_python_lib/client/sample_status_conversions.py |
New conversion layer: expand sample-status buckets into per-sample rows with exact-nanos timestamps. |
src/dp_python_lib/client/query_client.py |
Adds SampleStatusFilter helper + wires sampleStatusSelector into _build_query_spec(). |
src/dp_python_lib/client/query_conversions.py |
Documentation update noting valueStatus deprecation in favor of sample status. |
src/dp_python_lib/client/annotation_client.py |
Mounts the new SampleStatusClient on the annotation facade. |
src/dp_python_lib/client/__init__.py |
Exposes sample status client/types/helpers from dp_python_lib.client. |
tests/unit/test_sample_status_client.py |
Unit tests for request building, validation, paging/streaming behavior, and error handling. |
tests/unit/test_sample_status_conversions.py |
Unit tests for exact timestamp expansion and bucket→row conversion semantics. |
tests/unit/test_query_client.py |
Unit tests for SampleStatusFilter and request building with sampleStatusSelector. |
tests/integration/test_sample_status_client_integration.py |
Live-service integration tests for save/query/delete, axis round-trip, and upsert semantics. |
tests/integration/conftest.py |
Minor docstring clarification for integration-test prerequisites. |
doc/cookbook/sample-status.md |
New cookbook recipe documenting labeling, reading back, and query filtering usage. |
doc/cookbook/README.md |
Adds cookbook index entry for sample status recipe. |
doc/cookbook/query.md |
Adds cross-reference to sample status as the replacement for valueStatus. |
CLAUDE.md |
Adds sample status client/conversions overview and usage notes. |
.dev/tools/check-cookbook-snippets.py |
Extends snippet-checker preamble imports to include new sample status symbols. |
pyproject.toml |
Raises grpcio/protobuf (and grpcio-tools) minimum versions to match regenerated stubs. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Four review findings from the PR #37 discussion, plus a test rename. Reject empty time axes in _timestamp_count(). A hand-built DataTimestamps can carry a zero-count SamplingClock or an empty TimestampList, and both report their oneof arm as set, so neither was caught by the unset-axis check. These surfaced as a column-length mismatch naming the PV rather than the real problem. The axis builders already make this unreachable; the guard puts the write path in step with expand_data_timestamps(), which applies the same rule on the read path. Validate sample_status_filter in QueryParams. SampleStatusFilter.include() /exclude() make MODE_UNSPECIFIED unreachable through the helpers, but the parameter accepts any SampleStatusSelector, so a default-constructed one set the field present-but-invalid and was rejected by the server. Checking it at construction names the problem and points at the helpers. Compute expected nanoseconds with integer arithmetic in the integration tests. The float path was exact for present-day microsecond datetimes (it breaks only near the int64 nanosecond limit), so this was not a live flake -- but tests asserting exact nanosecond round-trips should not derive their expectation from a float whose ULP at these epochs is ~238ns. Correct the cookbook's "is None" sentence. client.annotation is what becomes None; reaching through it raises AttributeError rather than yielding None. Fixed in all three annotation cookbook pages, since the wording was shared. Rename test_streaming_request_carries_the_selector_too, which called the unary builder. Coverage was not missing -- iter_query_samples_stream() uses that same builder -- but the name promised something the test did not do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CvckaDwdLbD29hd23ur2a
Review fixes — bbb0caaAll four Copilot findings addressed, plus one rename from review. Replies are in each thread; summary here.
The rename (not from Copilot)That test called Two claims in the PR description that were verified, not assumed
Docs corrected beyond the codeFinding #2 exposed a docs overstatement: CLAUDE.md and the cookbook both said Verification
The remaining caveat is unchanged and still flagged in the recipe and CLAUDE.md: status filtering of query results is unit-tested only, pending a Python ingestion path (#17) to attach statuses to real samples. |
Adds a client interface to the MLDP Sample Status API (new in dp-grpc 1.16.0, added by dp-grpc#139).
Closes #8
What's here
cbe9631e783775SampleStatusClient— the four live RPCs053c1496c5b292SampleStatusSelectorsupport inQueryClient0f937ff7fa3368Scope follows the agreed plan: the four live RPCs, the query-selector integration, bucket conversions, and the cookbook recipe. The deferred domain-registry RPCs (
saveSampleStatusDomain/querySampleStatusDomains) are not wrapped — they are reserved placeholders returning "not implemented".Two model properties drove the design
Both are easy to violate silently, so each is pinned by tests.
Absence means "no assertion." There is no implicit default status; labeling three samples says nothing about the rest. So absent
confidence/reasonssurface asNone— never0.0/"".0.0is a legitimate confidence value, so a fabricated default would be indistinguishable from a real "I am certain this is fine."Matching is by exact timestamp at nanosecond precision. No nearest-sample search, no tolerance window.
expand_data_timestamps()therefore computesSamplingClockpositions asstartTime + i * periodNanosin integer nanoseconds. Present-day epoch nanos need ~61 bits against float64's 53 mantissa bits, so a float-seconds round-trip would produce timestamps matching nothing.New patterns
Where the PV-metadata and machine-config patterns carried over they were followed; four places needed something new:
sampling_clock()/timestamp_list()axis builders, with per-timestamp length rules validated client-side so a mistake names the offending PV instead of bouncing the whole batch.timestamp_list()compares nanoseconds, not just seconds, when enforcing strict increase.PvMetadataQuery.delete_sample_statuses()requirespv_namesorall_pvs=True, raising on neither or both, so the destructive every-PV delete is never reached by omitting an argument.SampleStatusFilter.include()/.exclude()as the only constructors, making the server-rejectedMODE_UNSPECIFIEDzero value unreachable.A note at a shared seam
_build_query_spec()is shared with the future bucket-oriented client (#16), where the server rejectssampleStatusSelector— a bucket is a stored unit that cannot be partially filtered without rewriting it. A comment at the seam records that a bucket request builder must refusesample_status_filterrather than copy it through into a request the server would reject.Verification
limit=0test assertedrequest.limit == 0, which passes even with the PvMetadataClient query ignores limit=0 (truthiness bug) #13 truthiness bug, because a proto3 scalar set to 0 is indistinguishable from unset — both serialize to zero bytes and report emptyListFields(). Patching the protobuf property raisesAttributeError, so the test now spies on the assignment. That test carries more machinery than a unit test usually wants; it's the one spot worth a close look in review.Worth knowing
pyproject.tomland the only commit touching that file, but it belongs here substantively: the bug was reachable only because the same stub regen that added these RPCs raised the stubs' import-time grpcio requirement above the declared floor. Without it,pip installcan produce a package that fails at import.UNIMPLEMENTED— reachability alone is not a sufficient skip check, and without the probe this surfaces as seven assertion failures instead of a skip.🤖 Generated with Claude Code
https://claude.ai/code/session_017CvckaDwdLbD29hd23ur2a