Skip to content

feat: add sample status client, conversions, and query filtering - #37

Merged
craigmcchesney merged 7 commits into
mainfrom
feat/issue-8-sample-status-client
Aug 23, 2026
Merged

craigmcchesney merged 7 commits into
mainfrom
feat/issue-8-sample-status-client

Conversation

@craigmcchesney

Copy link
Copy Markdown
Collaborator

Adds a client interface to the MLDP Sample Status API (new in dp-grpc 1.16.0, added by dp-grpc#139).

Closes #8

What's here

Commit
cbe9631 Raise grpcio/protobuf floors to match the regenerated stubs
e783775 SampleStatusClient — the four live RPCs
053c149 Bucket conversions — per-sample expansion
6c5b292 SampleStatusSelector support in QueryClient
0f937ff Cookbook recipe, CLAUDE.md, integration tests
7fa3368 Record end-to-end verification

Scope follows the agreed plan: the four live RPCs, the query-selector integration, bucket conversions, and the cookbook recipe. The deferred domain-registry RPCs (saveSampleStatusDomain / querySampleStatusDomains) are not wrapped — they are reserved placeholders returning "not implemented".

Two model properties drove the design

Both are easy to violate silently, so each is pinned by tests.

Absence means "no assertion." There is no implicit default status; labeling three samples says nothing about the rest. So absent confidence/reasons surface as None — never 0.0/"". 0.0 is a legitimate confidence value, so a fabricated default would be indistinguishable from a real "I am certain this is fine."

Matching is by exact timestamp at nanosecond precision. No nearest-sample search, no tolerance window. expand_data_timestamps() therefore computes SamplingClock positions as startTime + i * periodNanos in integer nanoseconds. Present-day epoch nanos need ~61 bits against float64's 53 mantissa bits, so a float-seconds round-trip would produce timestamps matching nothing.

New patterns

Where the PV-metadata and machine-config patterns carried over they were followed; four places needed something new:

  • Structured frame construction — sampling_clock() / timestamp_list() axis builders, with per-timestamp length rules validated client-side so a mistake names the offending PV instead of bouncing the whole batch. timestamp_list() compares nanoseconds, not just seconds, when enforcing strict increase.
  • No criterion-builder class — deliberately. The request takes plain repeated string filters (ANDed across, ORed within), so plain lists are the honest representation rather than ceremony imitating PvMetadataQuery.
  • Explicit wildcard opt-in — delete_sample_statuses() requires pv_names or all_pvs=True, raising on neither or both, so the destructive every-PV delete is never reached by omitting an argument.
  • SampleStatusFilter.include()/.exclude() as the only constructors, making the server-rejected MODE_UNSPECIFIED zero value unreachable.

A note at a shared seam

_build_query_spec() is shared with the future bucket-oriented client (#16), where the server rejects sampleStatusSelector — a bucket is a stored unit that cannot be partially filtered without rewriting it. A comment at the seam records that a bucket request builder must refuse sample_status_filter rather than copy it through into a request the server would reject.

Verification

  • 418 tests pass against a live 1.16.0 stack (401 on CI's unit-only path); ruff lint/format clean; all 78 cookbook snippets type-check.
  • Integration tests confirm the exact-timestamp contract end to end — including a dense case labeling 100 samples at 1 kHz and checking every expanded position. That is the assertion unit tests over hand-built objects fundamentally cannot make.
  • Mutation-tested rather than trusting green tests. This caught a real gap: the first limit=0 test asserted request.limit == 0, which passes even with the PvMetadataClient query ignores limit=0 (truthiness bug) #13 truthiness bug, because a proto3 scalar set to 0 is indistinguishable from unset — both serialize to zero bytes and report empty ListFields(). Patching the protobuf property raises AttributeError, so the test now spies on the assignment. That test carries more machinery than a unit test usually wants; it's the one spot worth a close look in review.

Worth knowing

  • The floor fix is in this PR by choice. It's three lines in pyproject.toml and the only commit touching that file, but it belongs here substantively: the bug was reachable only because the same stub regen that added these RPCs raised the stubs' import-time grpcio requirement above the declared floor. Without it, pip install can produce a package that fails at import.
  • Integration tests probe for the API before running. A pre-1.16.0 Annotation Service accepts the connection and then answers every sample status call with UNIMPLEMENTED — reachability alone is not a sufficient skip check, and without the probe this surfaces as seven assertion failures instead of a skip.
  • One caveat remains: status filtering of query results is unit-tested only. The integration tests write statuses and read them back, but with no Python ingestion path (Add ingestion API client (full surface: ingestData + streaming) #17) those statuses have no underlying samples to attach to, so "this PV minus what the model flagged" hasn't been watched to actually drop rows. Flagged in the recipe and CLAUDE.md, with Add ingestion API client (full surface: ingestData + streaming) #17 as the trigger to re-verify.

🤖 Generated with Claude Code

https://claude.ai/code/session_017CvckaDwdLbD29hd23ur2a

craigmcchesney and others added 6 commits August 22, 2026 13:04
The 1.16.0 stub sync (PR #36) regenerated the gRPC stubs with a newer
toolchain.  The generated code now hard-requires newer runtimes at import
time, but the dependency floors were left at the previous values:

  - *_pb2_grpc.py declare GRPC_GENERATED_VERSION = '1.83.0' and raise a
    RuntimeError on import under an older grpcio.
  - *_pb2.py call ValidateProtobufRuntimeVersion(PUBLIC, 7, 35, 1).

The pins still allowed grpcio 1.82.1 / protobuf 7.35.0, so a resolver
picking the floor produced an install that fails on the first import:

    RuntimeError: The grpc package installed is at version 1.82.1, but the
    generated code in annotation_pb2_grpc.py depends on grpcio>=1.83.0.

Bump both runtime floors and the [codegen] grpcio-tools floor to match, as
the comment above the dependency block already directs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CvckaDwdLbD29hd23ur2a
Adds the client interface to the MLDP Sample Status API (dp-grpc #139),
which assigns int32 status codes to individual PV samples keyed by
(pvName, timestamp, domain, layer).

Wraps the four live RPCs, following the established client patterns:
three-tier error handling, ApiResultBase result objects whose typed
accessors return None/[] on error, the query_*/iter_* paging pair raising
RuntimeError on a page error, and the streaming internal-yield/public-raise
contract from QueryClient.  Mounted on the annotation facade as
client.annotation.sample_status.

The deferred domain-registry methods are deliberately not wrapped; they are
reserved placeholders returning "not implemented".

Four places needed new patterns rather than the existing ones:

- Frame construction is the first genuinely structured client input: a
  time axis plus parallel arrays.  sampling_clock() and timestamp_list()
  build the two axis forms, and the per-timestamp length rules for
  statusCodes/confidence/reasons are validated client-side so a mistake
  surfaces as a ValueError naming the offending PV rather than as a server
  rejection of the whole batch.  timestamp_list() enforces the
  strictly-increasing rule, comparing nanoseconds and not just seconds.

- No criterion-builder class, unlike PvMetadataQuery/ConfigurationQuery.
  The query filters are plain repeated strings ANDed together with no
  criterion oneof, so plain lists are the honest representation.

- The empty-pvNames wildcard is explicit on delete.  The same field is
  benign discovery on query but removes every PV's statuses on delete, so
  delete_sample_statuses() requires either pv_names or all_pvs=True and
  raises when given neither or both -- the destructive wildcard is never
  reached by omitting an argument.

- An all-empty reasons list is dropped rather than sent as empty strings,
  as the API directs, while a partially-populated one is kept whole.

Request building guards limit on `is not None` so a meaningful limit=0
survives (the issue #13 bug shape).  Its regression test spies on the
assignment, since a proto3 scalar set to 0 is indistinguishable from an
unset one by both reflection and wire bytes.

67 unit tests; verified against four targeted mutations (limit truthiness,
delete wildcard opt-in, timestamp ordering, empty-reasons handling), each
of which fails the suite.

Refs #8

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CvckaDwdLbD29hd23ur2a
Expands the compact SampleStatusBucket form (a time axis plus parallel
arrays) into per-sample SampleStatusRow objects.

SamplingClock expansion is computed in integer nanoseconds
(startTime + i * periodNanos).  Status-to-sample matching is by exact
timestamp at nanosecond precision, and present-day epoch nanoseconds
need ~61 bits, so a float-seconds round-trip would produce timestamps
that no longer match the samples they label.

Absent confidence and reasons surface as None rather than 0.0 or "".
Absence means "no assertion" throughout this API, and 0.0 is a
legitimate confidence value, so a fabricated default would be
indistinguishable from a real one.

Refs #8

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CvckaDwdLbD29hd23ur2a
Adds a SampleStatusFilter helper and an optional sample_status_filter
parameter on QueryParams, populated into QuerySpec field 4, so a
time-series query can be restricted to (or away from) samples carrying
particular sample statuses.

The selector is built only through include()/exclude(), which makes the
MODE_UNSPECIFIED zero value unreachable rather than something a caller
falls into by leaving the mode unset.

_build_query_spec() is shared with the future bucket-oriented client
(#16), where the server rejects sampleStatusSelector -- a bucket cannot
be partially filtered without rewriting it.  A note at the seam records
that a bucket request builder must refuse the parameter rather than
copying it through into a request the server would reject.

Refs #8

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CvckaDwdLbD29hd23ur2a
Adds doc/cookbook/sample-status.md (all 11 snippets pass the cookbook
checker), a cookbook README row, and a CLAUDE.md "Sample Status API"
section with Key Files entries.

Corrects the two places describing valueStatus as merely unpopulated to
name sample status as its replacement.

Integration tests probe for the API before running: reachability alone
is not enough, because a pre-1.16.0 Annotation Service accepts the
connection and then answers every sample status call with UNIMPLEMENTED
-- which surfaced as seven assertion failures rather than a skip against
the server currently on localhost:50053.

Also softens the two places hardcoding "docker compose up -d" as the way
to start the ecosystem, since the services can equally be run directly.

Refs #8

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CvckaDwdLbD29hd23ur2a
The integration tests now run rather than skip: the exact-timestamp
contract, absent-stays-absent, full-replace upsert, and layer
independence are all confirmed against a real Annotation Service.

Narrows the remaining caveat to what is actually still unverified --
status filtering of query results, which needs ingested sample data to
attach to (#17).

Refs #8

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CvckaDwdLbD29hd23ur2a
Copilot AI lite review requested due to automatic review settings August 23, 2026 17:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds first-class support for MLDP’s Sample Status API to dp-python-lib, including a new SampleStatusClient, per-sample “bucket → rows” conversions, and optional sample-status-based filtering for v2 sample queries.

Changes:

  • Introduces SampleStatusClient under the annotation facade (client.annotation.sample_status) with params/types and three-tier error handling.
  • Adds sample_status_conversions utilities to expand SampleStatusBucket results into flat per-sample SampleStatusRow objects with nanosecond-exact timestamp expansion.
  • Extends QueryClient/QueryParams with SampleStatusFilter → QuerySpec.sampleStatusSelector support, and documents/tests the new behavior.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/dp_python_lib/client/sample_status_client.py New Sample Status feature client + axis builders + request/response wrappers.
src/dp_python_lib/client/sample_status_conversions.py New conversion layer: expand sample-status buckets into per-sample rows with exact-nanos timestamps.
src/dp_python_lib/client/query_client.py Adds SampleStatusFilter helper + wires sampleStatusSelector into _build_query_spec().
src/dp_python_lib/client/query_conversions.py Documentation update noting valueStatus deprecation in favor of sample status.
src/dp_python_lib/client/annotation_client.py Mounts the new SampleStatusClient on the annotation facade.
src/dp_python_lib/client/__init__.py Exposes sample status client/types/helpers from dp_python_lib.client.
tests/unit/test_sample_status_client.py Unit tests for request building, validation, paging/streaming behavior, and error handling.
tests/unit/test_sample_status_conversions.py Unit tests for exact timestamp expansion and bucket→row conversion semantics.
tests/unit/test_query_client.py Unit tests for SampleStatusFilter and request building with sampleStatusSelector.
tests/integration/test_sample_status_client_integration.py Live-service integration tests for save/query/delete, axis round-trip, and upsert semantics.
tests/integration/conftest.py Minor docstring clarification for integration-test prerequisites.
doc/cookbook/sample-status.md New cookbook recipe documenting labeling, reading back, and query filtering usage.
doc/cookbook/README.md Adds cookbook index entry for sample status recipe.
doc/cookbook/query.md Adds cross-reference to sample status as the replacement for valueStatus.
CLAUDE.md Adds sample status client/conversions overview and usage notes.
.dev/tools/check-cookbook-snippets.py Extends snippet-checker preamble imports to include new sample status symbols.
pyproject.toml Raises grpcio/protobuf (and grpcio-tools) minimum versions to match regenerated stubs.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/dp_python_lib/client/sample_status_client.py
Comment thread src/dp_python_lib/client/query_client.py
Comment thread doc/cookbook/sample-status.md Outdated
Comment thread tests/integration/test_sample_status_client_integration.py
Four review findings from the PR #37 discussion, plus a test rename.

Reject empty time axes in _timestamp_count().  A hand-built DataTimestamps
can carry a zero-count SamplingClock or an empty TimestampList, and both
report their oneof arm as set, so neither was caught by the unset-axis
check.  These surfaced as a column-length mismatch naming the PV rather
than the real problem.  The axis builders already make this unreachable;
the guard puts the write path in step with expand_data_timestamps(), which
applies the same rule on the read path.

Validate sample_status_filter in QueryParams.  SampleStatusFilter.include()
/exclude() make MODE_UNSPECIFIED unreachable through the helpers, but the
parameter accepts any SampleStatusSelector, so a default-constructed one set
the field present-but-invalid and was rejected by the server.  Checking it at
construction names the problem and points at the helpers.

Compute expected nanoseconds with integer arithmetic in the integration
tests.  The float path was exact for present-day microsecond datetimes (it
breaks only near the int64 nanosecond limit), so this was not a live flake --
but tests asserting exact nanosecond round-trips should not derive their
expectation from a float whose ULP at these epochs is ~238ns.

Correct the cookbook's "is None" sentence.  client.annotation is what becomes
None; reaching through it raises AttributeError rather than yielding None.
Fixed in all three annotation cookbook pages, since the wording was shared.

Rename test_streaming_request_carries_the_selector_too, which called the
unary builder.  Coverage was not missing -- iter_query_samples_stream() uses
that same builder -- but the name promised something the test did not do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CvckaDwdLbD29hd23ur2a
@craigmcchesney

Copy link
Copy Markdown
Collaborator Author

Review fixes — bbb0caa

All four Copilot findings addressed, plus one rename from review. Replies are in each thread; summary here.

# Finding Outcome
1 _timestamp_count() allows empty axes Fixed — guard added, both arms
2 QueryParams does not validate the selector Fixed — the substantive one
3 Cookbook None wording Fixed in all three annotation pages
4 Float arithmetic in integration test Changed — valid in principle, not actually flaky
— test_streaming_request_carries_the_selector_too Renamed

The rename (not from Copilot)

That test called _build_query_samples_request() — the unary builder — so its name promised something it did not do. Coverage was not missing: iter_query_samples_stream() uses that same builder (query_client.py:709), so exercising it does cover the streaming path. Renamed to test_selector_reaches_the_streaming_path_via_the_shared_builder, with a comment recording why, so a future split into two builders fails loudly rather than silently leaving the streaming path unfiltered.

Two claims in the PR description that were verified, not assumed

  • Integer-nanosecond expansion is genuinely necessary. datetime.timestamp() has a ULP of ~238ns at present-day epochs — it cannot represent nanoseconds at all. The design reasoning holds.
  • The limit=0 spy test earns its machinery. Confirmed a proto3 scalar set to 0 is indistinguishable from unset via reflection, so watching the assignment really is the only way to catch the PvMetadataClient query ignores limit=0 (truthiness bug) #13 truthiness bug.

Docs corrected beyond the code

Finding #2 exposed a docs overstatement: CLAUDE.md and the cookbook both said include()/exclude() were "the only ways" to build a selector, making MODE_UNSPECIFIED unreachable. That was the intent, but the parameter accepts any SampleStatusSelector, so it was never actually unreachable. Both now describe what the code enforces.

Verification

  • 423 passed, 1 skipped (401 unit → 406 unit, +5 pinning the new guards). Integration tests ran against a live 1.16.0 server, so the _exact_nanos change is confirmed end to end
  • ruff lint + format clean; 78/78 cookbook snippets type-check
  • Each new validation manually confirmed to raise with its intended message

The remaining caveat is unchanged and still flagged in the recipe and CLAUDE.md: status filtering of query results is unit-tested only, pending a Python ingestion path (#17) to attach statuses to real samples.

@craigmcchesney
craigmcchesney merged commit 3c2efca into main Aug 23, 2026
6 checks passed
@craigmcchesney
craigmcchesney deleted the feat/issue-8-sample-status-client branch September 11, 2026 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

interface to sample status API

2 participants