feat(scripts): verify the features the gate never touches - #228
Merged
Conversation
The release gate answers one question - does video arrive and decode. The package also offers snapshots, PTZ, two-way audio, cloud recordings and thumbnails, and nothing verified any of them on real hardware. The probe reports four outcomes, not three. A feature the camera does not have, a feature that was not exercised, and a feature that was exercised and failed mean different things, and collapsing any two is the defect this project has now fixed four times - a probe that could not run, reported as a zero result. IsSupportPlayback is the live example: every camera on the reference account reports 0, so cloud recordings return no events forever. Scored FAIL that reads as a broken feature. Scored UNSUPPORTED it reads as the truth, so recordings are gated on that flag before being attempted. Talk sends 25 frames of silence and passes only if the PCM provider was actually pulled - the ack is the camera's, the pull is ours. PTZ stops after every nudge and returns the head the other way, and stops again on the error path, so a failed probe cannot leave a camera panning. Advisory: reported, never gating. The same reasoning the decode probe shipped under - a measurement that has never run against this fleet is not one to start blocking releases with.
The first fleet run of the feature probe reported FAIL for snapshot on every camera. That was the probe, not the product: async_snapshot takes an output path and this called it with none, so all seven produced the same TypeError. It now passes a path and checks the file that lands - the returned bool is the library's claim, the file size is the independent check, and a True return over a 12-byte file is the same "bytes of the right shape" hole the decode gate exists to close. Thumbnails reported FAIL on all six live cameras too. That is not the feature: from the OWNING account all six return a CloudFront URL and only the offline camera returns None. The CI runner authenticates as a shared-home member, and from there the cloud answers with nothing - no error, just empty. Leading hypothesis, not a conclusion; the error text now says so, so nobody reads a fleet FAIL there as a defect without ruling the account out first. Per-probe timeout drops from 20s to 10s. Five probes across seven cameras at 20s worst case turned a 7 minute fleet run into 25. The measured latencies are nowhere near either bound - PTZ acks land in 0.01-0.19s, talk's SPEAKERSTART in 0.01-0.38s. What the run did verify, and these are real: PTZ moves and returns on the A001064, and two-way audio works on all three A000088s - the camera accepted the stream AND pulled our PCM frames, which is the half that proves it rather than an ack alone.
The corrected probe passed snapshot on all three DTLS cameras and timed out on both SDES ones. That was the budget, not the feature: a DTLS camera waits for the first keyframe via on_frame, while an SDES camera streams about five seconds to a temp file and then shells out to ffmpeg to extract the JPEG. Ten seconds cannot cover the second path. The budget is now per-transport - 10s for DTLS, 25s for SDES - rather than one number that is wrong for one of them. Everything else stays at 10s, where the measured latencies (PTZ 0.01-0.19s, talk 0.01-0.38s) sit nowhere near the bound.
Two UNSUPPORTED verdicts in the feature run were wrong, in different ways. Two-way audio was the harness. An SDES session only negotiates a return track when the offer advertises sendrecv, and the fleet harness opened without talk=True - so talk_supported was False and the probe reported UNSUPPORTED for cameras that support it. The harness not asking, misread as the camera not offering. It asks now. Recordings was not the harness. IsSupportPlayback is 1 on the A001064 and both A001513s, which have no SD card, and 0 on every A000088, which reports SDcardStatus 1. It says where a camera's recordings live, not whether the model can play back. UNSUPPORTED is therefore honest for the API being probed - and it exposes a real gap: the only retrieval path in this package is async_open_cloud_playback, so four of seven cameras record to a card that nothing here can read. Recorded as ROAD-TO-1.0 item 6 rather than hidden by widening the flag. Nothing regressed; it was never built. But the product describes playback without qualifying it by storage, so either the capability or the description has to change.
Two-way audio reported 'no PCM frames pulled' on every SDES camera. The talk path was working; the probe was hanging up before the callee picked up. async_start_talk returns as soon as the camera ACKs SPEAKERSTART (848 -> 851), but the pump that polls the provider runs on its own thread and waits until the camera audio address is filled in by the bridge on first INBOUND audio. The probe held for one second and stopped, which on SDES is reliably before that address exists. It now holds up to six seconds and returns the moment the first frame is pulled, so a working camera costs a fraction of a second and only a genuinely silent one pays the full wait. The pull still decides the verdict rather than the ack - passing on the ack alone would hide the opposite failure, where the camera agrees and we never send anything.
…silence SDES talk still reported "no PCM frames pulled" after the hold went to six seconds, which killed my own diagnosis: the hold was not the problem. The problem is that the probe ignored the API's answer. async_start_talk returns a bool - False when the camera refuses SPEAKERSTART (848/851) or talk is not available on the session - and the probe awaited it and threw the value away. When it is False the pump never gets speaker_on and never polls the provider, so "no PCM frames pulled" was reporting the symptom while the return value had already given the reason. The two outcomes now read differently: "camera refused SPEAKERSTART" against "SPEAKERSTART accepted but the pump never polled us", the latter naming the other two things the pump waits on - the camera audio address and speaker_on. One of those is the next run's answer rather than another guess from me.
async_start_talk returned True whenever the ack wait expired, because _speaker_ack_accepted treats an absent or unfamiliar ack as acceptance. That generosity is about interpreting the camera's answer - no genuine refusal has ever been identified - and says nothing about the case where the command was never sent at all. speaker_on is set on exactly one line, immediately after the bridge thread dispatches SPEAKERSTART(848), so a falsy flag after the ack wait means our own bridge never sent it. That happens whenever ffmpeg has exited: a stall, a camera drop, the abandon ceiling. Require the flag as well as the ack. talk_supported now requires the session to still be running for the same reason. It answered from the talk-state dict, which outlives ffmpeg, while the bridge thread and the pump's socket do not - so async_speak reused a session that could no longer carry talk instead of opening a fresh one, and the aidot.talk service returned success on silence. Measured on the fleet: three SDES cameras returned True from async_start_talk and never pulled a PCM frame, and the run log contains no SPEAKERSTART line at all. The DTLS path is untouched: it shares _speaker_ack_accepted, whose behaviour does not change.
max_seconds was hold-2, so the recording - and on the SDES path the bridge thread that lives and dies with that ffmpeg - ended before asyncio.sleep(hold) even returned. Every feature probe then ran against a closed session, and the snapshot ran first, spending another 25s on SDES before talk was asked for at all. The last run read that as three SDES cameras failing two-way audio. Three changes, because one alone would leave the result depending on the other two being right: - Leave LIVE_PROBE_BUDGET_S of session life past the recording window. - Run talk and PTZ before the snapshot, since they ride the open session and the snapshot opens its own. - Report NOT_RUN, not FAIL, when the session is already closed. This module's whole premise is that unsupported, not-run and failed are three different results; a corpse scored as FAIL collapsed the second into the third and accused the camera. PTZ gets the same guard. Its send can succeed into a socket nobody is reading, so a closed session yielded PASS on no evidence - the quieter half of the same defect.
1.0.0b1 was tagged but never reached PyPI. This supersedes it and carries the two-way audio fix.
The last run reported ptz=PASS on the A001064 while its own log carried four 'no active stream session' warnings for that camera - one per command the probe sent. async_ptz_move does not raise when it cannot send; it logs, returns False and nothing leaves the host. The probe caught only exceptions, so four refused commands scored as a pass. That is the fifth time this project has found a probe that could not run reported as a zero result, and it means PTZ had never been exercised on hardware at all. Both halves are fixed. The probe now reads every return value and names which commands were refused. And the harness registers the open session as _stream_session for the probe window - the library sets that from its keepalive, streaming and serve loops, which are the paths Home Assistant goes through, and not from the bare async_open_webrtc_stream the harness calls. It is restored afterwards so nothing inherits a session that is about to be closed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The release gate answers one question - does video arrive and decode. The
package also offers snapshots, PTZ, two-way audio, cloud recordings and
thumbnails, and nothing verified any of them on real hardware.
Four outcomes, deliberately not three
A feature the camera does not have, a feature that was not exercised, and a
feature that was exercised and failed mean different things. Collapsing any two
is the defect this project has now fixed four times - a probe that could not
run, reported as a zero result.
IsSupportPlaybackis the live example: every camera on the reference accountreports 0, so cloud recordings return no events forever. Scored FAIL that reads
as a broken feature. Scored UNSUPPORTED it reads as the truth.
What the first fleet run found
Verified, and these are real:
PCM provider was actually pulled: the ack is the camera's, the pull is ours
Two FAIL columns were the probe, not the product, and both are fixed here:
async_snapshottakes an output path; the first version called it with none,so all seven cameras produced the same TypeError. It now passes a path and
checks the file that lands - the bool is the library's claim, the file size is
the independent check
CloudFront URL. The CI runner authenticates as a shared-home member and gets
nothing back - no error, just empty. Leading hypothesis, not a conclusion, and
the error text now says so, so a fleet FAIL there is not read as a defect
before the account is ruled out
Timeouts
20s to 10s per probe. Five probes across seven cameras at 20s worst case turned
a 7 minute fleet run into 25. Measured latencies are nowhere near either bound -
PTZ acks land in 0.01-0.19s, talk's SPEAKERSTART in 0.01-0.38s.
Safety
PTZ stops after every nudge and returns the head the other way, and stops again
on the error path, so a failed probe cannot leave a camera panning. Talk sends
silence. Advisory: reported, never gating - the same reasoning the decode probe
shipped under.