Improve YouTube WebSub handling and live-status diagnostics

[David Pine (IEvangelist)]

Summary

• Add safe, operation-specific YouTube diagnostics for subscription requests, channel resolution, discovery, known-video checks, and notification confirmation. Record successful discovery observations and distinguish HTTP acceptance from callback verification.
• Acknowledge and discard invalid signed/unsigned WebSub notifications without processing them. Log hub.mode=denied as an untrusted report without mutating subscription or live state.
• Capture the hub's transient-error classification and typed Retry-After metadata without exposing API keys, verification tokens, HMAC secrets, or raw provider response bodies.
• Correct quota documentation for the separate Search Queries bucket and explain that discovery intervals and notification coalescing do not enforce a project-wide daily budget.

Validation

• Initial diagnostics: 248 non-Redis StaticHost tests passed.
• Callback and diagnostic follow-up: 137 targeted YouTube/LiveEndpoints tests passed using:

  dotnet test .\tests\StaticHost.Tests\StaticHost.Tests.csproj --configuration Release --no-restore --filter "(FullyQualifiedName~YouTube|FullyQualifiedName~LiveEndpointsTests)&Category!=RedisIntegration" -p:ShouldRunBuildScript=false -p:ShouldRunNpmInstall=false

• No frontend build or production deployment performed.

Live investigation and limitations

Production telemetry and a standalone local manual probe reproduced HTTP 503 from Google's subscription hub after approximately 20 seconds, with Retry-After: 120 and Transient error; please try again later. The public tunnel reached the local listener, but no hub verification or denial callback was observed. Controlled async/sync and HTTPS/HTTP-topic comparisons all reproduced the failure.

This PR fixes confirmed callback-handling and diagnostic gaps; it does not claim to resolve Google's subscription failure or prove end-to-end livestream detection. Production retains the documented HTTPS topic, asynchronous verification, existing polling intervals, and retry policy. A shared daily API quota limiter is not implemented, and production project's actual quota allocation remains unconfirmed. The standalone manual probe is outside this repository and is not included in this PR.

[Copilot]

🟡 Changes recommended

Provider responses can still be buffered without an effective size limit, and the final review comments remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Improves YouTube WebSub callback handling, diagnostics, quota documentation, and related test coverage.

Changes:

• Adds secret-safe, operation-specific diagnostics and Retry-After metadata.
• Handles invalid notifications and denial reports without mutating live state.
• Expands callback, discovery, and diagnostic tests and documentation.

File summaries

File	Reviewed changes
tests/StaticHost.Tests/Live/YouTubeWebSubServiceTests.cs	Updates WebSub logging and discovery assertions.
tests/StaticHost.Tests/Live/YouTubeDiagnosticsTests.cs	Tests diagnostic classification, redaction, and recovery metadata.
tests/StaticHost.Tests/Live/LiveEndpointsTests.cs	Tests callback validation, denial handling, and confirmation behavior.
tests/StaticHost.Tests/Live/InMemoryLiveStatusInfrastructure.cs	Adds test state and confirmation observability helpers.
src/statichost/StaticHost/Live/YouTube/YouTubeWebSubService.cs	Adds discovery and subscription diagnostics.
src/statichost/StaticHost/Live/YouTube/YouTubeLiveConfirmationQueue.cs	Avoids duplicate failure logging.
src/statichost/StaticHost/Live/YouTube/YouTubeDiagnostics.cs	Implements safe provider failure classification and metadata handling.
src/statichost/StaticHost/Live/YouTube/YouTubeClient.cs	Captures provider failure diagnostics.
src/statichost/StaticHost/Live/README.md	Documents diagnostics, callbacks, and quota behavior.
src/statichost/StaticHost/Live/LiveStatusOptions.cs	Clarifies discovery quota behavior.
src/statichost/StaticHost/Live/LiveEndpoints.cs	Handles signatures, denials, and confirmation diagnostics.

Review details

Suppressed comments (3)

src/statichost/StaticHost/Live/LiveEndpoints.cs:354

• This message is also emitted for a retry of a previously accepted verification: TryConfirm returns true from RecentConfirmation without changing RenewAt or clearing Retry. Saying every accepted callback establishes/renews the lease and resets backoff makes the diagnostic claim a state mutation that did not occur; distinguish the initial confirmation from a retry (or log the transition result).

        logger.LogInformation(
            "YouTube {Operation} verified; granted lease {LeaseSeconds}s. Verification establishes or renews the lease and resets subscription backoff.",
            "WebSubVerification", leaseSeconds);

src/statichost/StaticHost/Live/YouTube/YouTubeDiagnostics.cs:37

• This read is capped at 4,097 characters, but all current YouTube callers invoke GetAsync/PostAsync with the default ResponseContentRead, so HttpClient has already buffered the entire provider response before this method runs. A large error page can therefore still cause unbounded response buffering despite the documented 4,096-character diagnostic limit; use ResponseHeadersRead plus a bounded response-body read (or an equivalent size limit) at the request call sites.

            using var stream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false);
            using var reader = new StreamReader(stream);
            var buffer = new char[BodyLimit + 1];
            var count = await reader.ReadBlockAsync(buffer.AsMemory(), cancellationToken).ConfigureAwait(false);
            details = DescribeBody(new string(buffer, 0, Math.Min(count, BodyLimit)), count > BodyLimit, secrets);

src/statichost/StaticHost/Live/YouTube/YouTubeWebSubService.cs:175

• The HTTP 2xx acceptance is logged only after MarkRequestSentAsync completes. If the hub accepts the POST but the Redis/state write fails, this log is skipped and the outer handler reports only a BackgroundTick failure, so operators cannot distinguish provider acceptance from local persistence failure. Record the acceptance before (or in a finally around) the state write, while keeping the message explicit that it is not verification.

                await _subscriptions.MarkRequestSentAsync(
                    request,
                    _time.GetUtcNow(),
                    cancellationToken).ConfigureAwait(false);
                logger.LogInformation(

• Files reviewed: 11/11 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[aspire-repo-bot]

Frontend HTML artifact ready

The latest frontend build uploaded the frontend-dist artifact for PR #1685. Use the VS Code button below to open this PR with GitHub Artifacts Explorer and browse the built HTML locally.

This comment updates automatically when a new frontend build artifact is uploaded.