Repository navigation
Expand gathered telemetry: golden signals, traces, business metrics, and service logging #159
Copy link
Copy link
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
Summary
The observability stack (Grafana, Loki, Prometheus, Jaeger, OTel collector) is in place, but the API currently only emits default ASP.NET Core/HttpClient/HotChocolate traces, one custom metric pair (tournaments_created, active_tournaments), and unstructured request logs. This issue tracks expanding what we actually gather so the stack is useful for both API health and tournament-flow business questions.
Golden signals (metrics)
-
graphql.server.request.durationhistogram, taggedgraphql.operation.type+graphql.operation.name— p90/p95/p99 and request rate per GraphQL operation (the ASP.NET Core histogram can't distinguish operations since everything hits/graphql) -
graphql.server.errors_totalcounter, taggedgraphql.operation.name+graphql.error_code— GraphQL errors return HTTP 200, so they're invisible to HTTP-status-based error rate today
Traces
- Add
OpenTelemetry.Instrumentation.EntityFrameworkCore(AddEntityFrameworkCoreInstrumentation()) — currently missing, so DB time is invisible inside request traces - Span for
TournamentAutoCloseJob(Hangfire job execution is currently untraced) - Span around
BracketMutations.GenerateBracketto separate algorithm time from persistence time - Span/event on the optimistic-concurrency retry paths (
JoinTournament,AddParticipant,CorrectMatchResult,UpdateRoundto see retry counts, not just final outcome
Business metrics
- Tournaments:
tournaments_closed_total{reason="manual|auto|full"},tournaments_deleted_total - Participants:
tournament_join_attempts_total{result="success|full|closed|already_joined"}, slot-contention counter - Matches/Brackets:
matches_played_total,matches_needing_replay_total,match_result_corrections_total,bracket_generation_total{result="success|failure"} - Auth:
login_attempts_total{result="success|failure"},user_registrations_total,refresh_token_failures_total{reason}
Logging
- Push domain IDs (
TournamentId,MatchId,BracketId, etc.) into the Serilog log context viaLogContext.PushPropertyinBracketMutations,ParticipantMutations,MatchMutations,TournamentMutations,UserMutations, so existing request/error logs become filterable by entity - Targeted INFO logs (same across environments — verbosity controlled via
MinimumLevelconfig, not env-conditional code) for state transitions that don't surface as a GraphQL error and wouldn't be obvious from a trace span alone:BracketCompletionService— bracket marked completeBracketService— bracket generation outcome (seed/round counts)MatchCascadePositionCalculator— cascade applied, downstream matches affectedMatchCorrectionService— correction applied (previous vs. new result), retry count on concurrency conflict
Non-Goals
- Rate-limiter rejection metrics
- Per-field/DataLoader-level custom metrics beyond what HotChocolate's own instrumentation already traces
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity