Skip to content

Expand gathered telemetry: golden signals, traces, business metrics, and service logging #159

Closed

Description

@Dejmenek

Summary

The observability stack (Grafana, Loki, Prometheus, Jaeger, OTel collector) is in place, but the API currently only emits default ASP.NET Core/HttpClient/HotChocolate traces, one custom metric pair (tournaments_created, active_tournaments), and unstructured request logs. This issue tracks expanding what we actually gather so the stack is useful for both API health and tournament-flow business questions.

Golden signals (metrics)

  • graphql.server.request.duration histogram, tagged graphql.operation.type + graphql.operation.name — p90/p95/p99 and request rate per GraphQL operation (the ASP.NET Core histogram can't distinguish operations since everything hits /graphql)
  • graphql.server.errors_total counter, tagged graphql.operation.name + graphql.error_code — GraphQL errors return HTTP 200, so they're invisible to HTTP-status-based error rate today

Traces

  • Add OpenTelemetry.Instrumentation.EntityFrameworkCore (AddEntityFrameworkCoreInstrumentation()) — currently missing, so DB time is invisible inside request traces
  • Span for TournamentAutoCloseJob (Hangfire job execution is currently untraced)
  • Span around BracketMutations.GenerateBracket to separate algorithm time from persistence time
  • Span/event on the optimistic-concurrency retry paths (JoinTournament, AddParticipant, CorrectMatchResult, UpdateRound to see retry counts, not just final outcome

Business metrics

  • Tournaments: tournaments_closed_total{reason="manual|auto|full"}, tournaments_deleted_total
  • Participants: tournament_join_attempts_total{result="success|full|closed|already_joined"}, slot-contention counter
  • Matches/Brackets: matches_played_total, matches_needing_replay_total, match_result_corrections_total, bracket_generation_total{result="success|failure"}
  • Auth: login_attempts_total{result="success|failure"}, user_registrations_total, refresh_token_failures_total{reason}

Logging

  • Push domain IDs (TournamentId, MatchId, BracketId, etc.) into the Serilog log context via LogContext.PushProperty in BracketMutations, ParticipantMutations, MatchMutations, TournamentMutations, UserMutations, so existing request/error logs become filterable by entity
  • Targeted INFO logs (same across environments — verbosity controlled via MinimumLevel config, not env-conditional code) for state transitions that don't surface as a GraphQL error and wouldn't be obvious from a trace span alone:
    • BracketCompletionService — bracket marked complete
    • BracketService — bracket generation outcome (seed/round counts)
    • MatchCascadePositionCalculator — cascade applied, downstream matches affected
    • MatchCorrectionService — correction applied (previous vs. new result), retry count on concurrency conflict

Non-Goals

  • Rate-limiter rejection metrics
  • Per-field/DataLoader-level custom metrics beyond what HotChocolate's own instrumentation already traces
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions