Skip to content

perf: scenario-based load generator using EDG SDK with metrics observability #76

Description

@e7217

Context

The NATS-level baseline (separate issue) measures the transport floor, but does not exercise EDG's actual ingest path: SDK serialization → core handler (internal/core/handler.go) → validation → platform.data.validated publish → JetStream → Telegraf → VictoriaMetrics. To answer realistic capacity questions ("can EDG sustain 10K msg/s with 100 simulated adapters?", "what's p99 end-to-end at burst?"), we need a scenario-driven load generator that uses the same SDK adapters do.

This complements ADR 0001 (#65) chaos tests: chaos validates correctness under failure, this validates performance under load.

Goals

  1. Load generator binary at cmd/loadgen/:
    • Uses adapters/go/sdk so it exercises the same publish path as a real adapter
    • Configurable: number of simulated adapters, msg/s per adapter, payload size, ramp-up duration, total run time
    • Supports steady-state, ramp, and burst profiles
  2. Observability:
    • Generator exports Prometheus metrics: sent count, send error rate, in-flight, p50/p95/p99 publish latency
    • Reuses/extends EDG core metrics for backlog, validation failures, JetStream pending
    • Grafana dashboard JSON committed under deploy/grafana/loadgen.json
  3. Documentation at docs/perf/scenarios.md:
    • How to run, how to interpret metrics, how to capture a result snapshot
    • At least one recorded scenario result (e.g., "100 adapters × 100 msg/s, 5 min steady-state")
  4. Reusable as a regression check — easy to rerun after changes to handler.go or stream config

Out of Scope

Acceptance Criteria

  • cmd/loadgen/ builds and runs against a local EDG stack
  • At least three scenario profiles available via flags or config (steady, ramp, burst)
  • Prometheus /metrics endpoint exposes generator-side latency histograms
  • Grafana dashboard renders generator metrics alongside core EDG metrics
  • docs/perf/scenarios.md includes one captured run with environment metadata and observations
  • Generator uses adapters/go/sdk (no bypassing the SDK)

Open Questions

  • Should payload generation include realistic per-asset distributions (mix of number/text/bool values), or fixed-shape payloads first? — fixed first, realistic as follow-up.
  • Where to record historical results? — start with markdown snapshots, revisit if regressions become a regular concern.

Metadata

Metadata

Assignees

No one assigned

    Labels

    designArchitecture and design discussionsenhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions