Skip to content
This repository was archived by the owner on Aug 3, 2026. It is now read-only.

README.md

Latest commit

 

History

History
175 lines (135 loc) · 8.29 KB

README.md

File metadata and controls

175 lines (135 loc) · 8.29 KB
Raw

hyperi-rustlib docs

Shared Rust library for HyperI services. Wire three lines at startup and you get config cascade, structured logs, Prometheus metrics, health probes, OTel traces, graceful shutdown, K8s pre-stop, and deployment-artefact generation for free. Add a Transport, a TieredSink, a BatchEngine and the same deal extends: counters, span propagation, DLQ routing, backpressure, scaling signals — all automatic.

This is the index. Read ARCHITECTURE.md for the 10,000-foot view of how the modules fit together, INTEGRATION.md for a recipe walkthrough on building a DFE service, AUTO-WIRING.md for the "you-get-this-for-free" model, and FEATURE-FLAGS.md for how features cascade into one another.

What you get for free

Wire this at startup And these come along No need to
config::setup(opts) 8-layer cascade, env-var nesting, .env, sensitive masking, hot-reload, /config admin endpoint, section registry Wire figment, write a settings loader, build a reload watcher
logger::setup_default() Structured tracing, JSON-in-container / human-on-TTY autodetect, RFC 3339 timestamps, sensitive-field masking, flooding helpers Install a tracing subscriber, format JSON, pick a logger crate
MetricsManager::new("app") Prometheus exporter, /metrics endpoint, process metrics, cardinality cap, /metrics/manifest catalogue Stand up an exporter, wire a process collector, hand-roll a manifest
ServiceRuntime::new(...) All of the above + memory guard + scaling pressure + worker pool + batch engine + shutdown token + K8s pre-stop delay + runtime context Glue them together manually; six modules wire themselves
Any Transport impl 3-tier filter engine, DLQ routing, per-direction/action metrics, W3C traceparent propagation Add filters, wire DLQ, instrument send/recv
TieredSink::new(...) Transport + disk-spillover spool + circuit breaker + retry + DLQ fallback + backpressure signal Compose those primitives by hand

That's the value proposition. Everything else in these docs is "and here's how the pieces work".

10,000-foot view

flowchart TB
    subgraph App["DFE app"]
        Cfg["Config::default()"] --> SR["ServiceRuntime::new()"]
    end

    subgraph Pillars["Core pillars (auto-wired)"]
        Config
        Logger
        Metrics
        OTel
        Health
        Shutdown
    end

    subgraph Runtime["Runtime"]
        RC["RuntimeContext (K8s/Docker/BareMetal)"]
        MG["MemoryGuard"]
        SP["ScalingPressure"]
        WP["WorkerPool"]
        BE["BatchEngine"]
    end

    subgraph Pipeline["Pipeline"]
        T["Transport (Kafka/gRPC/HTTP/Redis/...)"]
        TF["TransportFilterEngine"]
        TS["TieredSink"]
        SPL["Spool"]
        DLQ
    end

    subgraph Deploy["Deployment artefacts"]
        DC["DeploymentContract"]
        DF["Dockerfile"]
        CH["chart/"]
        AC["argocd-application.yaml"]
    end

    SR --> Pillars
    SR --> Runtime
    Pipeline --> Pillars
    TF -.-> T
    TS --> T
    TS --> SPL
    TS --> DLQ
    DC --> DF
    DC --> CH
    DC --> AC
Loading

Solid arrows are runtime data/control flow. Dashed arrows mark embedded sub-components (filter engine lives inside every transport).

Where to read what

Start here

Data plane (WorkBatch + self-regulation)

  • SELF-REGULATION.md -- ON by default; the three brains (MemoryGuard / ScalingPressure / UnifiedPressure), observe + tune
  • BACKPRESSURE.md -- gate the source never the sink; the per-stage brake/commit-token table; streaming sub-blocks
  • KAFKA-PATH.md -- the three batch sizes, sizing profiles + librdkafka names, rho~0.7 loop, partition-limited diagnostic

Core pillars (always-on, auto-wired)

Runtime

Transport

Deployment

Pipeline

Less-common subsystems

Planned (not in current release)

  • Content-based log scrubbing (gitleaks rules + PII validators composed via strmatch). The current release ships field-name masking via MaskingWriter only — see core-pillars/LOGGING.md for what's shipped.

Workflow artefacts (not user docs)

  • superpowers/ — design specs and execution plans for in-flight work
  • MIGRATIONS.md — API surface changes by rustlib version; consumer-rebuild playbook

Project facts